Key Responsibilities:
-
Design, develop, and maintain ETL (Extract, Transform, Load) processes to ensure the seamless
integration of raw data from various sources into our data lakes or warehouses.
-
Utilize Python, PySpark, SQL and AirFlow etc., to process, analyze, and store large-scale
datasets efficiently.
-
Write and maintain SQL queries for data retrieval, transformation, and storage in relational
databases like Redshift or PostgreSQL.
-
Support cloud-based data platforms such as AWS, Azure, or GCP, with a focus on orchestrating
AI retraining cycles, versioning, and automated pipeline monitoring.
-
Familiarity in converting unstructured data into vectors using frameworks like LangChain or
LlamaIndex and storing them.
-
Collaborate with cross-functional teams, including data scientists, ML engineers, and domain
experts to design and implement scalable solutions.
-
Troubleshoot and resolve performance issues, data quality problems, and errors in data pipelines.
-
Document processes, code, and best practices for future reference and team training.