Job Summary
Brief Description
Join our data team to build reliable data pipelines using Azure and Databricks, collaborate with U.S.-based stakeholders, and apply AI-assisted engineering practices. Own Azure Databricks pipelines from requirements through production support. Partner directly with U.S.-based stakeholders, source-system owners, and BI developers to deliver secure, reliable datasets for enterprise reporting and analytics.
RESPONSIBILITIES
• Pipelines & Ingestion: Independently build database, file, and API ingestion into ADLS Gen2 and Azure Databricks; implement incremental loads, change data capture where appropriate, and safe reprocessing.
• Transformations & Modeling: Develop SQL and Python/PySpark transformations and Delta Lake tables; build bronze, silver, and gold layers and analytics-ready models with clear data contracts.
• Reliability & Orchestration: Schedule dependencies through Databricks Jobs or Azure Data Factory; implement validation, monitoring, alerts, retries, and recovery procedures to meet agreed data freshness targets.
• Security & Delivery: Apply Unity Catalog access controls and lineage, use Azure Key Vault or approved secret management, and deliver automated tests and CI/CD releases through Git-based review.
• Performance & Partnership: Tune Spark jobs, queries, and table layout for performance and cost; gather requirements with U.S.-based stakeholders and document mappings, quality rules, and operational runbooks.
• AI-Assisted Engineering: Use approved AI tools to assist with SQL/Python development, testing, troubleshooting, and documentation; validate generated code and outputs and protect client data.
Preferred Skills
Experience ranges are indicative; equivalent demonstrated capability will be considered.
• Required - Experience: Typical range: 3-5 years in data engineering, including 2+ years delivering Azure Databricks solutions. Demonstrate independent pipeline delivery from requirements through deployment and production support.
• Required - Core Skills: Strong SQL, Python/PySpark, Delta Lake, ADLS Gen2, ETL/ELT, and data modeling; hands-on incremental ingestion, orchestration, automated testing, and Spark performance tuning.
• Required - Delivery: Practical Unity Catalog, secure Azure access, Git/CI/CD, monitoring, and recovery experience; ability to manage commitments and explain data quality issues to U.S.-based stakeholders.
• Preferred - Additional Platforms: Experience with one or more: Azure Data Factory, Event Hubs, Structured Streaming, Lakeflow pipelines, Terraform, Microsoft Fabric, or Azure Synapse Analytics. Depth across all tools is not expected.
• Preferred - Integration & Certification: Power BI integration or analytics data modeling is a plus. Relevant Databricks data engineering certification is preferred; equivalent practical experience is welcomed.
SKILLS : Azure Databricks · ADLS Gen2 · SQL · Python/PySpark · Delta Lake · Unity Catalog · ETL/ELT · Orchestration · CI/CD · Performance Tuning