Data Platform Engineer

Key Responsibilities:

• Implement automated data engineering frameworks using medallion architecture via Delta Lake for reliable and harmonized data transformations at scale, leveraging Databricks and Apache Spark (PySpark, Spark SQL) for distributed data processing and transformation.

• Monitor and optimize pipeline performance using autoscaling, caching, and cost-effective strategies.

• Automate platform lifecycle management using scripting (Python, Terraform, PowerShell) and CI/CD pipelines (Azure Pipelines), enabling rapid resilience and secure deployment of data products and platform features.

• Establish and automate governance and quality frameworks aligned with platform principles, standards, and guardrails.

• Develop and manage robust ETL/ELT workflows (APIs, databases, files, streaming sources) for both batch and real-time processing, ensuring high data quality and reliability.

Required Experience:

• More than 6 years of proven experience building and scaling regional data platforms on Microsoft Azure with expertise in Databricks, Apache Spark (PySpark, Spark SQL), Python, SQL, Microsoft Fabric, Azure Data Factory, ADLS Gen2, Power BI, and Azure Functions.

• Strong experience in data product lifecycle development.

• Proficient in Git, Azure DevOps, CI/CD, and source control, including Databricks Asset Bundles.

• Expertise in developing and managing automation capabilities using Azure Pipelines, Python, PySpark, and PowerShell.

• Proven ability to identify, troubleshoot, and resolve complex technical issues with automated solutions.

• Strong understanding of security and compliance principles with the ability to provide guidance and oversight.

• Experience enabling AI/ML use cases on the platform (feature pipelines, model inference, data flows, serving ML outputs for downstream analytics) is highly valued.

• Experience working in the retail, luxury, or consumer industry is preferred.

• Excellent written and verbal communication skills in English, with the ability to work effectively in a globally distributed team.

• Experience producing high-quality documentation, including technical specifications, functional specifications, and user playbooks.

• Experience working within the SAFe Agile framework, driving design reviews, demos, and user training.

  Education

• Bachelor’s degree in Computer Science, IT, or a related field.

• Certifications in data engineering or related technologies are highly valued.

• Certifications in Databricks and Azure are highly preferred, e.g. Azure Solutions Architect Expert, Databricks Certified Data Engineer Associate.

  
  #LI-AL1