Databricks Developer

Job Summary

We are seeking a highly skilled Sr. Developer with 6 to 10 years of experience to join our dynamic team. The ideal candidate will have expertise in Spark in Scala Databricks Unity Catalog Admin and other related technologies. This role involves developing and optimizing data workflows ensuring data integrity and contributing to innovative projects that drive our companys success. The position is hybrid with day shifts and no travel required.

Responsibilities

• Develop and optimize data workflows using Spark in Scala to ensure efficient data processing and analysis.
• Implement and manage Databricks Unity Catalog Admin to maintain data governance and security.
• Utilize Databricks CLI for seamless integration and automation of data tasks.
• Design and schedule workflows with Apache Airflow to ensure timely data pipeline execution.
• Manage data storage and retrieval using Amazon S3 and Amazon Redshift for robust data solutions.
• Write and maintain Python scripts to support data processing and analysis tasks.
• Leverage Databricks SQL for querying and analyzing large datasets effectively.
• Implement Databricks Delta Lake to ensure data reliability and consistency.
• Create and manage Databricks Workflows to automate and streamline data processes.
• Develop PySpark applications to handle large-scale data processing tasks efficiently.
• Collaborate with cross-functional teams to deliver high-quality data solutions that meet business needs.
• Ensure data integrity and quality through rigorous testing and validation processes.
• Contribute to the continuous improvement of data practices and technologies within the organization.

Qualifications

• Possess strong expertise in Spark in Scala and Databricks Unity Catalog Admin.
• Demonstrate proficiency in using Databricks CLI and Apache Airflow.
• Have experience with Amazon S3 and Amazon Redshift for data management.
• Show advanced skills in Python and Databricks SQL.
• Be knowledgeable in Databricks Delta Lake and Databricks Workflows.
• Exhibit strong problem-solving skills and attention to detail.
• Be able to work effectively in a hybrid work model with day shifts.