Databricks

Job Summary

We are seeking a Sr. Developer with 8 to 12 years of experience to join our team. The ideal candidate will have expertise in Spark in Scala Delta Sharing and Databricks Unity Catalog Admin. This role involves working with cutting-edge technologies like Databricks CLI Delta Live Pipelines and Structured Streaming. The candidate will contribute to risk management and data solutions using Apache Airflow Amazon S3 and Amazon Redshift. Proficiency in Python Databricks SQL and PySpark is esse

Responsibilities

• Develop and maintain scalable data pipelines using Spark in Scala to process large datasets efficiently.
• Implement Delta Sharing and Databricks Unity Catalog Admin to ensure secure and seamless data sharing across platforms.
• Utilize Databricks CLI and Delta Live Pipelines to automate data workflows and enhance operational efficiency.
• Design and execute Structured Streaming applications to process real-time data streams with precision.
• Collaborate with cross-functional teams to integrate risk management solutions into data processing frameworks.
• Leverage Apache Airflow to orchestrate complex data workflows and ensure timely data delivery.
• Manage data storage and retrieval using Amazon S3 and Amazon Redshift to optimize performance and cost.
• Apply Python programming skills to develop robust data processing scripts and applications.
• Utilize Databricks SQL to perform advanced data analysis and generate actionable insights.
• Implement Databricks Delta Lake to ensure data reliability and consistency across the data lifecycle.
• Configure and manage Databricks Workflows to streamline data operations and improve productivity.
• Develop PySpark applications to process and analyze large-scale datasets with high efficiency.
• Contribute to the companys data strategy by delivering innovative solutions that drive business growth and societal impact.

Qualifications

• Possess a strong understanding of Spark in Scala and its application in data processing.
• Demonstrate expertise in Delta Sharing and Databricks Unity Catalog Admin for secure data management.
• Have experience with Databricks CLI and Delta Live Pipelines for workflow automation.
• Be proficient in Structured Streaming for real-time data processing.
• Show capability in risk management integration within data frameworks.
• Have hands-on experience with Apache Airflow for workflow orchestration.
• Be skilled in using Amazon S3 and Amazon Redshift for data storage solutions.
• Exhibit proficiency in Python for data processing and scripting.
• Be adept at using Databricks SQL for data analysis.
• Understand the principles of Databricks Delta Lake for data consistency.
• Have experience managing Databricks Workflows for operational efficiency.
• Be skilled in developing PySpark applications for large-scale data processing.