Role Description:
We are seeking a skilled professional to maintain and support batch jobs in a legacy environment. The role involves managing and monitoring ETL processes, addressing issues, and enhancing existing PL/SQL scripts. The ideal candidate will have strong expertise in Informatica, SQL Server, and data warehousing concepts, along with experience in troubleshooting and improving batch job performance.
Key Responsibilities:
· Design and implement robust ETL pipelines using AWS Glue, Lambda, Redshift and S3.
· Monitor and optimize the performance of data workflows and batch processing jobs.
· Troubleshoot and resolve issues related to data pipeline failures, inconsistencies, and performance bottlenecks.
· Collaborate with cross-functional teams to define data requirements and ensure data quality and accuracy.
· Develop and maintain automated solutions for data transformation, migration, and integration tasks.
· Implement best practices for data security, data governance, and compliance within AWS environments.
· Continuously improve and optimize AWS Glue jobs, Lambda functions, and S3 storage management.
· Maintain comprehensive documentation for data pipeline architecture, job schedules, and issue resolution processes.
Required Skills and Experience:
· Strong experience with Data Engineering practices.
· Experience in AWS services, particularly AWS Glue, Lambda, S3, and other AWS data tools.
· Proficiency in SQL, python , Pyspark, numpy etc and experience in working with large-scale data sets.
· Experience in designing and implementing ETL pipelines in cloud environments.
· Expertise in troubleshooting and optimizing data processing workflows.
· Familiarity with data warehousing concepts and cloud-native data architecture.
· Knowledge of automation and orchestration tools in a cloud-based environment.
· Strong problem-solving skills and the ability to debug and improve the performance of data jobs.
· Excellent communication skills and the ability to work collaboratively with cross-functional teams.
· Good to have knowledge of DBT & Snowflake
We are an Equal Opportunity Employer:
We value diversity at Incedo. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.