Lead Software Engineer Data Engineering AWS Data bricks

Job Description

Required Skills

Responsibilities

• Develop appropriate level designs and ensure consensus from peers where necessary.
• Collaborate with software engineers and cross-functional teams to design and implement deployment strategies using AWS Cloud and Databricks pipelines.
• Work with software engineers and teams to design, develop, test, and implement solutions within applications.
• Engage with technical experts, key stakeholders, and team members to resolve complex problems effectively.
• Understand leadership objectives and proactively address issues before they impact customers.
• Design, develop, and maintain robust data pipelines to ingest, process, and store large volumes of data from various sources.
• Implement ETL (Extract, Transform, Load) processes to ensure data quality and integrity using tools like Apache Spark and PySpark.
• Monitor and optimize the performance of data systems and pipelines.
• Implement best practices for data storage, retrieval, and processing.
• Maintain comprehensive documentation of data systems, processes, and workflows.
• Ensure compliance with data governance and security policies.