Position Overview: We are seeking a skilled and motivated Sr Data Engineer to join our team. The ideal candidate will have a strong background in designing, developing, and optimizing data pipelines and ETL processes, with hands-on experience in Python, Databricks, and/or IICS (Informatica) or similar. The ideal candidate will also have proven experience leading teams of data analysts & ETL developers for solution delivery. This role requires a collaborative mindset, a passion for data engineering best practices, and the ability to deliver scalable and efficient data solutions that align with business objectives.
Key Responsibilities:
- Data Pipeline Development: Design, develop, and implement data pipelines and ETL processes using Python and Databricks and/or IICS (Informatica).
- Collaboration: Work closely with data scientists and analysts to understand data requirements and deliver solutions that meet business needs.
- Optimization: Optimize existing data workflows for performance, scalability, and reliability.
- Monitoring and Troubleshooting: Monitor and troubleshoot data pipeline issues, ensuring timely resolution to maintain data integrity and availability.
- Documentation: Document data processes, workflows, and technical specifications to ensure clarity and maintainability.
- Continuous Learning: Stay updated with the latest industry trends and technologies related to data engineering, cloud services, and data integration.
Technical Expertise:
- Proven experience in Python programming and data manipulation.
- Knowledge of Databricks and its ecosystem (e.g., Spark, Delta Lake).
- Experience with IICS (Informatica) or similar cloud-based data integration tools.
- Familiarity with SQL and database management systems (e.g., SQL Server, Snowflake).
- Understanding of data warehousing concepts and best practices.
- Experience with cloud platforms (e.g., AWS, Azure, GCP).
- Leadership & Collaboration:
- Ability to collaborate with business to determine technical solution requirements
- Experience leading dev teams using agile practices
- Problem-Solving and Optimization:
- Mid-level problem-solving skills with a focus on addressing performance issues in data workflows.
- Experience implementing optimization techniques and evaluating performance improvements using measurable metrics.
- Best Practices:
- Understanding of data engineering best practices, including data quality, reliability, and maintainability.
- Knowledge of robust monitoring practices and data governance principles.
- Soft Skills:
- Strong communication and collaboration skills.
- Familiarity with Agile development methodologies.
Preferred Skills:
- Hands-on experience with data storage solutions such as data lakes and data warehouses.
- Knowledge of data orchestration frameworks and their practical application in past projects.
- Insights into common pitfalls in the data lifecycle and strategies to avoid them.
