Senior Data Engineer

TheRole

We'reseeking a Senior Data Engineer to build and maintain theML/AIdata infrastructure powering our email security platform. In this role,you'lldesign andoptimizescalable data pipelines that enable threat detection and investigation while supporting both machine learning models and LLM-powered agents thatprovidecontext-aware security insights.

You'llwork on our Detection Intelligence Platform (DIP) buildingfeature engineering frameworks, and offline/online feature stores that serve as the foundation for ML modelresearchandcontext engineering for AI agents.You'llcollaborate with data scientists, ML engineers, and security researchers to builddatamodels and context stores that power our detection systems and enable human security analysts to investigate threats effectively.

Key Responsibilities:

• Developandmaintainscalable data pipelines on AWS/Azureusingtechnologies such asSpark,Airflow,Athena,Kubernetesetc.to process structured and unstructured email data at scale

• DesignandoptimizeIceberg-based data laketablesand schemasfor efficientstorage,queryingandversioningacross petabyte-scale datasetsdistributed across data centers globally

• Build and managefeature engineering frameworks that support offline batch processing and online real-time featureservingfor ML model training and inference

• Develop andmaintaintraining data pipelinesoptimizedfor distributed ML model training, ensuring data lineage and reproducibility

• Collaborate with data scientists and security researchers to understand data requirements and translate them into robust, production-grade data solutions

• Monitor andoptimizedata pipeline performance, implementing observability and alerting to ensure data freshness and quality

• Mentor junior engineers and foster a culture of engineering excellence and knowledge sharing

Required Experience:

• Several years of industry experience building andmaintainingdistributed data systems and high-scale data pipelines in a managed cloud environment (AWS/ Azure / GCP)using big data processing engines such as Spark, Flink,Dask,Ray,Beam,DataBricksWorkflowsor similar

• Deepproficiencyin Python for developing production-grade data processing code

• Strong experience with Infrastructure-as-Code frameworks, particularly Terraform

• Solid understandingand hands-on experience withopen table formats for data lakes(Apache Iceberg, Hudi,DeltaLake)and data modeling best practices

• Experience with AWS Athena, Glue, or similar data query and cataloging services

• Experience with Apache Airflow or similar workflow orchestration tools for batch and real-time pipeline management

• Demonstrated ability to design and implement scalable ETL/ELT pipelines handling complex data transformations

• Excellent communication skills and ability to collaborate effectively with technical and non-technical stakeholders

Good to have:

• Experiencewithfeature engineering frameworks and feature stores (e.g., Feast,Tecton, or custom solutions)

• Familiarity with Kubernetes for containerized data workloads and orchestration

• Background in building data infrastructure for machine learning and AI applications

• Experience with data quality frameworks and observability tools for data pipelines