[1319] Data Engineer

Job Details

This role requires a comprehensive understanding of the entire machine learning lifecycle combined with deep data engineering expertise.

The position involves designing and implementing end-to-end data pipelines, managing large-scale data migrations, establishing modern data lake architectures, and building complete MLOps pipelines. The candidate must have extensive knowledge of cloud platforms (AWS preferred) for ML/AI and data processing, data quality governance, and advanced infrastructure management using Infrastructure as Code practices. They must also demonstrate the ability to build stakeholder trust by positioning data as a strategic business enabler, not just a technical asset.

• Design, build, and maintain end-to-end data pipelines on AWS cloud platform.
• Implement POC projects and execute large-scale data migrations from legacy systems from on-premise systems to AWS cloud platform.
• Apply best practices in data modeling, ensuring scalability and performance on AWS cloud platform.
• Optimize queries, storage, and compute resources for cost efficiency.
• Build and maintain AWS data lake with proper optimization strategies
• Configure and manage AWS data lake security, including role-based access controls and governance.
• Develop ETL/ELT workflows on AWS cloud platform.
• Implement robust data quality controls and monitoring solutions
• Implement and design Feature Store for feature lineage, metadata management, and ACID transactions
• Build end-to-end MLOps pipelines for model training, validation, and endpoint deployment
• Implement model monitoring, explainability, and drift detection with automated notifications and model retraining triggers
• Manage model versioning and registry with proper governance and deployment controls
• Design and implement A/B testing for model evaluation and performance comparison
• Establish CI/CD pipelines for automated MLOps pipeline deployment and testing.

Job Requirements

• Core Technologies: Apache Spark/PySpark, Python, SQL, Table Formats (Apache Iceberg (Preferred) /Delta Lake/Apache Hudi), CDC, Enterprise Databases (Oracle/SQL Server/DB2/PostgreSQL)
• Streaming Technologies: Apache Kafka, MSK, Kinesis
• Cloud Platforms: S3, EC2, SageMaker, Glue, EMR, RedShift, Lake Formation, CloudWatch, QuickSight, Athena, Lambda, API Gateway
• Workflow Orchestration: AWS Step Functions/Glue Workflows or Amazon MWAA (Managed Airflow)
• Infrastructure & DevOps: CloudFormation, AWS CDK, Terraform, Docker, Linux/Unix, Git
• ML Libraries (Preferred): Scikit-learn, PyTorch, TensorFlow
• GenAI (Preferred): Applied experience working with LLMs, generative AI and Agentic AI (e.g., Hugging Face Transformers, LangChain, LangGraph, LlamaIndex, OpenAI API).