Data Engineer

You have a minimum of 3 years of experience working with various AWS services such as SQS, S3, Step Functions, EFS, Lambda, and OpenSearch. Your role will involve handling API integrations, particularly with large-scale endpoints. Proficiency in PySpark is required for data processing and parallelism within large-scale ingestion pipelines. Additionally, you should be familiar with AWS OpenSearch APIs for managing search indices. Your responsibilities will include utilizing Terraform expertise to automate and manage cloud infrastructure. Hands-on experience with AWS SageMaker is necessary, including working with machine learning models and endpoints. A strong understanding of data flow architectures, document stores, and journal-based systems is expected from you. Experience in parallelizing data processing workflows to meet performance and SLA requirements is essential. Familiarity with AWS tools like CloudWatch for monitoring pipelines is preferred. You should also possess strong problem-solving and debugging skills within distributed systems. Prior experience in optimizing ingestion pipelines for cost-efficiency and scalability is an advantage, along with a solid understanding of distributed data processing and workflow orchestration in AWS environments. In terms of soft skills, effective communication and collaboration skills are necessary for seamless teamwork across different functions. The ability to thrive in a fast-paced environment and deliver high-quality results within tight deadlines is crucial. An analytical mindset focused on performance optimization and continuous improvement will be beneficial in this role.,