BIG DATA TECHNICAL LEAD ENGINEER

Job Description

We are seeking a highly skilled Big Data Engineer with a strong background in designing and implementing ETL data pipelines using Apache Spark and either Scala or Python. The ideal candidate will have a deep understanding of big data frameworks, advanced SQL skills, and hands-on experience with Elasticsearch. You will be responsible for building scalable, efficient data solutions and ensuring data quality, reliability, and performance in large-scale environments. KEY RESPONSIBILITIES: Design and implement ETL pipelines: Develop, optimize, and maintain ETL processes for the ingestion, transformation, and extraction of large datasets using Spark (Scala/Python). Data modeling and processing: Build scalable data processing applications that handle high-volume and complex data from various sources. Spark Application Development: Write optimized, scalable Spark applications in Scala or Python to process and analyze large datasets. Elasticsearch Integration: Utilize Elasticsearch for indexing, searching, and querying big data; ensure the data in Elasticsearch is well-structured and optimized for performance. SQL Expertise: Develop complex SQL queries and scripts to perform data analysis, transformations, and ensure data integrity across the pipeline. Data Architecture: Collaborate with architects to design data solutions that are resilient, efficient, and easy to scale. Performance Tuning: Optimize Spark jobs and queries for performance, including tuning parameters, resource management, and troubleshooting issues related to data quality and performance. Collaboration: Work with cross-functional teams, including data scientists, analysts, and software engineers, to deliver end-to-end data solutions. Monitoring and Maintenance: Set up monitoring tools and frameworks to ensure the stability and reliability of data pipelines in production environments. REQUIRED SKILLS AND QUALIFICATIONS: 6+ years of experience in big data engineering or related fields. Strong expertise in Apache Spark for data processing, with hands-on experience developing Spark applications in Scala or Python. Experience in ETL pipeline development, including data ingestion, transformation, and processing of large-scale datasets. Proficient in Elasticsearch: Experience with Elasticsearch for querying and indexing large datasets, understanding of its architecture and performance optimization. Advanced SQL skills: Strong ability to write complex SQL queries for data extraction, aggregation, and analysis. Knowledge of distributed data processing and managing large data sets in a cloud or on-premise environment. Familiarity with Hadoop ecosystems, HDFS, and other big data technologies. Experience with version control (e.g., Git) and CI/CD pipelines. Strong problem-solving skills, attention to detail, and a mindset for performance tuning and optimization.