Senior Data Engineer

As an experienced candidate with over 8 years of overall experience and 5+ years of relevant experience, your role will involve the following key responsibilities: - Strong understanding of Spark core including RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming - Proficiency in Delta Lake features such as time travel, schema evolution, and data partitioning - Designing and building data pipelines using Spark and Delta Lake - Solid experience in Python/Scala/Java for Spark development - Knowledge of data ingestion from files, APIs, and databases - Familiarity with data validation and quality best practices - Working knowledge of data warehouse concepts and data modeling - Hands-on experience with Git for code versioning - Exposure to CI/CD pipelines and containerization tools Additionally, you should possess the following qualifications: - Strong understanding of Python, Spark (PySpark), SQL, and Delta Lake - Experience with ETL tools like DataStage, Prophecy, Informatica, or Ab Initio would be a plus Please note that this is a 6-month contract position based in Hyderabad.,