Data Engineer (Pyspark)

You have a deep experience in developing data processing tasks using PySpark/spark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations. You will be responsible for developing, programming, and maintaining applications using the Apache Spark and Python open-source framework. Your role will involve working with different aspects of the Spark ecosystem, including Spark SQL, DataFrames, Datasets, and streaming. As a Spark Developer, you must have strong programming skills in Python, Java, or Scala. It is essential to be familiar with big data processing tools and techniques, as well as have a good understanding of distributed systems. Your proven experience as a Spark Developer or a related role will be highly valuable in this position. Strong problem-solving and analytical thinking skills are required to excel in this role. Experience with building APIs for provisioning data to downstream systems will be beneficial. Working experience on any Cloud technology like AWS, Azure, or Google is an added advantage. Hands-on experience with AWS S3 Filesystem operations will also be beneficial for this position.,