Python , Pyspark

N Consulting Global

5 - 10 years

3 - 12 Lacs

Pune Maharashtra India

Posted:1 day ago| Platform: Foundit logo

Apply

Work Mode

On-site

Job Type

Full Time

Job Description

Design and implement scalable data processing pipelines usingPySparkfor batch and real-time data processing.
Develop Python scripts and libraries to integrate, cleanse, and process data from various data sources (databases, APIs, etc.).
Optimize PySpark jobs for performance and resource efficiency.
Collaborate with data engineers and analysts to define data transformation and aggregation requirements.
Analyze and troubleshoot issues in large-scale distributed data systems.
Develop custom functions for data manipulation and aggregation usingSpark DataFrame APIandRDD API.
Perform data analysis and create insightful visualizations for internal stakeholders.
Write unit tests, document code, and maintain code quality through peer reviews and best practices.
Monitor and optimize the performance ofApache Sparkclusters.
Work with cloud platforms such asAWS,Azure, orGCPto deploy Spark jobs and other data processing services.
Skills and Qualifications:Strong experience withPythonprogramming language and libraries such asPandas,NumPy,Pytest, etc.
Hands-on experience withApache SparkandPySparkfor distributed data processing.
Familiarity with data formats such asCSV,JSON,Parquet,Avro, and other big data file formats.
Solid understanding ofHadoop ecosystemcomponents (HDFS, YARN, etc.).
Proficiency with data wrangling, transformation, and aggregation.
Experience with performance tuning, resource management, and debugging Spark applications.
Understanding of cloud-based data storage solutions (S3, GCS, etc.) and distributed file systems.
Familiarity with data warehousing concepts and tools.
Good knowledge of SQL and relational databases (e.g., PostgreSQL, MySQL).
Strong problem-solving skills and ability to work with large datasets.
Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
Preferred Qualifications:Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
Experience with additional big data frameworks likeFlink,Kafka,Hive, etc.
Knowledge ofDockerand containerization for deployment.
Experience withETL toolslikeAirfloworLuigi.
Familiarity withmachine learningconcepts and frameworks (e.g.,MLlib,Scikit-learn).

More Jobs at N Consulting Global

SDET Automation Test Engineer

Pune

6 - 11 yrs

INR 11 - 15 Lacs

SDET QA Test Engineer

Pune

6 - 11 yrs

INR 10 - 14 Lacs

Technical Recruiter

Chennai

5.0 - 7.0 yrs

INR 7 - 9 Lacs

Sr. Software Developer (React)

Pune

5.0 - 10.0 yrs

INR 7 - 12 Lacs

Java Developer with Production support L2/L3

Bengaluru

4.0 - 8.0 yrs

INR 6 - 10 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Job-Specific Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.