GCP - Data Engineering

Notice period 30 days to immediate

Role description

Myrefers GCP PythonApache beamp3 to 8 years of overall IT experience which includes hands on experience in Big Data technologies

Mandatory Hands on experience in Python and PySpark Python as a language is practically usable for anything we are looking for application Development and Extract Transform Load and Data lake curation experience using Python

Build pySpark applications using Spark Dataframes in Python using Jupyter notebook and PyCharm IDE

Worked on optimizing spark jobs that processes huge volumes of data

Hands on experience in version control tools like Git

Worked on Amazons Analytics services like Amazon EMR Amazon Athena AWS Glue

Worked on Amazons Compute services like Amazon Lambda Amazon EC2 and Amazons Storage service like S3 and few other services like SNS

Experience knowledge of bash shell scripting will be a plus

Has built ETL processes to take data copy it structurally transform it etc involving a wide variety of formats like CSV TSV XML and JSON

Experience in working with fixed width delimited multi record file formats etc

Good to have knowledge of datawarehousing concepts dimensions facts schemas snowflake star etc

Have worked with columnar storage formats Parquet Avro ORC etc Well versed with compression techniques Snappy Gzip

Good to have knowledge of AWS databases atleast one Aurora RDS Redshift ElastiCache DynamoDB

Skills

Mandatory Skills :GCP, Apache Spark,Python,SparkSQL,Big Data Hadoop Ecosystem