Senior Data Engineer

Founded in 1976, CGI is among the world's largest independent IT and business consulting services firms. With 94,000 consultants and professionals globally, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services, and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68billion, and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more atcgi.com.
Job Title: Senior Data EngineerPosition: SSE LA ACExperience: 6+ years of experienceCategory: Software DevelopmentJob location: PunePosition ID: J0825-0171Work Type: HybridEmployment Type: Full Time PermanentQualification: Bachelors or Masters degree in Computer Science, Engineering, or a related field.Required Skills & ExperienceProficient in Python, with deep experience using pandas or polarsStrong understanding of ETL development, data extraction, and transformationHands-on experience with SQL and querying large datasetsExperience deploying workflows on Apache AirflowFamiliar with web scraping techniques (Selenium is a plus)Comfortable working with various data formats and large-scale datasetsExperience with Azure DevOps, including pipeline configuration and automationFamiliarity with Pytest or equivalent test frameworksStrong communication skills and a team-first attitude.Experience with DatabricksFamiliarity with AWS servicesWorking knowledge of Jenkins and advanced ADO Pipelines Key ResponsibilitiesDesign, build, and maintain pipelines in Python to collect data from a wide range of sources (APIs, SFTP servers, websites, emails, PDFs, etc.)Deploy and orchestrate workflows using Apache AirflowPerform web scraping using libraries like requests, BeautifulSoup, SeleniumHandle structured, semi-structured, and unstructured data efficientlyTransform datasets using pandas and/or polarsWrite unit and component tests using pytestCollaborate with platform teams to improve the data scraping frameworkQuery and analyze data using SQL (PostgreSQL, MSSQL, Databricks)Conduct code reviews, support best practices, and improve coding standards across the teamManage and maintain CI/CD pipelines (Azure DevOps Pipelines, Jenkins) Tech stack: Main/essential:Python - Pandas and/or Polars - EssentialSQLAzure DevOpsAirflow Additional:DatabricksAWSJenkinsADO Pipelines