82 Scrapy Jobs - Page 4

Senior ML Engineer Gurgaon, India; Ahmedabad, India; Hyderabad, India; Noida, India Information Technology 315679 Job Description About The Role: Grade Level (for internal use): 10 The Team : As a member of the Data Transformation - Cognitive Engineering team you will work on building and deploying ML powered products and capabilities to power natural language understanding, data extraction, information retrieval and data sourcing solutions for S&P Global Market Intelligence and our clients. You will spearhead deployment of AI products and pipelines while leading-by-example in a highly engaging work environment. You will work in a (truly) global team and encouraged for thoughtful risk-taking and self-initiative. What’s in it for you: Be a part of a global company and build solutions at enterprise scale Lead a highly skilled and technically strong team (including leadership) Contribute to solving high complexity, high impact problems Build production ready pipelines from ideation to deployment Responsibilities: Design, Develop and Deploy ML powered products and pipelines Mentor a team of Senior and Junior data scientists / ML Engineers in delivering large scale projects Play a central role in all stages of the AI product development life cycle, including: Designing Machine Learning systems and model scaling strategies Research & Implement ML and Deep learning algorithms for production Run necessary ML tests and benchmarks for model validation Fine-tune, retrain and scale existing model deployments Extend existing ML library’s and write packages for reproducing components Partner with business leaders, domain experts, and end-users to gain business understanding, data understanding, and collect requirements Interpret results and present them to business leaders Manage production pipelines for enterprise scale projects Perform code reviews & optimization for your projects and team Lead and mentor by example, including project scrums Technical Requirements: Proven track record as a senior / lead ML engineer Expert proficiency in Python (Numpy, Pandas, Spacy, Sklearn, Pytorch/TF2, HuggingFace etc.) Excellent exposure to large scale model deployment strategies and tools Excellent knowledge of ML & Deep Learning domain Solid exposure to Information Retrieval, Web scraping and Data Extraction at scale Exposure to the following technologies - R-Shiny/Dash/Streamlit, SQL, Docker, Airflow, Redis, Celery, Flask/Django/FastAPI, PySpark, Scrapy Experience with SOTA models related to NLP and expertise in text matching techniques, including sentence transformers, word embeddings, and similarity measures Open to learning new technologies and programming languages as required A Master’s / PhD from a recognized institute in a relevant specialization Good to have: 6-7+ years of relevant experience in ML Engineering Prior substantial experience from the Economics/Financial industry Prior work to show on Github, Kaggle, StackOverflow etc. What’s In It For You? Our Purpose: Progress is not a self-starter. It requires a catalyst to be set in motion. Information, imagination, people, technology–the right combination can unlock possibility and change the world. Our world is in transition and getting more complex by the day. We push past expected observations and seek out new levels of understanding so that we can help companies, governments and individuals make an impact on tomorrow. At S&P Global we transform data into Essential Intelligence®, pinpointing risks and opening possibilities. We Accelerate Progress. Our People: We're more than 35,000 strong worldwide—so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all. From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We’re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. We’re constantly seeking new solutions that have progress in mind. Join us and help create the critical insights that truly make a difference. Our Values: Integrity, Discovery, Partnership At S&P Global, we focus on Powering Global Markets. Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of integrity in all we do, bring a spirit of discovery to our work, and collaborate in close partnership with each other and our customers to achieve shared goals. Benefits: We take care of you, so you can take care of business. We care about our people. That’s why we provide everything you—and your career—need to thrive at S&P Global. Our benefits include: Health & Wellness: Health care coverage designed for the mind and body. Flexible Downtime: Generous time off helps keep you energized for your time on. Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills. Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs. Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best-in class benefits for families. Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference. For more information on benefits by country visit: https://spgbenefits.com/benefit-summaries Global Hiring and Opportunity at S&P Global: At S&P Global, we are committed to fostering a connected and engaged workplace where all individuals have access to opportunities based on their skills, experience, and contributions. Our hiring practices emphasize fairness, transparency, and merit, ensuring that we attract and retain top talent. By valuing different perspectives and promoting a culture of respect and collaboration, we drive innovation and power global markets. - Equal Opportunity Employer S&P Global is an equal opportunity employer and all qualified candidates will receive consideration for employment without regard to race/ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, marital status, military veteran status, unemployment status, or any other status protected by law. Only electronic job submissions will be considered for employment. If you need an accommodation during the application process due to a disability, please send an email to: EEO.Compliance@spglobal.com and your request will be forwarded to the appropriate person. US Candidates Only: The EEO is the Law Poster http://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf describes discrimination protections under federal law. Pay Transparency Nondiscrimination Provision - https://www.dol.gov/sites/dolgov/files/ofccp/pdf/pay-transp_%20English_formattedESQA508c.pdf - IFTECH202.1 - Middle Professional Tier I (EEO Job Group) Job ID: 315679 Posted On: 2025-05-20 Location: Gurgaon, Haryana, India

Job Title: AI/ML & Automation Intern Location: Remote Duration: 6 Months About us Astraeus Next Gen Pvt. Ltd. is building the next generation of intelligent automation and analytics tools powered by AI/ML. Our mission is to transform how businesses access, analyze, and act on data with zero-click intelligence. Role Overview We are seeking highly motivated interns from Tier 1 institutes (IITs, NITs, IIITs, BITS, etc.) with strong foundations in AI/ML, automation, full-stack development, deployment, and ETL pipelines. As an intern, you will work on live projects that involve developing intelligent automation tools, deploying scalable solutions, and building production-grade web apps and APIs. Key Responsibilities • Design, train, and evaluate ML models for real-world applications • Build scalable ETL pipelines to ingest and process large datasets • Automate web scraping, data collection, and report generation tasks • Develop and deploy backend services (Flask/FastAPI/Django) and frontend components (React.js/Next.js) • Deploy models and APIs on cloud platforms (AWS/GCP/Azure) using Docker, CI/CD pipelines • Collaborate with senior developers and product teams on full lifecycle development Requirements • Currently enrolled in a Tier 1 engineering institute (IIT, NIT, IIIT, BITS, etc.) • Strong programming skills in Python, JavaScript (Node/React), SQL • Hands-on experience with machine learning frameworks (Scikit-learn, TensorFlow, PyTorch) • Knowledge of full-stack development and modern web frameworks • Familiarity with cloud deployment, Docker, and REST APIs • Experience with automation/scraping tools (Selenium, Scrapy, BeautifulSoup) is a plus • Knowledge of MongoDB, Postgres, or other databases • Ability to work independently and in fast-paced, agile teams Good to Have • Exposure to tools like Apache Airflow, n8n, Kafka • Experience in report generation using Pandas, ReportLab, or similar tools • Contributions to open-source or personal projects in relevant domains What You’ll Get • Real-world experience building products. • Opportunity to convert to a full-time role • Work on high-impact projects in AI, automation, and intelligence systems To Apply: Submit your resume, portfolio (if any), and GitHub/LinkedIn profile to [shreejitsen@astraeusnextgen.com] Subject Line: AI/ML & Automation Intern Application - [Your Name] Note: This is an unpaid internship Show more Show less

Job Title: Sr. Python Developer – Web Scraper Location: Ahmedabad, Gujarat Job Type: Full Time Experience: 3+ Years Department: Python About Simform: Simform is a premier digital engineering company specializing in Cloud, Data, AI/ML, and Experience Engineering to create seamless digital experiences and scalable products. Simform is a strong partner for Microsoft, AWS, Google Cloud, and Databricks. With a presence in 5+ countries, Simform primarily serves North America, the UK, and the Northern European market. Simform takes pride in being one of the most reputed employers in the region, having created a thriving work culture with a high work-life balance that gives a sense of freedom and opportunity to grow. Role Overview: The Sr. Python Developer – Web Scraper will be responsible for building scalable, resilient, and intelligent web scraping solutions to extract structured and unstructured data from websites, APIs, and digital platforms. This role involves handling anti-bot mechanisms, dynamic content rendering, and real-time data extraction pipelines, while ensuring high data quality and performance. The ideal candidate will have strong Python skills, deep understanding of scraping techniques, and experience working with headless browsers and cloud platforms. Key Responsibilities: Design and implement scalable web scraping frameworks to collect data from complex and dynamic websites. Develop custom spiders/crawlers using Python libraries like Playwright, Puppeteer, Selenium, Scrapy, or BeautifulSoup. Apply advanced anti-bot evasion strategies such as CAPTCHA solving, IP rotation, user-agent spoofing, browser fingerprinting, and session/cookie management. Automate scraping tasks across distributed systems using tools like Celery, Airflow, cron, and ETL orchestration platforms. Optimize scraper performance for speed, accuracy, and resilience to website structure changes. Implement network interception, DOM traversal, WebSocket handling, and headless browser control. Store and manage scraped data in cloud or local storage using PostgreSQL, MongoDB, or S3. Integrate scraping systems with APIs or microservices for data consumption and downstream workflows. Monitor scraper reliability and handle retry logic, error logging, and dynamic throttling. Write modular, well-documented, and testable Python code with proper unit testing and version control. Collaborate with engineers, data scientists, and stakeholders to understand scraping goals and align them with business needs. Required Skills & Qualifications: Bachelor’s/Master’s degree in Computer Science, Engineering, or a related field. 3+ years of experience in Python development with specialization in web scraping. Deep understanding of modern anti-scraping defenses and bypass techniques (e.g., CAPTCHA, IP bans, dynamic rendering). Proficiency with headless browser tools like Playwright, Puppeteer, or Selenium. Strong grasp of DOM manipulation, JavaScript execution, network inspection, and asynchronous scraping using asyncio, aiohttp, etc. Experience in handling large-scale data extraction and storage using SQL and NoSQL databases. Hands-on experience deploying scrapers and automation workflows on AWS, GCP, or Azure. Familiarity with containerization using Docker and optional experience with Kubernetes. Comfortable with REST API integration, job scheduling, and microservices-based environments. Strong debugging, optimization, and testing skills. Clear understanding of legal and ethical scraping boundaries. Why Join Us: Young Team, Thriving Culture Flat-hierarchical, friendly, engineering-oriented, and growth-focused culture Well-balanced learning and growth opportunities Free health insurance Office facilities with a game zone, in-office kitchen with affordable lunch service, and free snacks Sponsorship for certifications/events and library service Flexible work timing, leaves for life events, WFH, and hybrid options

Key Responsibilities: Build and maintain backend services using Python frameworks like Django , Flask , or FastAPI . Design and consume RESTful APIs to integrate with frontend and external systems. Perform web scraping using: Selenium BeautifulSoup Scrapy or equivalent libraries Contribute to AI/ML-based applications , including model integration, API wrapping, and data handling. Work on data preprocessing, analysis , and pipeline creation for ML applications. Collaborate with cross-functional teams including frontend developers, designers, and PMs. Participate in agile practices: daily standups, sprint planning, and code reviews. Learn and assist in projects involving no-code/low-code platforms . Requirements: Bachelor’s degree in Computer Science, Engineering, or related field. 1-2 years of Python development experience. Strong experience in web scraping using tools like: Selenium , BeautifulSoup , and Scrapy Proficient in backend frameworks: Django , Flask , or FastAPI . Good understanding of REST APIs and ORMs . Experience working with relational databases (PostgreSQL, MySQL). Familiarity with Git and collaborative development workflows. Exposure to AI/ML concepts , model integration, or use of tools like scikit-learn , Pandas , NumPy . Good problem-solving and debugging skills. Nice to Have: Experience deploying ML models or building inference APIs. Familiarity with TensorFlow , PyTorch , or other ML frameworks. Knowledge of Docker , CI/CD , and cloud platforms (AWS, GCP, Azure). Experience with No-Code/Low-Code tools (training will be provided if not experienced). Understanding of frontend basics (React.js or similar) is a plus. Working LOcation : Mohali (Punjab) Experience required - 1-2 Years Job Type: Full-time Pay: Up to ₹30,000.00 per month Benefits: Health insurance Work Location: In person

Data Science and AI Developer **Job Description:** We are seeking a highly skilled and motivated Data Science and AI Developer to join our dynamic team. As a Data Science and AI Developer, you will be responsible for leveraging cutting-edge technologies to develop innovative solutions that drive business insights and enhance decision-making processes. No salary will be provided during the 3-month probation period. **Key Responsibilities:** 1. Develop and deploy machine learning models for predictive analytics, classification, clustering, and anomaly detection. 2. Design and implement algorithms for data mining, pattern recognition, and natural language processing. 3. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. 4. Utilize advanced statistical techniques to analyze complex datasets and extract actionable insights. 5. Implement scalable data pipelines for data ingestion, preprocessing, feature engineering, and model training. 6. Stay updated with the latest advancements in data science, machine learning, and artificial intelligence research. 7. Optimize model performance and scalability through experimentation and iteration. 8. Communicate findings and results to stakeholders through reports, presentations, and visualizations. 9. Ensure compliance with data privacy regulations and best practices in data handling and security. 10. Mentor junior team members and provide technical guidance and support. **Requirements:** 1. Bachelor’s or Master’s degree in Computer Science, Data Science, Statistics, or a related field. 2. Proven experience in developing and deploying machine learning models in production environments. 3. Proficiency in programming languages such as Python, R, or Scala, with strong software engineering skills. 4. Hands-on experience with machine learning libraries/frameworks such as TensorFlow, PyTorch, Scikit-learn, or Spark MLlib. 5. Solid understanding of data structures, algorithms, and computer science fundamentals. 6. Excellent problem-solving skills and the ability to think creatively to overcome challenges. 7. Strong communication and interpersonal skills, with the ability to work effectively in a collaborative team environment. 8. Certification in Data Science, Machine Learning, or Artificial Intelligence (e.g., Coursera, edX, Udacity, etc.). 9. Experience with cloud platforms such as AWS, Azure, or Google Cloud is a plus. 10. Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka) is an advantage. Data Manipulation and Analysis : NumPy, Pandas Data Visualization : Matplotlib, Seaborn, Power BI Machine Learning Libraries : Scikit-learn, TensorFlow, Keras Statistical Analysis : SciPy Web Scrapping : Scrapy IDE : PyCharm, Google Colab HTML/CSS/JavaScript/React JS Proficiency in these core web development technologies is a must. Python Django Expertise: In-depth knowledge of e-commerce functionalities or deep Python Django knowledge. Theming: Proven experience in designing and implementing custom themes for Python websites. Responsive Design: Strong understanding of responsive design principles and the ability to create visually appealing and user-friendly interfaces for various devices. Problem Solving: Excellent problem-solving skills with the ability to troubleshoot and resolve issues independently. Collaboration: Ability to work closely with cross-functional teams, including marketing and design, to bring creative visions to life. interns must know about how to connect front end with datascience Also must Know to connect datascience to frontend **Benefits:** - Competitive salary package - Flexible working hours - Opportunities for career growth and professional development - Dynamic and innovative work environment Job Type: Full-time Pay: ₹8,000.00 - ₹12,000.00 per month Schedule: Day shift Ability to commute/relocate: Thiruvananthapuram, Kerala: Reliably commute or planning to relocate before starting work (Preferred) Work Location: In person

Proficient in Python, web scraping (BeautifulSoup, Scrapy, Selenium), HTML/CSS/JS, HTTP, DevTools, IP rotation, proxies, automation, bypassing protections (e.g. Cloudflare), GCP, GraphQL, NoSQL, and Selenium Wire. Skilled in scalable data handling. Provident fund Health insurance

COMPANY PROFILE:Company Name: Gmware pvt. ltd. Gmware is an IT company that specializes in serving overseas clients in different verticals. We are looking for a candidate who can join our engineering team and help us develop and maintain various software products. Requirements :Proven 6month+ of relevant experience as a Web Crawling & Scraping EngineerStay updated on the latest trends and technologies in web scraping, crawling, and data processing.Strong problem-solving skillsExcellent communication and collaboration skills.Handle dynamic and complex websites, including those with anti-scraping mechanisms.Troubleshoot and resolve issues related to data extraction, including data integrity and performance challenges.Proficient in popular web scraping technologies and frameworks, such as Selenium, Beautiful soup, Scrapy Contact No. : 80555-03000