Posted:3 days ago| Platform: Linkedin logo

Apply

Work Mode

On-site

Job Type

Full Time

Job Description

Job Overview:

We are looking for a highly skilled Python Developer with strong experience in building both web and mobile app crawlers. The ideal candidate should have in-depth knowledge of Scrapy, Selenium, REST APIs, proxy rotation, and techniques to bypass anti-scraping mechanisms. You'll be responsible for building scalable, stealthy crawlers that extract data from websites and mobile apps (via APIs or reverse engineering traffic).


Responsibilities:

  • Develop and maintain web crawlers using Scrapy, Selenium, and custom spiders.
  • Build and manage mobile app crawlers by intercepting and decoding API requests, using tools like mitmproxy, Charles Proxy, or Burp Suite.
  • Handle proxy rotation, user-agent spoofing, cookie/session management, and headless browsers.
  • Monitor and adapt to changing structures of target websites and apps to maintain scraping accuracy.
  • Optimize performance, manage rate limits, and avoid detection (CAPTCHA, IP blocks).
  • Structure and store extracted data in JSON, CSV, or databases.
  • Maintain logs, error tracking, and reprocessing pipelines for failed jobs.


Required Skills:

  • Strong expertise in Python with deep knowledge of Scrapy, Selenium, and requests/BeautifulSoup.
  • Experience with mobile app traffic analysis using proxies/sniffers.
  • Understanding of RESTful APIs, HTTP methods, and JSON/XML formats.
  • Familiarity with proxy services, rotating residential/datacenter IPs, and anti-bot evasion.
  • Solid grasp of HTML, DOM parsing, and browser automation.
  • Hands-on with Git, Linux command line, and virtual environments.


Preferred Skills:

  • Experience with tools like mitmproxy, Charles Proxy, Burp Suite.
  • Familiarity with headless browsers like Puppeteer, Playwright (optional).
  • Ability to reverse engineer API calls from Android/iOS apps.
  • Knowledge of Docker, cloud deployment (AWS/GCP), and job schedulers.
  • Basic understanding of captcha-solving services (2Captcha, CapMonster, etc.).


Bonus Points:

  • Built crawlers for domains like e-commerce, travel, social media, or financial data.
  • Experience with CI/CD pipelines for automated crawling workflows.
  • Knowledge of data cleaning, ETL, or streaming pipelines (Kafka, Airflow, etc.).



More Jobs at TOSS Solutions [ Training | Outsourcing | Sales | Service ]

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now

RecommendedJobs for You