web scrapping

Job Overview:

We are looking for a highly skilled Python Developer with strong experience in building both web and mobile app crawlers. The ideal candidate should have in-depth knowledge of Scrapy, Selenium, REST APIs, proxy rotation, and techniques to bypass anti-scraping mechanisms. You'll be responsible for building scalable, stealthy crawlers that extract data from websites and mobile apps (via APIs or reverse engineering traffic).

Responsibilities:

• Develop and maintain web crawlers using Scrapy, Selenium, and custom spiders.
• Build and manage mobile app crawlers by intercepting and decoding API requests, using tools like mitmproxy, Charles Proxy, or Burp Suite.
• Handle proxy rotation, user-agent spoofing, cookie/session management, and headless browsers.
• Monitor and adapt to changing structures of target websites and apps to maintain scraping accuracy.
• Optimize performance, manage rate limits, and avoid detection (CAPTCHA, IP blocks).
• Structure and store extracted data in JSON, CSV, or databases.
• Maintain logs, error tracking, and reprocessing pipelines for failed jobs.

Required Skills:

• Strong expertise in Python with deep knowledge of Scrapy, Selenium, and requests/BeautifulSoup.
• Experience with mobile app traffic analysis using proxies/sniffers.
• Understanding of RESTful APIs, HTTP methods, and JSON/XML formats.
• Familiarity with proxy services, rotating residential/datacenter IPs, and anti-bot evasion.
• Solid grasp of HTML, DOM parsing, and browser automation.
• Hands-on with Git, Linux command line, and virtual environments.

Preferred Skills:

• Experience with tools like mitmproxy, Charles Proxy, Burp Suite.
• Familiarity with headless browsers like Puppeteer, Playwright (optional).
• Ability to reverse engineer API calls from Android/iOS apps.
• Knowledge of Docker, cloud deployment (AWS/GCP), and job schedulers.
• Basic understanding of captcha-solving services (2Captcha, CapMonster, etc.).

Bonus Points:

• Built crawlers for domains like e-commerce, travel, social media, or financial data.
• Experience with CI/CD pipelines for automated crawling workflows.
• Knowledge of data cleaning, ETL, or streaming pipelines (Kafka, Airflow, etc.).