A VPS is a natural home for web scrapers and data collection jobs: it runs 24/7, has fast network connectivity, and keeps long jobs off your laptop. But scraping also comes with technical, ethical and legal responsibilities. This guide covers both: how to set it up well, and how to stay on the right side of website owners and your hosting provider.

Typical uses

  • Price and availability monitoring (your own products, or permitted sources).
  • Collecting public datasets for research.
  • Monitoring your own sites (broken links, content changes, SEO checks). See VPS for SEO tools.
  • Aggregating data from APIs and feeds.

Prefer official APIs where they exist. They’re more stable, faster, and usually allowed by the terms of service.

Tools

Tool Best for
requests / httpx + BeautifulSoup or lxml (Python) Simple static pages
Scrapy (Python) Large crawls: concurrency, throttling, retries, pipelines
Playwright / Puppeteer JavaScript-heavy sites (headless Chromium)
Crawlee (Node.js/Python) Crawling with browser and HTTP modes, queues
n8n / scheduled scripts Light periodic jobs. See n8n

Set up an isolated Python environment for each project. See Python virtual environments.

Sizing

Workload Suggested VPS
HTTP-only scrapers (requests/Scrapy) 1–2 vCPU, 2 GB RAM. Mostly network-bound
Headless browsers (Playwright/Chromium) 2–4 vCPU, 4–8 GB RAM. Each browser uses 200–500 MB+
Large crawls with storage and processing 4+ vCPU, 8 GB+ RAM, plenty of disk

Bandwidth matters for big crawls. Check your plan’s monthly transfer. See VPS bandwidth explained.

Install headless Chromium for Playwright

python3 -m venv ~/scraper && . ~/scraper/bin/activate
pip install playwright
playwright install --with-deps chromium

Close browser contexts when done, and limit concurrency, or memory will climb until the out-of-memory killer steps in. See Linux OOM killer.

Scheduling and reliability

  • cron or systemd timers for periodic jobs. See crontab and systemd timers.
  • Queues (Redis plus RQ/Celery, or Scrapy’s scheduler) for large crawls with retries.
  • Logging and alerts: know when a job fails, or when a site’s HTML changes and your parser silently returns nothing.
  • Store results in PostgreSQL, SQLite, or files in object storage. Back them up. See MinIO.

Scrape responsibly

These practices protect the sites you visit, and your own server:

  • Respect robots.txt and the site’s terms of service.
  • Rate-limit your requests (Scrapy: AUTOTHROTTLE_ENABLED = True, DOWNLOAD_DELAY, CONCURRENT_REQUESTS_PER_DOMAIN). Back off on 429 and 503 responses.
  • Identify yourself with an honest User-Agent and contact information where appropriate.
  • Cache pages, and don’t re-fetch unchanged content.
  • Don’t collect personal data without a lawful basis. Privacy laws (GDPR and others) apply to scraped data too.
  • Don’t bypass access controls: logins you aren’t authorised to use, paywalls, or CAPTCHAs designed to stop automated access.

Copyright, database rights and computer misuse laws vary by country. Get legal advice for commercial projects.

Your hosting provider’s rules

Aggressive scraping looks like an attack to the target, and generates abuse reports to your provider. Repeated reports can lead to suspension. Keep crawl rates polite, and stop when asked. Read your provider’s acceptable use policy. See the VPS.DO legal policies.

Multiple IPs

Some legitimate projects (for example monitoring localized content in different regions) need several IP addresses or locations. Multi-IP VPS plans are available in the USA and Hong Kong. Use them for legitimate geo-testing and separation of services, not to evade rate limits or bans. See multi-IP VPS.

Quick reference

Need Tool / setting
Static pages requests/httpx + lxml
Large crawls Scrapy with AutoThrottle
JavaScript sites Playwright (playwright install --with-deps chromium)
Scheduling cron / systemd timers / queues
Politeness robots.txt, delays, back off on 429

Related: Python on Ubuntu and Docker VPS hosting.

  • Web Scraping
  • Python
  • VPS Guide