VPS for Web Scraping: Run Python Crawlers 24/7 with Scrapy, Playwright, and Rotating Proxies
Running a web scraper on your laptop means it stops when the laptop sleeps, updates interrupt it, and IP blocks follow you everywhere. A VPS provides 24/7 execution, a clean IP address separate from your personal browsing, persistent storage for scraped data, and the compute to run multiple spiders simultaneously. This guide sets up a production-ready web scraping stack on a VPS.
Choosing the Right Tool
- Scrapy: Best for high-volume HTML scraping — handles async requests, rate limiting, retries, pipelines, and scheduling natively. 100–10,000 requests/minute.
- Playwright/Selenium: Required for JavaScript-rendered pages (React, Vue, Angular SPAs) or sites that require human-like interaction. 5–50 pages/minute (browser overhead).
- httpx/aiohttp + BeautifulSoup: Lightweight for simple, low-volume scraping tasks.
Step 1: Install Dependencies
<code">sudo apt update
sudo apt install -y python3.12 python3.12-venv \
chromium-browser chromium-chromedriver \
postgresql postgresql-contrib
mkdir -p /opt/scraper && cd /opt/scraper
python3.12 -m venv .venv
source .venv/bin/activate
pip install scrapy playwright httpx beautifulsoup4 \
lxml psycopg2-binary SQLAlchemy \
fake-useragent scrapy-rotating-proxies
# Install Playwright browsers
playwright install chromium
Step 2: Create a Scrapy Spider
<code">cd /opt/scraper scrapy startproject myproject cd myproject
<code">nano myproject/spiders/products_spider.py
<code">import scrapy
from datetime import datetime
class ProductsSpider(scrapy.Spider):
name = 'products'
allowed_domains = ['example-store.com']
start_urls = ['https://example-store.com/products']
custom_settings = {
# Be a good citizen — don't hammer the server
'DOWNLOAD_DELAY': 1, # 1 second between requests
'RANDOMIZE_DOWNLOAD_DELAY': True, # 0.5–1.5s (randomized)
'CONCURRENT_REQUESTS': 2, # 2 concurrent requests
'CONCURRENT_REQUESTS_PER_DOMAIN': 2,
'ROBOTSTXT_OBEY': True, # Always obey robots.txt
'AUTOTHROTTLE_ENABLED': True, # Auto-adjust rate based on response time
'AUTOTHROTTLE_TARGET_CONCURRENCY': 1.0,
}
def parse(self, response):
# Extract products on this page
for product in response.css('.product-card'):
yield {
'name': product.css('.product-name::text').get(),
'price': product.css('.product-price::text').get(),
'url': response.urljoin(product.css('a::attr(href)').get()),
'scraped_at': datetime.utcnow().isoformat(),
}
# Follow pagination
next_page = response.css('a.next-page::attr(href)').get()
if next_page:
yield response.follow(next_page, callback=self.parse)
Step 3: Store Data in PostgreSQL
<code">nano myproject/pipelines.py
<code">import psycopg2
from datetime import datetime
class PostgreSQLPipeline:
def open_spider(self, spider):
self.conn = psycopg2.connect(
host='localhost',
database='scraper',
user='scraper',
password='password',
)
self.cursor = self.conn.cursor()
self.cursor.execute("""
CREATE TABLE IF NOT EXISTS products (
id SERIAL PRIMARY KEY,
name TEXT,
price TEXT,
url TEXT UNIQUE,
scraped_at TIMESTAMP DEFAULT NOW()
)
""")
self.conn.commit()
def process_item(self, item, spider):
self.cursor.execute("""
INSERT INTO products (name, price, url, scraped_at)
VALUES (%s, %s, %s, %s)
ON CONFLICT (url) DO UPDATE
SET price = EXCLUDED.price,
scraped_at = EXCLUDED.scraped_at
""", (item['name'], item['price'], item['url'], item['scraped_at']))
self.conn.commit()
return item
def close_spider(self, spider):
self.cursor.close()
self.conn.close()
<code">nano myproject/settings.py
<code"># Add to settings.py:
ITEM_PIPELINES = {
'myproject.pipelines.PostgreSQLPipeline': 300,
}
# User-agent rotation
USER_AGENT = 'Mozilla/5.0 (compatible; YourBot/1.0; +https://yourdomain.com/bot)'
# Respect rate limits and robots.txt
ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 1
# HTTP cache (avoid re-scraping unchanged pages)
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 86400 # Cache for 24 hours
HTTPCACHE_DIR = '/opt/scraper/httpcache'
Step 4: Playwright for JavaScript-Rendered Pages
<code">nano /opt/scraper/playwright_scraper.py
<code">"""Scrape JavaScript-rendered pages with Playwright."""
import asyncio
from playwright.async_api import async_playwright
import json
async def scrape_spa(url: str) -> dict:
async with async_playwright() as p:
browser = await p.chromium.launch(
headless=True,
args=[
'--no-sandbox', # Required for VPS (no display server)
'--disable-dev-shm-usage',
'--disable-gpu',
]
)
context = await browser.new_context(
user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36',
viewport={'width': 1280, 'height': 720},
)
page = await context.new_page()
# Block images/fonts to speed up loading
await page.route('**/*.{png,jpg,jpeg,gif,svg,woff,woff2}',
lambda route: route.abort())
await page.goto(url, wait_until='networkidle', timeout=30000)
# Wait for specific element to appear (confirm content loaded)
await page.wait_for_selector('.product-grid', timeout=10000)
# Extract data
products = await page.evaluate("""
() => Array.from(document.querySelectorAll('.product-item')).map(el => ({
name: el.querySelector('.name')?.textContent?.trim(),
price: el.querySelector('.price')?.textContent?.trim(),
url: el.querySelector('a')?.href,
}))
""")
await browser.close()
return products
async def main():
data = await scrape_spa('https://example-spa.com/products')
print(json.dumps(data, indent=2))
if __name__ == '__main__':
asyncio.run(main())
Step 5: Schedule Spiders with Cron
<code">crontab -e
<code"># Run products spider daily at 6 AM
0 6 * * * cd /opt/scraper/myproject && \
/opt/scraper/.venv/bin/scrapy crawl products \
>> /var/log/scraper.log 2>&1
# Run another spider every 4 hours
0 */4 * * * cd /opt/scraper/myproject && \
/opt/scraper/.venv/bin/scrapy crawl prices \
>> /var/log/scraper-prices.log 2>&1
Step 6: Rotating Proxies
<code">nano myproject/settings.py
<code"># Using scrapy-rotating-proxies with a list of residential proxies
ROTATING_PROXY_LIST = [
'user:pass@proxy1.provider.com:8000',
'user:pass@proxy2.provider.com:8000',
# ... more proxies
]
DOWNLOADER_MIDDLEWARES = {
'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}
ROTATING_PROXY_PAGE_RETRY_TIMES = 5
Ethical Scraping Guidelines
- Obey robots.txt: Always.
ROBOTSTXT_OBEY = Truein Scrapy does this automatically. - Rate limit: Don’t send more than 1 request/second to small sites. Use
DOWNLOAD_DELAY. - Identify your bot: Set a descriptive User-Agent including a contact URL.
- Cache aggressively: Don’t re-scrape unchanged content. Use Scrapy’s HTTP cache.
- Off-peak hours: Schedule scraping during low-traffic periods (2–6 AM local time for the target).
- Check terms of service: Some sites prohibit scraping. Review before starting.
Getting Started
For Scrapy-based HTML scraping, a 1–2 GB Ubuntu VPS at VPS.DO handles multiple concurrent spiders. For Playwright (headless Chromium), each browser instance uses 200–500 MB RAM — a 4 GB VPS runs 4–6 simultaneous browser sessions. USA VPS placement minimizes latency to US-hosted target sites; Hong Kong VPS reduces latency for APAC targets.
Conclusion
A VPS running Scrapy and Playwright provides 24/7 data collection with persistent storage, scheduled execution, and a stable IP separate from personal browsing. PostgreSQL stores scraped data with conflict handling for incremental updates, HTTP caching prevents re-scraping unchanged pages, and rotating proxies handle sites with strict bot detection. Always respect robots.txt, rate-limit aggressively, and review terms of service before scraping any website.
