SEO professionals and agencies increasingly run their tools on a VPS instead of a laptop: crawls of large sites finish without tying up your computer, scheduled jobs run overnight, and teams share one always-on machine. Here’s what works well on a VPS, how to size it, and how to keep your crawling polite and legal.
What to run on an SEO VPS
| Task | Tools (examples) |
|---|---|
| Technical site crawls | Screaming Frog SEO Spider (Windows/Linux), Sitebulb (Windows), open-source crawlers (e.g. Scrapy-based) |
| Rank tracking | Self-hosted rank trackers, or scripts using official search APIs and SERP API providers |
| Log file analysis | GoAccess, Screaming Frog Log File Analyser, ELK/OpenSearch stacks |
| Core Web Vitals monitoring | Lighthouse CI, PageSpeed Insights API scripts, sitespeed.io |
| Uptime and SSL monitoring | Uptime Kuma. See Uptime Kuma |
| Automation and reporting | n8n, Python scripts (Search Console API), cron jobs. See n8n |
| Self-hosted analytics | Plausible, Umami, Matomo. See Plausible |
Windows or Linux?
- Windows VPS: run desktop SEO tools (Screaming Frog, Sitebulb, and others) over Remote Desktop, exactly as on your PC. Windows needs a plan with at least 50 GB of disk on VPS.DO. See Windows VPS.
- Linux VPS: best for scripts, scheduled jobs, Lighthouse CI, log analysis and web-based tools. Screaming Frog also has a Linux version, with a headless command-line mode for scheduled crawls.
# example: headless Screaming Frog crawl on Linux (licence required for large crawls)
screamingfrogseospider --crawl https://example.com --headless --save-crawl \
--output-folder /srv/crawls/example --export-tabs "Internal:All"Sizing
Crawlers keep large amounts of URL data in RAM (or in a database storage mode):
| Workload | Suggested VPS |
|---|---|
| Crawls up to ~100k URLs, scripts, monitoring | 2 vCPU, 4–8 GB RAM |
| Crawls of 500k+ URLs, several projects | 4 vCPU, 16 GB RAM, more disk |
| Log analysis stacks (OpenSearch) | 4+ vCPU, 16 GB+ RAM |
Screaming Frog’s database storage mode writes crawl data to disk, letting you crawl far more URLs with less RAM. SSD storage helps. See how much RAM a VPS needs.
Testing from different locations
A VPS in another country lets you see what users there see: geo-redirects, localized content, CDN behaviour, hreflang handling, and real latency. Hosting servers in the USA, Hong Kong and Japan, for example, covers major markets. See choosing a VPS location.
Crawl responsibly
- Your own sites and clients’ sites: throttle crawl speed during business hours, so you don’t slow their servers.
- Third-party sites: respect
robots.txt, terms of service and rate limits. Use official APIs where available (Search Console, PageSpeed Insights, and SERP data providers). - Don’t use a VPS for scraping that violates terms of service, or for anything that looks like an attack (aggressive concurrency, ignoring 429 responses). Providers act on abuse reports. See the VPS.DO legal policies.
- Identify your crawler with a descriptive User-Agent where appropriate.
Security for a shared team machine
- Separate user accounts per team member (Linux), or per-user RDP accounts (Windows).
- Keep RDP and SSH protected: strong passwords or keys, non-default ports, firewall allowlists, or a VPN. See change the RDP port and Tailscale.
- Store API keys (Search Console, PageSpeed, SERP APIs) securely, not in shared spreadsheets. See secrets management.
- Back up crawl configurations and reports. See backup strategy.
Automate recurring reports
Schedule a weekly crawl and export with cron (Linux) or Task Scheduler (Windows), then push results to Google Sheets, a dashboard, or email with n8n or a Python script. See crontab.
Quick reference
| Need | Setup |
|---|---|
| Desktop SEO tools | Windows VPS + RDP |
| Scheduled crawls and scripts | Linux VPS, headless crawler, cron |
| Big crawls | More RAM, or database storage mode |
| Geo testing | Servers in target countries |
| Rules | robots.txt, rate limits, official APIs |
Related: web scraping on a VPS and Windows Server initial setup.