SEO tools guess how search engines see your site. Your server access logs record it. Every request Googlebot makes is there: which URLs, how often, what status codes it got, and how long the server took. On your own VPS you have full access to these logs, which is a real advantage over shared hosting.

What logs reveal

  • Which sections Googlebot crawls most, and which it ignores.
  • Errors bots hit (404s, 5xx) that users may never report.
  • Crawl wasted on parameters, redirects, and junk URLs.
  • Whether new or updated pages get crawled, and how fast.
  • Slow responses that throttle crawling.
  • Orphan pages: URLs crawled but not linked from your site, or important pages never crawled.

Log format

The default Nginx “combined” format:

66.249.66.1 - - [04/Nov/2026:10:12:01 +0000] "GET /blog/install-nginx-ubuntu/ HTTP/1.1" 200 18234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Add the response time ($request_time) for performance analysis:

log_format seo '$remote_addr [$time_local] "$request" $status $body_bytes_sent '
               '"$http_referer" "$http_user_agent" $request_time';
access_log /var/log/nginx/access.log seo;

Keep enough history (at least 30 days) in logrotate. See Nginx logs and logrotate.

Verify real Googlebot

Anyone can fake the user agent. Verify with reverse and forward DNS:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The name must end in googlebot.com, google.com or googleusercontent.com, and resolve back to the same IP. Google also publishes its crawler IP ranges as JSON. Bingbot verifies the same way (search.msn.com).

Quick analysis with the command line

LOG=/var/log/nginx/access.log
# Googlebot requests per day
grep -h Googlebot $LOG* | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

# Status codes Googlebot received
grep Googlebot $LOG | awk '{print $9}' | sort | uniq -c | sort -rn

# Most crawled URLs
grep Googlebot $LOG | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

# 404s and 5xx served to Googlebot
grep Googlebot $LOG | awk '$9 ~ /^(404|5..)$/ {print $9, $7}' | sort | uniq -c | sort -rn | head

# Crawl spent on parameter URLs
grep Googlebot $LOG | awk '{print $7}' | grep -c '?'

For compressed rotated logs, use zgrep. See grep and awk.

GoAccess for a visual report

sudo apt install goaccess
grep Googlebot /var/log/nginx/access.log | goaccess --log-format=COMBINED -o googlebot.html -

You get top URLs, status codes, and requests over time in an HTML report. For larger sites, dedicated SEO log analyzers (Screaming Frog Log File Analyser and others) combine logs with crawl data.

What to look for

Finding Action
Many 404s on old URLs Add 301s to equivalents, or 410 for removed content. See 301 vs 302 redirects
5xx errors or timeouts Fix server capacity and application errors. Google slows crawling when it sees them
Crawl on parameters, filters, calendars Canonicals, robots.txt rules, fewer crawlable links to them
Redirect chains being crawled Point links and redirects straight to the final URL
Important pages rarely crawled Improve internal links, include them in the sitemap
Slow average response time for bots Caching, database tuning
Fake Googlebots hammering the site Rate-limit or block by IP. See Nginx rate limiting

Compare with Search Console

Search Console → Settings → Crawl stats shows Google’s own view: requests per day, response codes, average response time, and file types. Logs give you the URL-level detail behind those numbers. See Google Search Console setup.

After a migration

Logs are the fastest way to confirm Googlebot is following your redirects and crawling the new URLs. Check daily for the first weeks: old URLs should return 301 or 410, and new URLs should show up with 200s. See website migration SEO.

Privacy

Access logs contain visitors’ IP addresses. Limit retention, restrict access, and anonymize them if you keep them for long periods or export them to other tools.

Quick reference

Task Command / tool
Bot hits per day grep Googlebot … | awk … | uniq -c
Status codes awk '{print $9}' | sort | uniq -c
Verify Googlebot host IP, then host name
Visual report GoAccess
Google’s summary Search Console Crawl stats

Related: technical SEO on your own server and log management.

  • SEO
  • Nginx
  • Logs