robots.txt is a plain text file at the root of your site that tells crawlers which paths they may fetch. It’s simple, but one wrong line can stop Google from crawling an entire site. It’s also widely misunderstood: robots.txt controls crawling, not indexing, and it is not a security feature.

Where it lives

Exactly at https://example.com/robots.txt, one file per host and protocol. https://blog.example.com needs its own. It must return 200 with text/plain. If it returns 5xx, Google may pause crawling the whole site. If it returns 404, crawlers assume everything is allowed.

Basic syntax

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml
  • User-agent: which crawler the group applies to (* = all). A crawler uses the most specific group that matches its name, and ignores the others.
  • Disallow: path prefixes not to crawl. An empty Disallow: means “allow everything”.
  • Allow: exceptions inside a disallowed path. The longest matching rule wins.
  • Sitemap: absolute URL of your sitemap. It can appear anywhere, and multiple lines are allowed. See XML sitemaps.

Paths are case-sensitive and match from the start of the URL path.

Wildcards

Google and Bing support * (any characters) and $ (end of URL):

User-agent: *
Disallow: /*?sort=          # any URL with a sort parameter
Disallow: /*.pdf$           # URLs ending in .pdf
Disallow: /search           # /search, /search/, /search?q=… and /searching

Common recipes

Allow everything (the most common correct file):

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

WordPress:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Don’t block /wp-content/ or /wp-includes/: Google needs CSS and JS to render pages. See WordPress sitemaps.

Staging site (plus a password!):

User-agent: *
Disallow: /

Block AI training crawlers, but keep search:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow:

Google-Extended controls the use of your content for Google’s AI models, without affecting Googlebot search crawling. Reputable crawlers honour these rules. Abusive bots ignore robots.txt entirely, and need blocking at the server or firewall. See Cloudflare WAF rules and Nginx rate limiting.

robots.txt is not noindex

If you disallow a URL, Google won’t crawl it, but it can still index the URL (without content) if other pages link to it. To keep a page out of search results:

  • allow crawling, and add <meta name="robots" content="noindex">, or the X-Robots-Tag: noindex header;
  • don’t also block it in robots.txt, or Google never sees the noindex.

Not a security feature

robots.txt is public. Listing /secret-admin-panel/ advertises it. Protect private areas with authentication and IP restrictions. See Nginx basic auth.

Common mistakes

  • Disallow: / copied from staging to production. This is the classic way to lose all rankings. Check it after every launch.
  • Blocking CSS, JS or image directories, so Google can’t render the page.
  • Using robots.txt to remove pages from the index (use noindex or 410).
  • Expecting Crawl-delay to work for Google (Google ignores it; Bing and Yandex honour it).
  • Serving robots.txt with a 5xx error, or behind a login or WAF challenge.
  • Forgetting that each subdomain needs its own file.

Serving it with Nginx

A static file in the web root works. For per-environment control, you can return it directly:

location = /robots.txt {
    add_header Content-Type text/plain;
    return 200 "User-agent: *\nDisallow:\n\nSitemap: https://example.com/sitemap.xml\n";
}

Testing

  • Search Console → Settings → robots.txt report: shows the version Google fetched, and any errors.
  • URL Inspection tells you whether a specific URL is blocked.
  • curl -i https://example.com/robots.txt: check the status code and content.

Quick reference

Goal Rule
Allow all Disallow: (empty)
Block all Disallow: /
Block a folder Disallow: /folder/
Exception Allow: /folder/file
Parameter URLs Disallow: /*?param=
Sitemap Sitemap: https://…/sitemap.xml
Hide from search noindex, not robots.txt

Related: technical SEO on your own server and Google Search Console setup.

  • SEO
  • Web Development
  • Nginx