SquishPixel Blog · SEO · AI search · Site security

Robots.txt guide covers how to manage search and AI crawlers

Search Engine Journal has published a guide to robots.txt, the plain text file in a site's root directory that tells crawlers which URLs they may crawl. It says AI search tools and agents now visit sites alongside traditional crawlers, so site owners need to know which rules each one follows. Blocking the wrong bot can keep content out of large language model answers or search results, while leaving needless URLs open can waste crawl budget on large sites.

The guide explains the basics. Robots.txt supports only two wildcards: the asterisk, which matches zero or more characters, and the dollar sign, which matches the end of a URL. Path matching is case-sensitive. A bot with its own group of rules follows that group instead of the rules under User-agent: *. Within a group, Google applies the most specific rule, and if rules are equally specific it picks the least restrictive one.

It then walks through common uses: internal search pages, faceted navigation (filter and sort URLs), PDFs, form directories, account pages, scripts that do not affect rendering, and sitemap lines. It warns that blocked pages can still be indexed from links, so noindex is the tool for removal, and the page must stay crawlable for Google to read it. It also says private content needs authentication, not robots.txt.

On AI bots, the guide advises separate rules for training, search and user-requested crawlers. For example, a site can allow OAI-SearchBot and ChatGPT-User while blocking GPTBot. Google-Extended and Applebot-Extended control how content is used, not whether it is crawled. The guide says OpenAI states robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Anthropic says its bots honor it. For testing, it points to the robots.txt report in Google Search Console, URL Inspection and Google's open-source parser.

Why it matters

A robots.txt file is short, but small mistakes can have large effects. A rule missing a trailing slash can block pages you want indexed, and blocking a script can hide content from Google. Blocking retrieval bots can also make it harder for AI tools to find and cite your pages, which is a different decision from opting out of training.

The guide also shows the limits of the file. Bots that ignore it can still collect your content, and some user-requested AI visits may bypass it. If you need real enforcement, the article says to use authentication, server or CDN controls, or a firewall, and to check your server logs to see what actually happens.

Source: Search Engine Journal. This brief summarises their reporting; read the original for the full detail.