Robots.txt Guide: Rules, Common Mistakes and How to Block AI Crawlers
How robots.txt works, the syntax and matching rules Google uses, mistakes that de-index sites, and the user-agents for GPTBot, ClaudeBot, Google-Extended and other AI bots.
Editorial disclosure: Pyalm publishes and maintains the products and free tools discussed on this site. Regulatory statements are linked to primary sources where applicable. No independent professional review is claimed unless a reviewer is explicitly named.
robots.txt is a plain text file at the root of your site that tells crawlers which paths they shouldn't fetch. It's simple, powerful and — when it goes wrong — capable of quietly removing a whole site from Google.
The basics
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xml- User-agent — which crawler the group applies to (
*= all). - Disallow — paths not to crawl.
Disallow:(empty) allows everything;Disallow: /blocks everything. - Allow — exceptions inside a disallowed path.
- Sitemap — the full URL of your sitemap.
matches any characters;$anchors the end (/.pdf$).
How Google picks a rule
- It uses the most specific user-agent group that matches (e.g.
Googlebotover*). - Within that group, the longest matching path wins.
- If an Allow and a Disallow are equally long, Allow wins.
robots.txt is not a privacy tool
- Blocked pages can still appear in search (as a bare URL) if other sites link to them.
- Malicious bots ignore it — and it advertises your "hidden" paths.
- To keep a page out of Google, allow crawling and add
<meta name="robots" content="noindex">. - To protect content, use authentication.
AI crawlers
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Browsing on a user's request |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Claude search |
| Google-Extended | Gemini / Vertex AI training (doesn't affect Search) | |
| Applebot-Extended | Apple | AI training |
| PerplexityBot | Perplexity | Answer engine index |
| CCBot | Common Crawl | Open dataset used by many models |
Blocking training bots keeps your content out of future models; blocking search/answer bots can reduce your visibility in AI-powered search. Many businesses block training crawlers but allow search ones. It's a business decision — there's no universally right answer.
Build and test yours
The free Robots.txt Generator & Tester lets you block common private areas, pick which AI crawlers to block, add custom rules and your sitemap — then test any URL against the rules for Googlebot or any AI bot, using Google's matching logic. You can also paste an existing robots.txt to test it.
Mistakes that hurt
Disallow: /left over from a staging site — the classic de-indexing disaster- Blocking CSS and JS files, so Google can't render your pages
- Blocking pages you've also marked
noindex(Google can't see the noindex) - Putting robots.txt anywhere except the root of each host
SEO and infrastructure, built in
Pyalm builds SEO-first websites for businesses in India and the UAE — technical SEO, structured data, sitemaps, correct DNS and email authentication, and fast hosting — so these checks pass from day one.
Related free tools
WHOIS Lookup | DNS Lookup | Sitemap Generator | Robots.txt Generator | SERP Preview | All free tools