Choose what crawlers may access, decide which AI bots to allow, add your sitemap, and copy a valid robots.txt. Paste any path to check whether it's blocked.
# robots.txt generated with pyalm.com/free-tools/robots-txt-generator User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://www.example.com/sitemap.xml
Pyalm builds SEO-first websites for businesses in India and the UAE — technical SEO, structured data, fast hosting and the right DNS and email setup from day one.
robots.txt tells well-behaved crawlers which paths not to fetch. It is not access control: blocked URLs can still be indexed if linked from elsewhere, and bad bots ignore it. To keep a page out of Google, allow crawling and use a noindex tag instead.
Bots such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), CCBot (Common Crawl) and PerplexityBot each have their own user-agent. Blocking them controls whether your content is used for AI training or answers — a business decision with trade-offs for visibility in AI search.
📖 Read the full guide: Robots.txt Guide: Rules, Common Mistakes and How to Block AI Crawlers
Turn a list of URLs into a valid XML sitemap with lastmod, changefreq and priority.
Preview how your title and meta description look on Google — desktop and mobile, with pixel widths.
Check who a domain is registered with, when it expires and its nameservers — via official RDAP data.
At the root of each host: https://www.example.com/robots.txt. A robots.txt on www doesn't apply to a subdomain like shop.example.com.
No. Google-Extended only controls use of your content for Gemini and Vertex AI. Normal Search crawling and ranking use Googlebot.
Google uses the most specific (longest) matching rule; if they're equal length, Allow wins. The tester applies the same logic.