Enter a website to load its robots.txt, see whether it blocks anything important, and test any path for Googlebot, Bingbot or AI crawlers. You can edit the rules to try changes first.
Pyalm builds SEO-first websites for businesses in India and the UAE — technical SEO, structured data, fast hosting and the right DNS and email setup from day one.
Whether robots.txt exists and returns 200, is served as text/plain and under Google's 500 KiB limit, whether Googlebot is blocked from the site, whether it lists your sitemap, and directives Google doesn't support such as noindex and crawl-delay.
A missing robots.txt (404) simply means everything may be crawled. A robots.txt that returns a server error (5xx) is worse: Google may pause crawling the whole site until it can read the file.
📖 Read the full guide: Robots.txt Guide: Rules, Common Mistakes and How to Block AI Crawlers
Build a robots.txt with crawler rules, AI-bot blocking and sitemap — then test any URL against it.
Find a website's XML sitemap automatically and validate it: URLs, lastmod, limits and broken entries.
Check any website's meta tags, canonical, robots.txt and XML sitemap in one go, with a score and a fix list.
Find a page's canonical URL and check it's valid, absolute, in the <head> and points to a live page.
Enter the page URL. The checker fetches the site's robots.txt and tests that exact path for Googlebot, showing the rule that matched.
Google uses the most specific user-agent group and the longest matching rule; on a tie, Allow wins. The tester follows the same logic.
Use the robots.txt generator to build one with crawler and AI-bot rules, then come back to test it.