robots.txt Checker
Lab451 reads robots.txt first, exactly as crawlers do, then the sitemap.xml, llms.txt, llms-full.txt and tdmrep.json it governs. See which AI crawlers are let in, check every package and link inside the files, and find anything hidden from human readers.
Free for 3 new sites a day. Sign in to automate daily monitoring and reporting.
robots.txt decides which agents read the rest, so it's checked first. Agents keep coming back to these files, so keep them up to date.
Which AI crawlers you allow or block, rules that block every crawler, lines parsers skip, and Sitemap: lines pointing at sitemaps that no longer exist. Allowed crawlers are tracked over time, so a plugin update that quietly blocks ClaudeBot shows up.
Whether it parses, URLs on the wrong host, a sample of entries that now 404, and lastmod dates stamped identically on every page.
Format, links that return 404, package names nobody has registered, dead hostnames, and instructions hidden in comments that a Markdown preview never shows you.
The same reference and hidden-content checks across the full text, which is where install commands tend to go stale.
Whether your text-and-data-mining reservation is written in a form that counts. A malformed file is the same as no file.
It never changes your site, and a file it couldn't read is reported as unchecked rather than clean. Missing files? The generator builds a visibility-first robots.txt and the three files it points agents to, verified.
Monitoring
A plugin update or CDN setting can rewrite robots.txt overnight and shut out the agents your sitemap and llms.txt were written for. Monitoring runs Checker on your domains 6 times a day, keeps every version of every file, emails you the moment a critical issue appears, and sends a weekly report of what changed. Included with Pro (20 domains) and Agency (320 domains).
robots.txt, llms.txt, llms-full.txt, sitemap.xml and tdmrep.json, every 4 hours, with the same checks as the free checker.
A hidden instruction or an unregistered package is emailed the first time it appears. It isn't repeated on every check.
What changed line by line, which issues are new or fixed, and which checks couldn't complete. An unreadable file is never reported as clean.
Every version of every file is kept for 180 days, including which AI crawlers robots.txt allowed on each date.
Checker FAQ
The files crawlers and AI agents request from your domain, in the order they do: /robots.txt first, then /sitemap.xml, /llms.txt, /llms-full.txt and /.well-known/tdmrep.json. It reads them from their standard locations, exactly as a crawler would see them.
Enter your domain. The report shows which AI crawlers your robots.txt allows and which it blocks, flags rules that block every crawler, and lists lines parsers will skip. A block on an agent also hides your sitemap.xml, llms.txt and llms-full.txt from it, so those are worth fixing first.
Crawlers that follow robots.txt check it before fetching any other file on your domain. If an agent is disallowed, it won't read your llms.txt or sitemap either. robots.txt is also where the Sitemap: line tells search engines where your sitemap lives; the checker confirms that sitemap actually exists.
CMS and SEO plugin updates, theme changes, hosting migrations and CDN settings that manage robots.txt for you can all rewrite it. Monitoring keeps every version and shows which AI crawlers were allowed on each date, so a quiet change shows up.
No. The checker only reads public files. A file it could not read is reported as unchecked, never as clean.
Monitoring runs the same checks on your domains 6 times a day, emails you the first time a critical issue appears, and sends a weekly report of what changed. It is included with Pro (20 domains) and Agency (320 domains). See pricing.