Lab451 Docs

Quick Start

Getting started with Lab451 takes about 30 seconds. Here's the fastest path from zero to all four files:

1. Single URL — all four file types in one click.
   Enter your URL, click Select All, then click Generate. Lab451 crawls your site once and builds all four files from that crawl, with robots.txt pointing at the sitemap it just made. You can download them individually or as a zip.
2. Single URL — one file type at a time.
   Enter your URL, choose llms.txt, llms-full.txt, sitemap.xml, or robots.txt, then click Generate. Repeat for each additional file you want. Download or zip them together when you're done.
3. Multiple URLs in one session.
   Enter your first URL, pick a file type, click Generate. Then enter the next URL and repeat. Note: When outputs span more than one site, Download All (zip) is disabled to prevent filename collisions — download each file individually instead.
4. Upload the files to your domain's root: yourdomain.com/llms.txt, etc.

Uploading Files

All four files should live at the root of your domain — the same directory as your homepage. Here's where each file goes:

yourdomain.com/llms.txt
yourdomain.com/llms-full.txt
yourdomain.com/sitemap.xml
yourdomain.com/robots.txt

Most web hosts let you upload via FTP, SFTP, or a file manager in the control panel. If you're on a platform like Webflow or Squarespace, check their custom file hosting documentation. If you're using a static site generator (11ty, Next.js, Hugo, Gatsby, etc.), drop the files into your public/ or static/ directory. For WordPress, upload them via FTP or your hosting panel's file manager to the root public_html/ folder.

llms.txt — File Spec

Lab451 generates llms.txt following the llmstxt.org specification. The file is Markdown-formatted and structured as follows:

<!-- Advisory header: reference material, not instructions -->
# example.com

> Site name — key pages and resources.

This llms.txt file lists the most important pages on example.com …

## Main Pages

- [Home](https://example.com/): Page title
- [Pricing](https://example.com/pricing): Page title

## Blog

- [First Post](https://example.com/blog/first-post): Page title

## Optional

- [Full Documentation](https://example.com/llms-full.txt): …
- [XML Sitemap](https://example.com/sitemap.xml): …
- [Robots.txt](https://example.com/robots.txt): …

Lab451 builds this from a crawl of your site. Top-level pages go under Main Pages; deeper pages are grouped by their first path segment, so /blog/… becomes a Blog section. Each link carries the page's own <title>. The advisory header is an HTML comment, so Markdown viewers don't display it.

llms-full.txt — File Spec

llms-full.txt extends the standard with the full cleaned text of every indexed page. It's used by RAG pipelines and AI systems that retrieve document content at query time.

<!-- Advisory header -->
# example.com

> Site name — complete site content for LLM ingestion.

## Main Pages

### Pricing
Source: https://example.com/pricing
Last-Modified: Tue, 03 Nov 2026 09:12:00 GMT
Retrieved: 2026-11-10

Visible text of the page …

---

### Next page
…

Pages are grouped the same way as in llms.txt. Each one records its source URL, the Last-Modified date when your server sends one, and the date Lab451 read it. Only text a human visitor can see is included (see Hidden content), each page is included once, and very large pages are truncated rather than dropped. Expect the file to grow with your site.

sitemap.xml — File Spec

Generated according to the sitemaps.org protocol. Every discovered page gets <loc>, <changefreq> and <priority>.

<lastmod> is included only when Lab451 finds the page's real modification date: the page's own metadata first (article:modified_time, JSON-LD dateModified and similar), then a believable Last-Modified header, then the publication date. If none exists, lastmod is left out rather than guessed. A source that stamps the same day on nearly every page is a build date, not an edit date, and is ignored. Search engines learn to distrust lastmod from sites that stamp today's date on every URL.

robots.txt — File Spec

Generated according to RFC 9309. Lab451 does not write a generic template — it inspects your site first and tailors the file.

• Platform detection. Lab451 fingerprints your homepage and adds rules for WordPress, WooCommerce, Shopify, Drupal, Joomla, Magento, BigCommerce, Ghost, Squarespace, Wix, Webflow, Next.js, or Nuxt.
• Verified sitemaps. Candidate sitemap URLs are fetched and confirmed to return XML before being listed, so you never advertise a broken sitemap. Sitemap lines in your existing robots.txt are preserved.
• Trusted crawlers allow-listed. Sixteen search engines and twenty-five AI crawlers get explicit groups, split into retrieval bots (which cite you) and training bots (which do not).
• Scrapers blocked. Twenty commercial backlink and lead-generation crawlers are denied.
• System clutter hidden. Config files, backups, logs, auth paths, and tracking-parameter URLs are disallowed, while CSS and JS stay crawlable so Google renders your pages correctly.

AI training preference — how to

One robots.txt group, proposed in an IETF draft, tells every participating AI provider whether your pages may be used to train their models. Anthropic, Microsoft, Google and OpenAI have said they will honour it; Microsoft starts on 1 February 2027. Lab451 can write it for you, check it, and tell you when it changes. It is optional: if you do nothing, your robots.txt states no preference, which is how things worked before the draft. Background, rollout dates and catches →

What it looks like

User-agent: __usage_ai-training
Disallow: /
Allow: /press/

That reads: "don't train on this site, except the pages under /press/". The name is exact: two underscores, then usage_ai-training. A near-miss spelling is treated as an unknown crawler and states nothing.

Set it in the generator

1. Enter your domain and select robots.txt, or Select All. The AI training preference box appears below the buttons.
2. Choose one of three options.
   • No preference (default): nothing is written.
   • Opt out of AI training: the whole site.
   • Opt out, except some sections: the whole site, minus sections you pick.
3. For "except", pick the sections from the dropdown: /blog/, /news/, /press/, /docs/, /about/, /faq/ or /products/. Each one you pick appears as a chip; click × to remove it. Pick only sections your site has: /blog/ means pages at yoursite.com/blog/…
4. Need another section? Choose Custom path…, type it and press Add (or Enter). press-kit, /press-kit/ and a pasted https://yoursite.com/press-kit/ all become /press-kit/. Paths with spaces, # or accented characters are refused with a message, never dropped silently.
5. Read the line under the chips before you generate, e.g. "Opted out except /blog/, /press/". If nothing is chosen it warns that the whole site will be opted out.
6. Click Generate. The success message confirms what was written, e.g. "AI training opt-out (IETF draft) included, except /press/."

What Lab451 writes

Two groups change, and they always agree with each other:

• The shared group, User-agent: __usage_ai-training, labelled in the file as the IETF draft.
• The per-crawler training group (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and ten others) is set to the same Disallow and Allow lines. Under the draft a crawler's own group wins over the shared one, so leaving those on Allow: / would cancel your opt-out for every crawler named.

Search engines, AI search agents, link-preview bots and the default * group are not touched.

Adding it by hand

Already maintain your own robots.txt? Add the group anywhere in the file, as its own block:

# AI training preference (IETF draft)
User-agent: __usage_ai-training
Disallow: /

• Check your named training crawlers. If GPTBot, ClaudeBot or Google-Extended have a group with Allow: /, change it to match, or the shared opt-out won't apply to them.
• User-agent: * doesn't count. The wildcard group never expresses a training preference, whatever it says.
• It looks forward only. It applies to pages fetched while it is published. Models already trained are unaffected.

Check and monitor your training preference

The Checker reads the shared group in your published robots.txt and reports it on the robots.txt row, e.g. "Shared AI training preference (IETF draft): opted out of AI training except 1 path". It also raises these findings:

• Misspelled token → Medium (Low if a correct group is also present). The misspelled group states nothing, so what you meant isn't in effect.
• Per-crawler rules contradict the opt-out → Low. A named training crawler allows everything while the shared group opts out. That crawler's own group wins.
• Google-Extended side effect → Info. You opted out and have no Google-Extended group, so Gemini grounding is affected too.
• Training crawlers refused one by one → Info. You already block two or more training crawlers by name; the shared group would reach providers you haven't listed.
• Your current preference → Info. A choice, not a problem, so it never turns a clean report into one with issues.

Monitoring alerts

On Pro and Agency, Monitoring emails you the moment the preference changes, rather than waiting for the weekly report: "AI training preference changed on yoursite.com", with the before, the after, and the new lines. The preference that counts is the one published when a page is fetched, so a plugin, CMS update or CDN setting that quietly rewrites it matters on the day it happens.

• One email per change, using your immediate-alert setting under Alerts on the monitoring page.
• A check that couldn't read robots.txt (blocked, timed out) is skipped. The next good read is compared with the last good one, so you get neither a false alert nor a missed change.
• The weekly report and the change history list it too, e.g. "AI training preference: no AI training preference → opted out of AI training (whole site)".

Why we validate and verify

Nobody reads a file written for agents. It gets fetched over HTTPS from your domain and treated as authoritative because it came from you. That's the whole point of the format — and the whole problem. Here's what we do about it.

In 2026, researchers scanned thousands of live llms.txt files across corporate domains and found hundreds of references to packages that had never been registered. Not typos exactly — correctly spelled names for things that didn't exist. Anyone could register them. In a controlled test, doing so got code running inside a large company's network in about four minutes. One real case involved a documented npx command for a binary that only shipped inside a scoped package; npx resolved the bare name against the public registry instead, where somebody else had already claimed it.

None of that requires anyone to attack your site. It just requires a stale command in your docs and an agent willing to run it.

What we check

Seven registries, every hostname, every deploy target. Findings graded critical to info, each with the page it came from. Before your files are handed over, every reference in them gets resolved.

Package names

We pull install commands out of your content — npm, npx, pnpm, yarn, bun, pip, uv, poetry, pipx, cargo, gem, bundle, dotnet, composer and go get — and check each name against its registry: npm, PyPI, crates.io, RubyGems, NuGet, Packagist and the Go module proxy.

• Unregistered name → Critical. The name is free for anyone to claim. This is the finding that matters most.
• Registered but suspicious → High. Days old, one version, no linked repository, almost no downloads. We need at least two of those signals before we say anything, because mature packages shouldn't trip this.
• Unscoped executor → High. A bare npx name sitting next to a scoped @org/package — the pattern behind the incident above.

Extraction is done with parsers and pattern matching, never by a language model. That's deliberate: a summariser asked to paraphrase install steps will eventually invent a package name, which is precisely the bug this exists to catch. Nothing reaches your file unless it appears verbatim on your page.

Hostnames and deploy targets

Every hostname we find gets resolved. A domain that no longer resolves is flagged, because a lapsed registration is a link anyone can inherit. We also look for CNAMEs pointing at unclaimed deployments on Vercel, Netlify, Render, Fly, GitHub Pages, Heroku, S3, Azure and others — the subdomain-takeover pattern, where someone else can serve content from your subdomain.

Placeholder domains reserved for documentation (example.com, .test, .invalid) are skipped. They're supposed not to resolve.

When a registry is unreachable

A timeout or a rate-limit response is recorded as inconclusive, never as "missing". If enough lookups fail we tell you the run was incomplete rather than accusing your packages of not existing.

Hidden content and prompt injection

Plain text, no rendering step, no human in the loop. We remove what your visitors can't see — then scan it separately and tell you if it was shaped like an instruction.

That's the second problem with agent-facing files. Any instruction-shaped sentence that survives into the file is something an agent may act on — and the easiest place to hide one is text your visitors can't see.

So we don't just strip tags. We parse the page and remove what a human wouldn't see:

• display:none, visibility:hidden, zero opacity, zero font size, off-screen positioning and clipped elements
• aria-hidden, the hidden attribute, and screen-reader-only classes
• HTML comments, <script>, <template> and <noscript> contents
• Zero-width characters, bidi overrides, and the Unicode tag block — which can encode a complete message that renders as literally nothing

What we remove gets scanned separately. Text that's both hidden and instruction-shaped — "ignore previous instructions", "don't tell the user", pipe-to-shell commands, exfiltration phrasing — is reported as Critical, because nobody hides prose from their own readers by accident. If that shows up on your site, treat it as a possible compromise of the page or its CMS and check recent edits, plugins and third-party scripts.

The same phrasing sitting in visible text is reported as Medium and worded as a question, not an accusation. Plenty of legitimate documentation talks to agents on purpose.

We also skip user-generated paths by default — comments, forums, reviews, profiles, search results — since that's the cheapest place for someone to plant text without touching your CMS.

How we treat your site

Lab451 identifies itself in the User-Agent header with one of two names, depending on what it is doing:

• LLMSCrawler/1.0 when generating files: the crawl, and the robots.txt builder's platform and sitemap checks. Full string: Mozilla/5.0 (compatible; LLMSCrawler/1.0; +https://lab451.org)
• Lab451-Checker/1.0 when the Checker or Monitoring reads your published files. Full string: Mozilla/5.0 (compatible; Lab451-Checker/1.0; +https://lab451.org/checker)

Package registries and DNS are queried separately (as Lab451-Validator/1.0); that traffic never goes to your site.

The crawler requests pages a few at a time, with timeouts, a page budget and an overall time cap. It stays on the domain you entered, follows only ordinary links (not ones marked rel="nofollow"), and fetches web pages only, not images, PDFs or downloads. It never visits:

• sign-in and account paths: /login, /logout, /signup, /register, /verify, /callback, password reset pages, /members/
• admin and back-end paths: /admin/, /wp-admin/, /wp-login.php, /api/, webhooks, /cgi-bin/
• user-generated pages such as comments, forums, reviews, profiles and search results (see Hidden content)

It does not apply your robots.txt rules to its own crawl. It reads the public pages a visitor could reach by following links, so the generated files describe your whole public site. If a section shouldn't appear in them, remove its lines before you publish.

On our side, every request target is resolved and checked before we connect, and the connection is pinned to the address we checked so it can't be swapped mid-flight. Redirects are followed one hop at a time with the same check applied to each. Oversized pages are truncated rather than dropped, so a heavy page still contributes what it can instead of vanishing.

If your site sits behind Cloudflare, Akamai or similar and returns 403 to unknown user agents, we'll tell you that's what happened rather than pretending the page is private. Allow-listing LLMSCrawler (for the generator) and Lab451-Checker (for the Checker and Monitoring), or generating from a staging copy, both work.

Your data

• Free and guest use. We record the domain and the date for daily limits. Your generated file content isn't stored.
• Pro. Generated files and their reference reports are saved to your history so you can come back to them. Delete your account and they go with it.
• We only read what's public. No authentication, no cookies, no logins. If a page needs a session to view, we can't see it.
• Reports are yours. If we find something serious on your site, we tell you. We don't publish findings tied to a named domain.

Found something? Tell us

If you spot a security issue in Lab451 itself, email [email protected] with enough detail to reproduce it. We'll confirm we got it, keep you posted while we fix it, and credit your account if necessary. Please don't run automated scans against the live service — Instead, ask us and we'll test accordingly.

If Lab451 flags something on your site and you think it's wrong, send us the URL and the findings. False positives are worth more to us than a quiet complaint — every one we hear about gets turned into a test case.