Docs — Lab451 | llms.txt, llms-full.txt, sitemap.xml & robots.txt Specs

Quick Start

Getting started with Lab451 takes about 30 seconds. Here's the fastest path from zero to all four files:

Go to lab451.org and paste your site URL
Select first 'file type': llms.txt, llms-full.txt, sitemap.xml or robots.txt
Wait for the crawl to complete (3 seconds to 1 min depending on size)
Repeat: Select your second file type
Preview each file in the tabbed viewer
Click Download All to grab a ZIP with all four files
Upload the files to your domain's root: yourdomain.com/llms.txt, etc.

Tip. By default you generate 1 to 4 'file types' for one site at a time. However, Lab451 also generates multiple 'file types' using multiple sites in one session. In that case, you need to download or copy each site individually.

For example:

domain.com/llms.txt
domain.com/llms-full.txt
domain.com/sitemap.xml
domain.com/robots.txt

blog.domain.com/llms.txt
blog.domain.com/robots.txt

internet.com/llms.txt
internet.com/llms-full.txt

That's it. No account needed for the free plan. Files are instantly ready for production.

Uploading Files

All four files should live at the root of your domain — the same directory as your homepage. Here's where each file goes:

yourdomain.com/llms.txt
yourdomain.com/llms-full.txt
yourdomain.com/sitemap.xml
yourdomain.com/robots.txt

Most web hosts let you upload via FTP, SFTP, or a file manager in the control panel. If you're on a platform like Webflow or Squarespace, check their custom file hosting documentation. If you're using a static site generator (11ty, Next.js, Hugo, Gatsby, etc.), drop the files into your public/ or static/ directory. For WordPress, upload them via FTP or your hosting panel's file manager to the root public_html/ folder.

llms.txt — File Spec

Lab451 generates llms.txt following the llmstxt.org specification. The file is Markdown-formatted and structured as follows:

# Site Name

> One-sentence description of the site.

Extended description explaining the site's purpose,
target audience, and main content areas.

## Main Sections

- [Section Title](URL): Brief description
- [Section Title](URL): Brief description

## Key Topics

- Topic 1
- Topic 2
- Topic 3

## Guidelines for AI

Instructions for how AI models should use this content.

Lab451 generates this structure automatically by analyzing your site's content, navigation, and metadata.

llms-full.txt — File Spec

llms-full.txt extends the standard with the full cleaned text of every indexed page. It's used by RAG pipelines and AI systems that retrieve document content at query time.

# Site Name — Full Content Index

## [Page Title](URL)

> Meta description

Full page content, cleaned and stripped of navigation,
footers, and boilerplate. Markdown formatting preserved
where possible.

---

## [Next Page](URL)

[content continues...]

Large sites should be aware that llms-full.txt can grow quite large. Lab451 applies smart deduplication and content filtering to keep file sizes reasonable.

sitemap.xml — File Spec

Generated according to the sitemaps.org protocol. Includes <loc>, <lastmod>, <changefreq>, and <priority> for every discovered page.

robots.txt — File Spec

Generated according to RFC 9309. Configured to allow all major AI crawlers while protecting common sensitive paths. Auto-references your sitemap.xml.

Lab451 Docs