Lab451 Docs

Quick Start

Getting started with Lab451 takes about 30 seconds. Here's the fastest path from zero to all four files:

1. Single URL — all four file types in one click.
   Enter your URL, click Select All, then click Generate. Lab451 queues all four file types and runs them in sequence. You can download them individually or as a zip.
2. Single URL — one file type at a time.
   Enter your URL, choose llms.txt, llms-full.txt, sitemap.xml, or robots.txt, then click Generate. Repeat for each additional file you want. Download or zip them together when you're done.
3. Multiple URLs in one session.
   Enter your first URL, pick a file type, click Generate. Then enter the next URL and repeat. Note: When outputs span more than one site, Download All (zip) is disabled to prevent filename collisions — download each file individually instead.
4. Upload the files to your domain's root: yourdomain.com/llms.txt, etc.

Uploading Files

All four files should live at the root of your domain — the same directory as your homepage. Here's where each file goes:

yourdomain.com/llms.txt
yourdomain.com/llms-full.txt
yourdomain.com/sitemap.xml
yourdomain.com/robots.txt

Most web hosts let you upload via FTP, SFTP, or a file manager in the control panel. If you're on a platform like Webflow or Squarespace, check their custom file hosting documentation. If you're using a static site generator (11ty, Next.js, Hugo, Gatsby, etc.), drop the files into your public/ or static/ directory. For WordPress, upload them via FTP or your hosting panel's file manager to the root public_html/ folder.

llms.txt — File Spec

Lab451 generates llms.txt following the llmstxt.org specification. The file is Markdown-formatted and structured as follows:

# Site Name

> One-sentence description of the site.

Extended description explaining the site's purpose,
target audience, and main content areas.

## Main Sections

- [Section Title](URL): Brief description
- [Section Title](URL): Brief description

## Key Topics

- Topic 1
- Topic 2
- Topic 3

## Guidelines for AI

Instructions for how AI models should use this content.

Lab451 generates this structure automatically by analyzing your site's content, navigation, and metadata.

llms-full.txt — File Spec

llms-full.txt extends the standard with the full cleaned text of every indexed page. It's used by RAG pipelines and AI systems that retrieve document content at query time.

# Site Name — Full Content Index

## [Page Title](URL)

> Meta description

Full page content, cleaned and stripped of navigation,
footers, and boilerplate. Markdown formatting preserved
where possible.

---

## [Next Page](URL)

[content continues...]

Large sites should be aware that llms-full.txt can grow quite large. Lab451 applies smart deduplication and content filtering to keep file sizes reasonable.

sitemap.xml — File Spec

Generated according to the sitemaps.org protocol. Includes <loc>, <lastmod>, <changefreq>, and <priority> for every discovered page.

robots.txt — File Spec

Generated according to RFC 9309. Lab451 does not write a generic template — it inspects your site first and tailors the file.

• Platform detection. Lab451 fingerprints your homepage and adds rules for WordPress, WooCommerce, Shopify, Drupal, Joomla, Magento, BigCommerce, Ghost, Squarespace, Wix, Webflow, Next.js, or Nuxt.
• Verified sitemaps. Candidate sitemap URLs are fetched and confirmed to return XML before being listed, so you never advertise a broken sitemap. Sitemap lines in your existing robots.txt are preserved.
• Trusted crawlers allow-listed. Sixteen search engines and twenty-five AI crawlers get explicit groups, split into retrieval bots (which cite you) and training bots (which do not).
• Scrapers blocked. Twenty commercial backlink and lead-generation crawlers are denied.
• System clutter hidden. Config files, backups, logs, auth paths, and tracking-parameter URLs are disallowed, while CSS and JS stay crawlable so Google renders your pages correctly.