llms.txt — What It Is and Why It MattersFree llms.txt Generator — Build Your AI-Readable Site Index in Seconds
llms.txt is a plain-text file placed at your domain root that tells AI language models and crawlers what your site covers, which pages are most important, and which AI agents are permitted to index and cite your content. It is the fastest single technical change an SEO professional can make to improve AI visibility — and this tool generates a correctly formatted file from your domain in under 30 seconds.
Unlike robots.txt (which controls crawl access) or sitemap.xml (which lists URLs for search engines), llms.txt is written specifically for LLM crawlers — GPTBot, PerplexityBot, ClaudeBot, Grok-Bot, and Google-Extended. It includes a BLUF description of what your site is, a curated list of key pages with annotations, and explicit per-bot access declarations.
2×+
Citation lift observed
What Is llms.txt — and How Does It Work?
llms.txt is an emerging specification for AI-readable site configuration files, proposed in late 2024 and now supported by major AI engines including OpenAI's GPTBot and Anthropic's ClaudeBot. The file sits at yourdomain.com/llms.txt — the same location as robots.txt and sitemap.xml.
When an AI crawler visits your domain, it checks for llms.txt before indexing content. A well-formed file tells the crawler your site's category, a BLUF summary of what you do, the most important pages to index, and whether each crawler agent is allowed to read and use your content. This reduces misclassification (where AI engines incorrectly categorise your site), improves topical authority scoring, and speeds up indexing of your key pages.
# Notion Cue
> Notion Cue is a free AEO platform helping SEO professionals track and improve brand visibility across ChatGPT, Perplexity, Gemini, Grok, Copilot and Claude.
## Key Pages
- [AI Visibility Checker](https://notioncue.com/ai-visibility-tool): Free AEO score across 6 LLMs
- [AEO Guide](https://notioncue.com/aeo-guide): Complete Answer Engine Optimisation guide
- [Blog](https://notioncue.com/blog): AEO research and strategy
## AI Crawler Hints
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Grok-Bot
Allow: /
llms.txt vs robots.txt vs sitemap.xml — Which Does What
Each file serves a different purpose in the AI crawling pipeline. All three are needed for complete AEO coverage:
llms.txt — Intent & categorisation
- Tells AI engines what your site covers and how to cite it
- Provides a BLUF description for LLM classification
- Lists key pages with annotations for priority indexing
- Declares per-bot access intentions (reinforces robots.txt)
- Designed specifically for LLM crawlers, not search bots
robots.txt — Crawl access control
- Controls whether a bot can crawl your pages at all
- Must explicitly allow GPTBot, PerplexityBot, ClaudeBot
- Takes precedence over llms.txt when there's a conflict
- Supported universally — older and more binding standard
- Does not tell AI what your site is about
Generate robots.txt →How to Deploy llms.txt — Step by Step
Once you've generated your file above, deployment takes under 5 minutes on any platform:
01
Generate the file
Paste your domain above and click Generate. The tool fetches your live homepage, writes a BLUF description, infers your key pages, and builds the file with correct syntax.
02
Download and review
Click Download to save llms.txt. Open it in a text editor and verify the key pages list includes your most important URLs. Add or remove pages as needed — the format is plain text.
03
Upload to domain root
Place the file at yourdomain.com/llms.txt — the same directory as robots.txt. In Next.js: drop it in /public/. In WordPress: upload via File Manager to the root. In Webflow: use the Assets panel or Netlify deploy.
04
Validate the live file
Run the llms.txt Validator to confirm the file is live, correctly formatted, and that robots.txt doesn't conflict with your bot access declarations.
Run validator →05
Check your AEO score
Run the AI Visibility Checker 2–4 weeks after deployment. The Technical tab shows llms.txt status and the score impact. Perplexity typically reflects the change within days.
Run AEO scan →Generated your file? Validate it live to confirm it's correctly deployed and all 8 AI bots can read it.
Validate live →What a Well-Formed llms.txt File Includes
Required elements
- H1 title — your brand name as the first line heading
- Blockquote BLUF — a one-sentence description in
> quote format - Key pages list — markdown links to your 5–15 most important pages with annotations
- Bot declarations — per-agent User-agent / Allow blocks for each major AI crawler
Optional elements
- Optional section — secondary pages, changelogs, API docs, support resources
- Contact field — email or contact URL for AI engine operators
- License declaration — content usage terms (CC BY, proprietary, etc.)
- Language field — for multilingual sites serving multiple markets
File size guidelines
- Standard llms.txt: keep under 5KB for most LLM context windows
- llms-full.txt: can be larger — used for comprehensive page indexes
- Avoid duplicating your full sitemap — AI crawlers don't need every URL
- Prioritise quality of page annotations over quantity of URLs listed
BLUF description tips
- One sentence, under 30 words
- State what you do, who you help, and the primary outcome
- Avoid marketing language — write for a machine, not a reader
- Include your primary category keyword for classification accuracy
Deploying llms.txt by Platform
Next.js
Place llms.txt in /public/. It will be served at /llms.txt automatically by Next.js static file serving. No configuration needed.
WordPress
Upload via Appearance → Theme Editor → root directory, or use the Yoast/Rank Math file manager. Alternatively FTP to /public_html/llms.txt.
Webflow
Go to Project Settings → Assets. Upload llms.txt, then set the URL to /llms.txt via the custom file manager in the Publishing panel.
Shopify
Add via Online Store → Themes → Edit Code → Assets. Or use a custom app to serve the file from the root path.
Vercel
Place in /public/llms.txt in your repo — it deploys automatically. Verify at your domain root after the next deploy.
Netlify
Place in the /public or root directory before build. Add a _redirects rule if you need to serve from a non-standard path.
Related AEO Tools
llms.txt is one layer of technical AEO. These tools cover the rest:
Further Reading
→llms.txt: the complete implementation guide for 2026→Why 94% of AI-cited pages have structured data→robots.txt for AI crawlers: what changed in 2025→The technical AEO checklist: 12 fixes before you publishFrequently Asked Questions
What is llms.txt?
llms.txt is a plain text file placed at the root of your domain (yourdomain.com/llms.txt) that tells AI language models and crawlers what your site covers, how to navigate it, and which AI agents are permitted to access your content. It is the AI equivalent of sitemap.xml — a machine-readable index designed specifically for LLM crawlers rather than search engine bots.
Why does llms.txt improve AI visibility?
Without llms.txt, AI crawlers have to infer your site's purpose from your homepage and whatever pages they happen to crawl. With a correctly formatted llms.txt, you explicitly tell GPTBot, PerplexityBot, ClaudeBot, and Grok-Bot your site's category, key pages, and BLUF description. This reduces misclassification, improves topical authority scoring, and gives AI engines more confidence when citing you in answers.
Is llms.txt an official standard?
llms.txt is an emerging community specification proposed in late 2024, not yet an official W3C or IETF standard. However, major AI engines including OpenAI (GPTBot) and Anthropic (ClaudeBot) have acknowledged awareness of the spec, and the pattern of placing machine-readable intent files at domain roots is well-established (robots.txt, humans.txt, security.txt). Sites with a correctly formatted llms.txt consistently show higher citation rates than comparable sites without one.
What's the difference between llms.txt and llms-full.txt?
llms.txt is the standard file — a concise index of your key pages with a BLUF description and bot access declarations. It should cover your 5–15 most important pages and stay well under 5KB so it fits comfortably in LLM context windows. llms-full.txt is an extended version for sites with large content libraries — it includes secondary pages, API documentation, changelogs, and additional resources. Most sites only need the standard file.
Does the generator read my actual site content?
Yes — the generator fetches your live homepage before generating the file. It uses your actual page title, meta description, and visible content to write a BLUF description and infer your key pages. If your homepage can't be fetched (blocked, requires login), the generator falls back to domain-based inference and tells you clearly which mode was used.
Where exactly do I place the llms.txt file?
At the root of your domain: yourdomain.com/llms.txt — the same level as robots.txt and sitemap.xml. In Next.js, place it in the /public directory. In most CMS platforms (WordPress, Webflow, Shopify), you can upload it via your file manager or CDN. The file must be accessible at the root path without redirects for AI crawlers to pick it up correctly.
What should I include in the AI crawler hints section?
Explicit User-agent blocks for each major AI crawler: GPTBot (ChatGPT), PerplexityBot, ClaudeBot (Anthropic), Grok-Bot (xAI), and Google-Extended (Gemini). Each block should have an Allow: / directive. This is especially important if your robots.txt uses wildcard Disallow rules that might be catching AI crawlers — the explicit Allow in llms.txt reinforces your intent, but robots.txt must also allow these bots for them to crawl successfully.
How quickly does adding llms.txt affect AI citations?
Perplexity indexes new content in near real-time — improvements can appear within days. Google AI Overviews typically take 1–4 weeks. ChatGPT works from training snapshots, so impact there is gradual and may take weeks to months. For fastest results, combine llms.txt with FAQPage schema markup, a BLUF page opening, and confirmed AI bot access in robots.txt.
Ready to complete your technical AEO setup?
llms.txt + robots.txt + FAQPage schema covers the three fastest technical wins for AI citation improvement.
Check your AEO score →