A new category of website audit emerged in early 2026: the agent-readiness score. Unlike a traditional SEO audit, which checks whether your site ranks well and reads clearly to a human visitor, an agent-readiness audit asks a different question entirely — can an autonomous AI agent discover what your site offers, authenticate against your systems, call your tools, and in some cases transact with you, all without a human clicking through a browser session?
These audits typically score a domain from 0 to 100 across five different categories and roughly sixteen individual checks, and most sites tested land somewhere around 20 to 30 out of 100 — a tier best described as "bot-aware": technically reachable by crawlers, but structurally unprepared for the agentic web that is arriving faster than most technical teams have planned for.
The underlying shift driving this new audit category is architectural. AI models increasingly do not read your website directly the way a human browser does. An agent runtime fetches your page, parses it, and decides what to execute — including whether to bother running your JavaScript at all. The runtime is the new gatekeeper, sitting between your content and whatever model eventually acts on it. This guide walks through every category and check that a modern agent-readiness audit tests, in a logical order, with the underlying standard, why it matters, and exact working implementation for each one. Where a standard is still an early-stage draft rather than a settled specification, that is called out explicitly, because implementation effort should be proportional to how mature a standard actually is.
What Does an Overall Agent-Ready Score Actually Measure?
A typical agent-readiness audit aggregates five weighted categories into a single 0-100 score: Discoverability, Content, Bot Access Control, API/Auth/MCP/Skill Discovery, and an optional Commerce category for sites that sell products or services. Each category contains individual pass/fail or partial-credit checks, and the categories are rarely weighted equally — API/Auth/MCP/Skill Discovery is typically the heaviest single category, reflecting where the agentic web is moving fastest.
Score tiers commonly range from "bot-aware" at the low end, meaning a site is reachable but not structured for agents, up through progressively more capable tiers as more checks pass. A score around 29, which is common for a well-built but agent-unoptimized modern SaaS site, typically reflects strong marks on basic discoverability (robots.txt, sitemap) and content readability, combined with near-total absence across the API, auth, and commerce discovery layers most sites have never heard of, let alone implemented.
Category 1: Discoverability — Can an Agent Even Find Your Site's Rules?
This category checks the most foundational layer: whether an agent, before doing anything else, can retrieve the basic machine-readable documents that describe what it is and is not allowed to do on your domain.
Check: robots.txt With Current Rules
Goal: Publish robots.txt with current, unambiguous rules covering AI crawlers specifically. This is the most basic check in the entire audit, and it is also the one most sites get subtly wrong — duplicate user-agent blocks from multiple sources, an empty Allow directive, or a platform-managed rule silently overriding a manually written one. The fix is a single, clean robots.txt with one entry per user-agent, no duplicates, explicitly naming GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, and Google-Extended rather than relying solely on a wildcard User-agent: * block, since AI-specific crawlers increasingly warrant AI-specific rules distinct from general web crawler policy.
Check: Sitemap Referenced From robots.txt
Goal: Publish a sitemap and reference it from robots.txt with a valid, current structure. How to implement: add a Sitemap: directive pointing to your sitemap.xml at the bottom of robots.txt. This is standard technical SEO hygiene that most sites already have correctly configured, but it is worth explicitly verifying the sitemap is current, contains only canonical URLs, and does not silently 404 due to a CMS migration or caching issue nobody has checked in months.
Check: Link Headers for Agent Discovery
Goal: Include Link response headers for agent discovery, per RFC 9264. How to implement: add Link response headers on your homepage that point agents toward machine-readable resources — an API catalog, a webfinger endpoint, an ai.txt manifest. This is typically a server or edge-rule configuration change rather than an application code change:
Link: ; rel="mcp-server",
; rel="api-catalog" Check: DNS for AI Discovery (DNS-AID)
Goal: Publish DNS AI Discovery records for DNS-based agent discovery. What it is: DNS-AID is an active IETF Internet-Draft that standardizes publishing AI agents in DNS itself, so that other agents can discover them through the same distributed, cached, federated infrastructure that has resolved domain names for three decades. Governance of the reference implementation has moved to an open-source foundation with backing from several major DNS infrastructure providers.
How discovery actually works: the specification defines three progressively broader lookup modes. A direct lookup by name works when the requester already knows both the organization and the specific agent. A capability search works when the requester knows the organization but not the specific agent — "does acme.example have any agent that does fraud detection?" And a domain-wide crawl of an agent index works when the requester knows only the required capability, not the organization at all. Records use SVCB (Service Binding) resource records, the same modern DNS record type browsers already use for HTTPS service discovery, with DNSSEC and DANE TLSA records layered on top for cryptographic trust.
How to implement a basic record using the deterministic naming pattern _<agent-name>._<protocol>._agents.<your-domain>:
_prompt-tracker._mcp._agents.notioncue.com. 3600 IN SVCB 1 agent.notioncue.com. (
alpn="mcp"
port=443
cap="https://notioncue.com/.well-known/mcp.json"
)An honest caveat worth stating clearly: DNS-AID is a draft standard, not yet a ratified RFC, and real-world adoption outside a small set of infrastructure operators remains early. Global DNSSEC adoption is also still low enough that a domain publishing DNS-AID records without properly maintained DNSSEC is publishing a discovery mechanism without the trust layer it depends on — worth implementing directionally, not worth over-investing engineering time in until the specification stabilizes further.
Category 2: Content — Can an Agent Actually Read What You Publish?
This category has a single check with an outsized practical impact on every other category.
Check: Markdown Negotiation via the Accept Header
Goal: Return HTML as clean Markdown when an agent explicitly requests it via content negotiation. How to implement: your server should return a Markdown version of a page when the request's Accept header specifies text/markdown, while continuing to serve standard HTML by default to regular browsers.
This proposal builds on a decades-old, entirely standard HTTP feature — content negotiation via the Accept header — applied to a new use case. The logic is straightforward: an agent parsing a page for its semantic content, not its visual layout, gains almost nothing from wading through navigation markup, ad containers, and styling classes to extract the actual substance. A clean Markdown response, requested explicitly via content negotiation, gives the agent exactly the content it needs with none of the parsing overhead. Most sites score zero on this check today, which makes it one of the highest-leverage, lowest-competition implementations available — a self-contained edge function that detects the Accept header and serves a converted Markdown response does not touch your main application logic:
// Edge function: serve Markdown when requested via Accept header
export default {
async fetch(request, env) {
const accept = request.headers.get('Accept') || '';
const response = await fetch(request);
if (accept.includes('text/markdown')) {
const html = await response.text();
const markdown = convertHtmlToMarkdown(html); // your conversion function
return new Response(markdown, {
headers: { 'Content-Type': 'text/markdown; charset=utf-8' }
});
}
return response;
}
};Category 3: Bot Access Control — Are You Actually Letting the Right Bots In?
This category checks whether your access-control configuration distinguishes between crawlers and agent categories with appropriate granularity, rather than treating all automated traffic as a single undifferentiated block.
Check: AI Bot Rules in robots.txt
Goal: Add explicit User-agent rules covering the major named AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, anthropic-ai, and meta-externalagent among them. This overlaps with the Discoverability robots.txt check but scores specifically for AI-crawler-name coverage rather than mere presence of the file — a site can pass the basic robots.txt check while still failing this one if it only ever addressed User-agent: * and never named a single AI crawler explicitly.
Check: Content Signals in robots.txt
Goal: Declare AI content usage preferences with Content Signals directives in robots.txt. Content Signals is an emerging directive format that lets an operator declare per-use permissions distinct from crawl access itself: whether content may be used for search indexing, for real-time AI grounding, and for model training, each independently. Implementation is a single line added to your existing robots.txt:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /This declaration format is explicitly framed, in the directive text itself, as an express reservation of rights under Article 4 of the EU's Digital Single Market copyright directive — meaning a website operator setting ai-train=no is making a legally meaningful assertion under EU law specifically about text-and-data-mining opt-out, not merely a polite technical request. Choose your values deliberately rather than copying a default: allowing ai-input (live retrieval for citation and grounding) while restricting ai-train is a coherent, common position for a publisher who wants AI visibility without unrestricted training-data harvesting.
Check: Signed-Request Bot Authentication
Goal: Let your site identify itself as a legitimate bot or agent using cryptographic request signing, and be able to verify incoming agent requests the same way. How to implement: publish a JSON Web Key Set at a well-known path so your site can cryptographically identify itself when it sends outbound bot or agent requests, and so receiving systems can verify those requests genuinely originate from your domain.
This class of check is often marked informational rather than strictly scored, and the reason matters: this is an active area of IETF draft standardization applying HTTP Message Signatures (RFC 9421) to automated traffic identity. Each bot operator generates a signing keypair, publishes the public key as a JSON Web Key Set at a standard well-known path, and signs every outbound request with a header naming the domain to verify against. Support for verifying these signatures is spreading across major content delivery and edge-security providers, and some AI platforms are actively experimenting with using it for their own crawler identity. The relevant IETF working group's own milestones target standards-track publication in 2026 — real momentum, but still pre-ratification.
The practical guidance for most site operators: if your hosting or edge-security provider offers a "verified bots" feature built on this kind of signature verification, enabling it lets you selectively trust cryptographically-verified agent traffic over easily-spoofed User-Agent strings alone. Publishing your own signing directory — proving your site's own outbound requests, if any, are genuine — matters primarily if your site itself operates agents making requests to other sites, which is a smaller subset of implementers than the receiving-and-verifying side.
Category 4: API, Auth, MCP & Skill Discovery — The Heaviest Category, and Where Almost Every Site Fails
Six checks live here, and this is where the widest gap tends to show up between what sites have implemented and what an agentic future actually requires.
Check: Machine-Readable API Catalog per RFC 9727
Goal: Publish an API catalog for automated API discovery, per RFC 9727. How to implement: create a well-known catalog returning an application/linkset+json array. Each entry should include an "anchor" URL for the API and link relations for service-desc (an OpenAPI spec), service-doc (documentation), and status (a health or uptime endpoint):
{
"linkset": [
{
"anchor": "https://notioncue.com/api/v1",
"service-desc": [{ "href": "https://notioncue.com/api/v1/openapi.json" }],
"service-doc": [{ "href": "https://notioncue.com/docs/api" }],
"status": [{ "href": "https://notioncue.com/api/v1/health" }]
}
]
}Check: OAuth/OIDC Discovery Metadata
Goal: Publish OAuth/OIDC discovery metadata so agents can authenticate with your APIs. How to implement: if your API has protected endpoints, publish a well-known OAuth authorization server configuration (for OAuth 2.0) or OpenID Connect configuration so agents can programmatically discover your authorization endpoints, token endpoints, scopes, and grant types, per RFC 8414. Without this, an agent attempting to authenticate against your API has to be manually hardcoded with your specific endpoint URLs rather than discovering them automatically:
// GET /.well-known/oauth-authorization-server
{
"issuer": "https://notioncue.com",
"authorization_endpoint": "https://notioncue.com/oauth/authorize",
"token_endpoint": "https://notioncue.com/oauth/token",
"scopes_supported": ["read:citations", "read:prompts", "write:prompts"],
"response_types_supported": ["code"],
"grant_types_supported": ["authorization_code", "refresh_token"]
}Check: OAuth Protected Resource Metadata
Goal: Publish OAuth Protected Resource Metadata so agents can discover how to authenticate against a specific resource. How to implement: your resource server should identify which authorization server(s) can issue tokens for it, and which scopes it supports. This is the counterpart to the authorization-server metadata above, defined by RFC 9728, and it lives at a URL the resource server itself controls rather than the authorization server:
// GET /.well-known/oauth-protected-resource
{
"resource": "https://notioncue.com/api/v1",
"authorization_servers": ["https://notioncue.com"],
"scopes_supported": ["read:citations", "read:prompts"],
"bearer_methods_supported": ["header"]
}Check: Auth.md Agent Registration Metadata
Goal: Publish plain-language agent registration instructions. How to implement: serve an auth.md file at your domain root or under /.well-known/ with agent registration instructions — how to register a client, which authentication flows and client types you support, and where to find your scopes and claims documentation. This is a newer, human-and-agent-readable Markdown convention — deliberately simpler than the structured JSON metadata formats above — intended as a fallback an agent can parse via natural-language understanding when the structured OAuth metadata endpoints are absent or incomplete.
Check: MCP Server Card
Goal: Publish a Model Context Protocol Server Card for agent discovery. How to implement: serve a server card document at a well-known path (the canonical location has consolidated around /.well-known/mcp.json, though older implementations used a longer path and both may still appear in the wild during the transition). This is one of the fastest-maturing standards on this list. The Model Context Protocol has seen extremely rapid adoption — reaching tens of millions of monthly SDK downloads and more than ten thousand active public servers within roughly a year of its release — and governance of the core specification has since moved to a vendor-neutral open-source foundation with backing from most major AI platform providers.
If you run an MCP server, publish a card advertising its transport, endpoint, and capabilities without requiring a full connection handshake just to discover basic metadata:
// GET /.well-known/mcp.json
{
"name": "NotionCue MCP Server",
"description": "Query AI citation data, run prompt checks, and manage tracked prompts programmatically.",
"version": "1.2.0",
"serverUrl": "https://mcp.notioncue.com/mcp",
"transport": "streamable-http",
"auth": { "type": "oauth2" },
"tools": [
{ "name": "get_citation_status", "description": "Check citation status for a tracked prompt" },
{ "name": "run_prompt_check", "description": "Run an on-demand prompt check across all five AI engines" }
]
}Even if you do not currently run an MCP server, this check functions as a forward-looking readiness signal — several major AI development tools and desktop assistants natively probe this well-known endpoint before offering a one-click connection to any domain a user points them at, so its absence is a missed integration opportunity as much as a missed audit point.
Check: Agent Skills Index
Goal: Publish an agent skills discovery index listing discrete, documented capabilities an agent can invoke on your domain. How to implement: publish a skills index at a well-known path, with each entry including a name, type, description, and the location of a corresponding detailed skill-definition file:
// GET /.well-known/agent-skills-index.json
{
"version": "2.0.0",
"skills": [
{
"name": "citation-audit",
"type": "analysis",
"description": "Audit a domain's AI citation readiness across five engines",
"skillPath": "/skills/citation-audit/SKILL.md"
},
{
"name": "prompt-tracking-setup",
"type": "configuration",
"description": "Configure weekly prompt tracking for a new domain",
"skillPath": "/skills/prompt-tracking-setup/SKILL.md"
}
]
}Check: In-Browser Agent Tool Registration
Goal: Expose site tools directly to AI agents operating within the same browser session. How to implement: use an emerging browser JavaScript API to register callable tools directly on the page. Each tool needs a name, description, an input schema (JSON Schema), and an async execute function that either returns JSON or throws a structured error.
This capability is being developed through a W3C community group process — currently a draft report rather than a ratified standard, and available in some browsers only behind an experimental trial as of mid-2026. Unlike every other check in this category, which concerns server-side discovery documents, this is a browser-native API: it lets your page register callable tools directly in the browser's model-context runtime while the user is actively viewing the page, so an agent operating within that same browser session can invoke your page's own functions — booking a meeting, submitting a contact form, running a search — without needing a separate server-side connection at all:
// Registers tools directly in the browser session
if ('modelContext' in navigator) {
navigator.modelContext.provideContext({
tools: [
{
name: 'check_citation_rate',
description: 'Look up this domain\'s current AI citation rate for a given prompt',
inputSchema: {
type: 'object',
properties: { prompt: { type: 'string' } },
required: ['prompt']
},
execute: async ({ prompt }) => {
const res = await fetch('/api/citation-check', {
method: 'POST',
body: JSON.stringify({ prompt })
});
return res.json();
}
}
]
});
}Because it is browser-native and requires no server infrastructure beyond the page's existing JavaScript, this is genuinely one of the fastest checks on this entire list to implement for a modern JavaScript-heavy site — a few dozen lines added to an existing page — even though its pre-standard status means production reliance should be paired with a graceful fallback for browsers or agent runtimes that do not yet support it.
Category 5: Commerce — Can an Agent Actually Buy From You?
This category only applies to, and is only scored for, sites that sell products or services. It covers the agentic commerce protocol landscape at a level of technical depth most merchant-facing content has not yet reached.
Check: HTTP-Native Micropayment Protocol Support
Goal: Support machine-native HTTP payments for API-level, agent-to-agent transactions. How to implement: add payment middleware to your API routes that revives the long-dormant HTTP 402 "Payment Required" status code specifically for machine-to-machine stablecoin micropayments — an API endpoint returns 402 with payment terms, the requesting agent's wallet signs an authorization, and the request retries with payment attached, all within a single HTTP round trip with no human checkout flow involved:
import { paymentMiddleware } from 'agent-payment-middleware';
app.use('/api/premium-data', paymentMiddleware({
price: '$0.001',
network: 'base',
payTo: '0xYourWalletAddress'
}));This class of protocol is the correct choice specifically for API-call-level, sub-dollar, machine-to-machine payments — a price-comparison API charging a fraction of a cent per lookup, for instance — not for consumer retail checkout, which the remaining commerce checks address instead. Adoption has grown quickly across multiple chains since a major protocol version launched in late 2025, with tens of millions of agent transactions already processed industry-wide by mid-2026.
Check: Machine Payment Session Protocol Support
Goal: Support session-based machine payment protocols for sustained agent billing relationships. How to implement: publish payment discovery metadata via a well-known endpoint that exposes price and payment requirements, using middleware compatible with the emerging machine-payment-session model.
This category of protocol lets an agent pre-authorize a spending limit once and then stream many small, granular payments continuously within that session in either stablecoins or fiat, without a separate on-chain transaction per individual interaction — a meaningfully different model from per-request micropayment settlement, better suited to sustained, ongoing machine-to-machine billing relationships rather than one-off microtransactions. A major mainnet supporting this model launched in the first quarter of 2026 with more than a hundred integrated services at launch.
Check: Universal Commerce Catalog Protocol
Goal: Support content payments via a universal, cross-surface commerce catalog protocol. How to implement: serve a well-known profile document with your protocol version, capabilities, and endpoints, and ensure your product schema includes the relevant commerce-protocol metadata annotations.
This class of open standard exists to solve the "N x N integration bottleneck" — the problem of every merchant needing a bespoke integration for every different AI shopping surface. A merchant that publishes compliant catalog metadata once becomes discoverable and purchasable across every consuming surface simultaneously. These protocols typically rely on a separate agent-payments-authorization standard for the actual payment mandate, while the catalog protocol itself standardizes discovery, cart construction, and the merchant checkout journey rather than the money-movement layer. One such protocol, launched publicly at a major retail industry event in January 2026, was co-developed with several large ecommerce platforms and endorsed by more than twenty additional payment and retail organizations.
Check: Agentic Checkout Protocol
Goal: Support agentic checkout discovery to be visible in commerce agent runtimes. How to implement: publish a well-known checkout-protocol document or an HTTP header pointing to your service origin metadata, expose an OpenAPI reference, and specify your supported checkout flows.
This class of protocol defines a multi-actor flow — buyer, agent, merchant, and payment service provider — around a scoped, single-use payment token minted by the buyer's wallet provider, letting an agent complete checkout without the buyer's actual card details ever passing through the agent itself. This standardizes the checkout journey specifically, and it is the one area where genuine overlap exists with the universal commerce catalog category above; most large retailers integrate support for both in parallel rather than betting on one, following the same logic that drove multi-channel strategies during the early mobile-commerce era.
How Do You Prioritise Fixing All of This?
Sixteen checks is a lot to action at once, and not every organisation needs every check equally. A practical prioritisation, based on effort-to-impact ratio rather than raw check count:
Do this week, regardless of business type: fix robots.txt duplication and add explicit AI crawler rules; add a Content-Signal directive with a deliberate policy; publish a current sitemap referenced from robots.txt; verify no platform-managed rule is silently overriding your intended configuration.
Do this month, if you run any API surface: publish the RFC 9727 API catalog; add OAuth/OIDC discovery metadata if your API requires authentication; publish an MCP Server Card even at a minimal level of detail, since several major AI development tools probe for it automatically.
Do this quarter, if you are JavaScript-heavy or content-focused: implement Markdown content negotiation via the Accept header, since this is currently scored at near-zero across the vast majority of sites and represents genuinely uncontested competitive ground; evaluate in-browser agent tool registration for your highest-value interactive page flows.
Evaluate but do not rush, given draft-standard status: DNS-based agent discovery (still pre-RFC, low real-world adoption outside a small set of infrastructure operators); signed-request bot authentication for your own outbound traffic (a "verified bots" toggle on the receiving side, however, is worth enabling immediately since that requires no publishing on your part); the full commerce protocol stack, which should be driven by actual product and payments team bandwidth rather than SEO-team urgency, given the genuine engineering complexity of wallet integration and payment-provider compliance underneath each one.
How NotionCue Helps You Track Progress Across This Entire Checklist
Most of the checks in this guide are one-time or infrequent implementation work — publish a file, add a header, configure a middleware — rather than ongoing content production. What they share with everything else in this series is the same underlying requirement: an AI crawler or agent runtime has to actually be able to reach and correctly parse whatever you publish, which is precisely the layer the NotionCue AI Crawler Audit verifies. After implementing any of the discovery documents in this guide — your MCP Server Card, your API catalog, your OAuth metadata — running the Crawler Audit confirms the file is reachable in the exact server-rendered form an agent runtime receives, not just correctly formatted according to a validator that never checks real-world crawler access.
The NotionCue llms.txt Generator complements the newer discovery standards covered here directly: llms.txt remains one of the most broadly supported, lowest-friction discovery documents across the agentic web, and building it alongside your MCP Server Card and Agent Skills index gives an agent runtime multiple, mutually-reinforcing entry points into understanding what your domain offers, rather than depending on any single still-maturing standard working perfectly on its own.
Start your free NotionCue trial and run an agent-readiness self-check against your own domain this week alongside the NotionCue AI Crawler Audit — the checklist above tells you which discovery documents you are missing, and the Crawler Audit confirms the ones you do publish are actually reaching the crawlers and agent runtimes you built them for.
Google's own official generative-AI-search guidance, published in May 2026, explicitly states that llms.txt, content chunking, and AI-specific markup are not required for generative AI search visibility specifically — a useful, calibrating counterpoint to this entire guide. The checklist covered here is about agent readiness broadly, including commerce, authentication, and tool-calling scenarios that sit outside pure search visibility. Do not conflate the two: a site can be excellently optimised for AI Overview and ChatGPT citation, covered throughout the rest of this series, while still scoring low on an agent-readiness audit, because that audit is testing a different, broader, and in several cases still-emerging layer of the agentic web.
Frequently Asked Questions About Agent-Ready Websites
Do I need to implement every single check to be considered agent-ready?
No. The commerce category alone only applies to transactional businesses, and several checks — DNS-based agent discovery, signed-request bot authentication, in-browser tool registration — remain pre-standard enough that early implementation is a forward-looking investment rather than an urgent gap. Prioritise the checks with ratified or near-ratified standards behind them (robots.txt, sitemap, RFC 9727 API catalog, RFC 8414/9728 OAuth metadata, MCP Server Cards) before investing heavily in the newest draft-stage protocols.
Will a high agent-readiness score improve my AI citation rate in ChatGPT or Perplexity?
Indirectly and partially. This checklist covers a broader agentic-web readiness layer — authentication, tool-calling, commerce — that is largely distinct from the citation-and-content-structure work covered throughout the rest of this series, which more directly drives AI Overview, Perplexity, and ChatGPT citation rate. A site can score well here while still needing the BLUF structure, schema, and E-E-A-T work covered elsewhere to actually earn citations. Think of this checklist as infrastructure for a different, adjacent use case: autonomous agents transacting and calling tools on your domain, not language models citing your content in a generated answer.
How often should I re-run this audit?
Quarterly is a reasonable cadence given how actively these standards are still evolving — several of the checks covered in this guide, including DNS-based agent discovery and in-browser tool registration, were still in active draft revision through the first half of 2026, meaning the correct implementation details can shift meaningfully between quarters. A CMS migration, hosting provider change, or major framework upgrade is also worth triggering an immediate re-check, since several of these checks (robots.txt, Link headers, Markdown negotiation) are exactly the kind of configuration that silently breaks during infrastructure changes.