Every request a crawler makes gets a three digit answer. That number is the most direct signal your server sends about what a URL is and whether it is worth coming back to.
Most technical SEO material treats status codes as a Googlebot topic. The codes mean the same things to AI crawlers, but the consequences differ, mostly because AI crawlers have less patience and less accumulated history with your site than Googlebot does.
200 Is Not Always Good News
A 200 means the request succeeded and the response contains what was asked for. Fine, except when it is a lie.
The soft 404 is the common version of this lie. A URL for deleted content returns a friendly "we could not find that page" message with a 200 status. The page says not found. The status code says found. A crawler believes the status code, indexes an empty page as valid content, and now you have a real page in the index that contains nothing.
This is worse than a 404. A 404 removes the URL from consideration cleanly. A soft 404 adds a thin, meaningless page to whatever an engine understands about your site, and enough of them drag on the overall picture of your domain in the way the content pruning guide describes.
Check for these by requesting a URL you know does not exist and looking at the actual status returned, not what the page says. Add a random string to any path on your site. If that returns 200, your error handling is broken.
301 Versus 302 Still Matters
A 301 says permanent, treat the destination as authoritative, transfer whatever the old URL had earned. A 302 says temporary, keep treating the old URL as the real one.
Using 302 for a permanent move is a common mistake and it means you are telling crawlers not to consolidate onto the new URL. The migration guide covers why this compounds badly during a move.
Chains matter more here than for Googlebot. Each hop is a request, some AI crawlers follow fewer hops before giving up, and a chain that resolves fine for Google can dead end elsewhere. One hop from old to final, always.
404 Versus 410
Both mean gone. The difference is confidence. A 404 means not found, which leaves open the possibility it comes back. A 410 means deliberately removed, permanent, stop asking.
Crawlers recheck 404s for a while on the assumption something might have been temporarily broken. A 410 gets dropped faster.
Use 410 when you have genuinely retired content and want it out of the index quickly, which is the case during the kind of deliberate pruning covered in the pruning guide. Use 404 for genuinely unexpected missing pages. Most CMS platforms only emit 404 and getting 410 requires explicit configuration, which is probably why almost nobody uses it.
503 Is the One People Get Wrong Under Pressure
A 503 means temporarily unavailable, come back later. This is the correct response during planned maintenance, and it is the code most often replaced with something worse in a hurry.
Serving a 200 with a maintenance page tells a crawler your content is now a maintenance notice. Serving a 404 tells it your pages are gone. Both do real damage if a crawl happens during the window.
Use 503, and include a Retry-After header giving an estimate in seconds or as a date. Crawlers respect it and it keeps your index state intact through the outage.
429 and What Happens After It
A 429 means too many requests, slow down. If your rate limiting is aggressive enough to throw these at AI crawlers, you are choosing to be less visible in AI search, and that may or may not be deliberate.
This is worth checking specifically because WAF and CDN rules are often configured against bot traffic generally, without anyone distinguishing between scrapers and the crawlers that feed citation. The AI crawlers guide covers the allow list side of this.
The behavior after a 429 varies by crawler and is not well documented. Some back off and return, some deprioritize the domain for a while. Neither is good, and neither is visible in any dashboard except your server logs, which is what the log file analysis guide is for.
The 403 Nobody Notices
A 403 means forbidden. When it appears in response to an AI crawler it is usually not a decision anyone made on purpose. It is a security rule catching a user agent it does not recognize.
The reason this goes unnoticed for so long is that it is invisible everywhere except logs. Your site works. Users are fine. Google is fine, because Googlebot is on every default allow list. A newer AI crawler is not, gets blocked, and nothing anywhere tells you.
I would check this before assuming any AI visibility problem is a content problem. It is a five minute check and it is a complete explanation when it is the cause.
Checking All of This
The reliable way is requesting your own URLs with specific crawler user agents and reading the status you actually get back, which is a different question from what a browser gets.
The NotionCue AI Crawler Audit does this across the major AI crawlers and reports what each one receives, which surfaces the 403 case and the soft 404 case without requiring server access or command line work.
Start your free NotionCue trial and run it against your highest value pages. The status code layer is fast to check and it is a complete blocker when it is wrong, so it is worth ruling out first.
Two minute test: add nonsense to the end of any URL on your site and check the status code in your browser's network tab. If it says 200, you have soft 404s, and every deleted page on your site is currently being indexed as a valid empty page.
Common Questions
Does a 404 hurt my site?
Not inherently. Pages get removed and 404 is the honest answer. What hurts is a large volume of 404s on URLs that are still linked internally or listed in your sitemap, which wastes crawl attention on paths you already know are dead.
Should I redirect every 404 to the homepage?
No. Redirecting unrelated content to the homepage is treated as a soft 404 by most crawlers anyway, and it makes it harder to spot genuine broken link problems. Redirect to a genuinely relevant page or let it 404.
How long should maintenance 503s last before it becomes a problem?
Short outages with a Retry-After header are handled fine. Extended periods risk crawlers treating the state as more permanent, so if maintenance will run long it is worth keeping a static version of key pages served at 200 rather than blanket 503ing the whole domain.