Infinite scroll is a user experience decision that quietly became an indexing decision. The pattern loads a first batch of items, then fetches more as the visitor scrolls. For a person browsing on a phone it feels smooth. For a crawler that does not execute JavaScript, page one is the entire archive.
Everything past that first batch is reachable only by triggering a scroll event that no crawler is going to trigger. The posts exist. They have URLs. Nothing links to them in a way a parser can follow.
The Real Failure Is Discovery, Not Ranking
Worth being precise about what breaks here. An individual article at /blog/some-post is perfectly crawlable if something points at it. The archive page is not the problem in itself.
The problem is that for most sites the archive is the primary internal path to older content. Kill that path and older posts are orphaned. They sit in the sitemap, which is a discovery hint rather than an endorsement, with no internal links carrying any signal about whether they matter. The internal linking guide covers why that matters for how engines weigh a page.
So the content is not penalised. It is unreferenced, which in practice looks similar from the outside and has a completely different fix.
Numbered Pagination Still Works Fine
Old fashioned numbered pagination with real anchor tags is the boring correct answer. Page one links to page two, page two links to page three, a crawler walks the chain.
The requirement is that these are genuine links in the HTML:
<a href="/blog/page/2">Next</a>Not a button with a click handler. Not a div styled to look like a link. If the href is missing, there is no path.
Google retired support for rel="next" and rel="prev" as pagination hints years ago, and no AI engine has picked them up since. They are harmless to keep and they are not doing anything for you.
Hybrid Is the Version Most Teams Should Ship
You do not have to choose between smooth scrolling and crawlability. Render real paginated links in the HTML, then progressively enhance with JavaScript so visitors get the scroll behaviour.
Crawler receives page one with a working link to page two. Visitor receives infinite scroll. Same markup, different experience depending on whether JavaScript runs. This is the same server side rendering principle covered in the SSR versus CSR guide, applied to one specific component.
A "load more" button is the middle ground and it depends entirely on implementation. If the button is an anchor tag pointing at the real page two URL, and JavaScript intercepts the click to load in place, that works. If it is a button element with no href, it is infinite scroll with extra steps.
Deep Archives Need a Shorter Path
Even correct numbered pagination has a depth problem. Forty pages of archive means page forty is forty hops from the homepage. Crawlers allocate finite attention per site, and pages buried that deep get visited rarely or never.
Category and tag pages fix this by creating shortcuts. A post from three years ago that sits on page thirty eight of the main archive might sit on page two of its category archive. That is a much shorter path.
This only works if the taxonomy is real. Thirty tags each containing one post creates thirty thin pages and no useful shortcuts. A handful of genuine categories with meaningful membership does the job. The topical authority guide covers the clustering logic behind that.
Faceted Navigation Creates the Opposite Problem
Ecommerce sites usually have the inverse issue. Filters for colour, size, price, and brand combine into an enormous number of URLs, most of which are near duplicates of each other.
That is the duplication problem from the canonicalization guide at scale. Filtered views should carry a canonical pointing at the unfiltered category, unless a specific filtered combination genuinely deserves to be its own indexed page because people search for it directly.
Blocking filter parameters in robots.txt is the common instinct and it is usually wrong. Blocked URLs cannot be crawled, so the canonical tag on them is never read, so the consolidation never happens. Let crawlers reach them and let the canonical do the work.
What I Would Check First
Disable JavaScript in your browser and load your blog archive. Whatever you see is roughly what a non rendering crawler sees. If the page shows ten posts and no way forward, that is the whole story.
Then pick a post from two years ago and work out how many clicks separate it from your homepage using only real links. More than four or five and it is effectively hidden regardless of pagination style.
Confirming It
The NotionCue AI Crawler Audit reports what specific crawlers receive rather than what a browser renders, which is the distinction that matters for this particular failure. Archive and category paths are worth pointing it at directly, because they are the paths teams almost never test.
Start your free NotionCue trial and audit your archive URL, not just your top articles. The articles are usually fine. The route to them is where this breaks.
Thirty second version of this audit: open your archive in a private window with JavaScript off and try to reach page three. If you cannot, no crawler can either.
Common Questions
Does having a sitemap solve this?
Partly. A sitemap gets URLs discovered but carries no signal about relative importance, and it does not replace the internal link graph. Sitemaps and internal links do different jobs, as covered in the sitemap guide.
Should I put every archive page in the sitemap?
No. List the individual posts. Paginated archive URLs are navigation, not destinations, and filling a sitemap with them dilutes it.
Is infinite scroll ever fine?
On a social feed or anything where individual items are not meant to be independently discoverable, yes. For a blog archive or product listing where each item has its own URL you want found, no.