1. The site that passed every audit and still hid its content
I have worked on catalog sites where the technical audit came back clean. Crawl errors close to zero. A valid sitemap, submitted and accepted. Canonicals pointing at the right URLs. Fast server response. No accidental noindex. Good mobile rendering. And still, in Search Console, a slice of the long tail sat there with almost no impressions. Not because the pages were bad. Because search engines had barely found them.
The pages were live, and a shopper could reach them. Land on a category, open a filter, press «Show more» a few times, click a card. Every step worked fine for a human. The problem is that a human clicks and a crawler does not.
That is the subject of this post. On a large catalog, crawlability is an architecture problem, not a tooling problem. The audit tools tell you what they can see, and they see a clean site. If a page is only reachable through a button, a filter or a long chain of clicks, it effectively does not exist for search engines, however polished the technical SEO looks.
2. Why bots don’t click
Google’s guidance on links is one sentence long in its important part. Generally, Google can only crawl a link if it is an anchor element with an href attribute that holds a URL. Most links in other formats are not parsed by its crawlers. In practice, this means one thing. If the next page of results is loaded by a script that responds to a click, there is no URL in the markup, so there is nothing for the crawler to follow.
Read that against the way most modern catalogs work. A «Load more» button, infinite scroll, an A to Z picker, a filter sidebar. All of these are usually built in JavaScript. The user clicks or scrolls, a script fetches the next batch, the page grows, and the URL never changes.
From the user’s side, the page now shows 120 products. From the crawler’s side, the page shows the first screen and contains no link to the other items. Google’s pagination documentation makes the same point directly. «Load more» buttons and infinite scroll are generally implemented with JavaScript, and when crawling a site to find pages to index, Google generally crawls the URLs found in the href attribute of anchor elements.
Two consequences follow, and both are easy to miss while you are looking at a page that behaves perfectly:
- The items behind the button are not linked, so they are not discoverable from that listing page. If nothing else on the site links to them, they are invisible to a link-following crawler.
- The listing itself is not a series of pages. Search engines cannot see the shape or the depth of the catalog. They see one page and the first screen of results.
Google’s older advice was to pair infinite scroll with a paginated series, and to mark the sequence with rel=next and rel=prev. The rel part has since been retired. The rest of the advice still stands, and it is really just the link rule again. Give each set of results its own URL, and link to it with an anchor.
3. Depth is a budget, not a cosmetic
Crawl budget is argued about more than it deserves, mostly because it has two parts that get mixed up. There is a rate limit, which is how hard a crawler is willing to hit your server. And there is crawl demand, which is how much the crawler cares about your URLs relative to everything else it has to do.
The useful mental model is that depth from an entry point costs something. A page one click from the homepage is cheap to find and cheap to revisit. A page five or six clicks deep, sitting behind a filter, is expensive. The crawler chooses where to spend, and it usually spends on what looks important.
The symptom shows up in structure rather than in reports. The homepage links to a few prominent categories. Those link to subcategories. Subcategories link to collections. The collections hold the inventory that actually earns money. That is four or five hops before a crawler reaches the pages you care about, and a large catalog tends to push its collection pages to about that depth.
Depth does not only delay discovery. It also lowers how often a page gets revisited after it changes. Your prices change, your stock changes, your descriptions change, and the revisit schedule is set partly by how important the URL looks in the link graph. Deep pages look less important, even when their content is good.
This is a different problem from the one I covered in Keeping A Search Engine Fresh. That post was about crawl scheduling, meaning how often a crawler returns and what the mathematics of change rate looks like. This post is about reachability, meaning whether the crawler can get to the URL at all, and how far it has to walk to get there. You need both. Freshness work on a page nobody can find is wasted effort.
4. Auditing it: three data sources, one comparison
You cannot fix this from a single report. The method that has worked for me merges three sources into one view, per page type.
- Crawl data. Run a full crawl of your own site. This is your baseline. It shows what a crawler that follows links can discover.
- Bot request logs. Read your server logs for Googlebot and other crawlers. This is what bots actually requested, which is not the same thing as what your crawl found. Bots also reach URLs from sources outside your site.
- Sitemap and inventory. Your sitemap, or the product database behind it, is the list of what should exist.
Then compare them, and read the gaps:
- In the inventory but not in the crawl: candidates for orphan pages. If nothing on your site links to a URL, a link-following crawl will not find it. That is what an orphan is.
- In the crawl but crawling badly: the crawl reaches a URL, but the URL is a parameter variant, a sort order or a duplicate of a page you already have.
- In the logs but not in the sitemap: URLs that bots keep requesting and that you never intended to expose.
Two cleanups before you count anything. Filter out variants first, meaning query string duplicates, alternate sort orders and tracking parameters. If you skip this, the orphan list drowns in noise that is not really orphaned. Then filter out the pages you intentionally marked as non-indexable. They belong in a separate list, because the problem there is different: they are discoverable when they should not be.
What remains is the real orphan list. On the catalogs I have worked on it has been a small but meaningful slice of the long tail. Not the top pages, which are always linked, but the pages that had something to offer and no path in.
The other finding worth hunting is links that point at pages you do not want indexed. A link is a vote, and spending votes on non-indexable URLs wastes the crawler’s attention. Filter pages, comparison URLs and internal search results are the usual offenders.
Run this as a recurring audit, not a one-off. Page types change, templates get rebuilt, a new filter ships, and the orphan set grows back on its own.
5. The fix: flatten with URLs, not tricks
Two changes, in order of priority.
First, replace button-driven listings with real, crawlable, paginated URLs. Each page of results gets its own URL, linked sequentially with anchors. Keep the interface you like. If the «next» control is a button, make it an anchor that points at the next page’s URL, and let JavaScript improve the experience on top of that. The anchor is the part that matters.
Second, add a flat, URL-based link list page. One page, or a small set of them, that links directly to the deepest and most valuable pages in the catalog, in plain anchor tags, with no JavaScript required. Static URLs, a clear prefix to avoid slug collisions, and self-referencing canonicals if the list itself paginates.
Design it the way you would design a sitemap for humans. Links and titles, maybe a short description. No heavy media.
The aim is not to game anything. The aim is to shorten the path. A page that sat five clicks deep is now two: homepage, link list, target. A crawler that follows the flat list reaches the whole inventory in a small number of requests, and it spends less of its budget wandering through filter permutations that no user ever sees.
After the change, watch the shape of bot activity rather than only the total. Before, requests scatter across parameter variants and paginated URLs. After, more of the same budget lands on the URLs you care about. On the sites I have worked on, that redistribution was the clearest early signal that the work had landed, well before any ranking movement.
One honest note from that experience. When the first attempt is to hide inlinks from bots instead of adding crawlable paths, requests drop but they do not stop. Bots find URLs by other routes: sitemaps, external links, previously cached pages. Removal is not reachability. Adding a path is.
6. The cloaking question
A page whose only job is to give crawlers a shorter route has a whiff of trouble about it, so it is worth stating the rule plainly.
Cloaking means presenting different content to users and to search engines with the intent to manipulate rankings and mislead users. Google’s spam policy is explicit about the shape of it, including inserting text or keywords into a page only when the requesting user agent is a search engine rather than a human visitor.
A flat link list page is not cloaking, provided you keep it honest:
- Same content users can reach. Every URL on the list is a page a normal visitor can open.
- No hidden or bot-only text. Do not add keyword paragraphs that only crawlers can see, and do not vary the response by user agent.
- Links to the same canonical pages. The list should point at the same URLs the rest of the site treats as canonical, not at variants you created for it.
- Generated from the same source of truth. Build it from the catalog database, not from a hand-maintained spreadsheet, so it cannot drift as inventory changes.
- Visible enough. Link to it from somewhere real, so the page is part of the site rather than a secret.
What belongs on the page: links, titles, and a short description if it helps. What does not belong: heavy media, marketing copy, or anything a user cannot see.
There is a related trap worth naming. Serving a pre-rendered version of a page to crawlers is a known workaround, and Google treats it as acceptable only when the rendered content matches what users get. It also calls it a workaround rather than a recommendation, because it adds complexity and upkeep. If your flat page is really a bot-only render of a JavaScript listing, you are back in that grey area for no reason. Plain anchors to canonical URLs are simpler and carry no risk.
7. A checklist you can run this week
Six questions per page type. If any answer is no, you have a fix to make.
- Real anchor. Is the link an anchor element with an href attribute pointing at a URL? A click handler, or a link whose href only runs script, does not count.
- Unique URL. Does the listing have its own URL, free of session and sort parameters?
- Crawl depth. Is it within a set click limit from the homepage? Pick a number and hold the line.
- In the link graph. Will a crawler that only follows links find it?
- In the sitemap. Is it declared there?
- No wasted votes. Does this page link only to URLs you want indexed?
Rules of thumb I keep coming back to:
- If it needs a click handler, it needs an anchor too.
- The sitemap is a hint. Links are the vote.
- Deleting a link does not remove discoverability. Adding one does.
- Depth is a cost you pay per URL, on every crawl.
8. Where this still breaks
Flattening fixes discovery. It does not fix quality. If a page is thin, duplicated across variants, or answers a query nobody makes, a shorter path only means the crawler arrives faster and decides not to index it.
Depth fixes also do not rescue content that does not match what people search for. That part is easy to forget while the work is architectural. A technically perfect site with content that misses the query will keep losing to a messier site that answers it. Reachability buys you the chance to compete. It does not compete for you.
So the order that has worked for me is this. Make the catalog reachable. Make the pages worth reaching. Then work on how often and how quickly they get refreshed. Fix the path first, because everything downstream depends on it.