How to Do SEO for a Site with 1M+ Pages

Published Read 6 min read

On this page

Past a million pages, crawl budget is the binding constraint, not content quality on any single page. Googlebot will never crawl your whole site frequently, so the job stops being “get everything indexed” and becomes “get the right pages crawled, indexed, and kept fresh, and stop wasting crawl on everything else.” That reframe changes the entire workload. Instead of producing more, you measure where Googlebot actually spends its time, eliminate the waste, shrink the indexable footprint to the pages that earn their place, and direct internal authority at the URLs that matter. The lever is engineering, and the primary instrument is your server logs, because they are the only record of what Google actually crawled rather than what you hope it crawled.

Measure real crawl coverage with logs

Google itself frames crawl budget as the set of URLs it can and wants to crawl, and flags that sites above roughly 10,000 URLs, or those producing content faster than Google indexes it, need to manage it deliberately. At a million-plus pages you are far past that line, and the first question is empirical: what share of your URLs did Googlebot actually visit in the last 30 days, and how is that crawl distributed across page types?

Server log analysis answers this and nothing else can. Parse Googlebot hits and bucket them by template (product, category, faceted result, article, profile, utility). The distribution is usually alarming the first time you look: a large fraction of crawl spent on parameter URLs, paginated tails, and dead utility pages, while priority pages get visited rarely. GSC Crawl Stats gives you the aggregate trend and host-level totals; logs give you the per-URL-type breakdown you need to act. You are looking for two numbers per template: how often Google visits it, and whether those visits are landing on pages you actually want indexed.

Eliminate crawl waste

Every Googlebot request spent on a URL you do not want indexed is a request not spent on one you do. At scale this is the highest-leverage work.

The usual offenders: faceted navigation that multiplies a few thousand products into millions of filter-combination URLs, legacy and utility pages that no longer serve anyone, redirect chains that make Google follow three hops to reach content, and non-200 responses that consume requests for nothing. Block faceted and parameter crawling at the source. The old GSC URL Parameters tool that used to handle this was retired in 2022, so do not reach for it; control faceting through robots.txt disallow rules, noindex on the thin combinations, rel=nofollow on filter links where appropriate, or architecture that does not generate the combinatorial explosion in the first place. Collapse redirect chains to single hops. Find and fix the soft-404s and server errors that logs reveal Googlebot repeatedly hitting. The point is not tidiness; it is reallocating a fixed crawl supply toward pages that deserve it.

Manage the index, do not just grow it

“Crawled – currently not indexed” in Search Console is not a neutral status. It is Google telling you it fetched the page, evaluated it, and decided it was not worth showing, usually because the page is thin or duplicative. At a million pages, a large and growing count of “Crawled – currently not indexed” is a quality verdict on your page population, and the answer is rarely to add words to each one.

Set a minimum-indexing threshold: a defensible bar a page must clear (unique value, sufficient content, genuine search demand) to deserve a place in the index. Pages below it should be consolidated into a stronger canonical where they are near-duplicates, or noindexed and dropped from the sitemap where they are simply thin. This feels counterintuitive at scale, where the instinct is that more indexed pages mean more ranking chances. The opposite is closer to true here: a smaller set of pages that each clear the bar can outperform a sprawling set where the long tail of thin pages drags the whole domain’s perceived quality down and burns the crawl that the good pages needed.

Direct authority with architecture

Internal structure decides where both link equity and crawl attention flow, and at scale you have to engineer it deliberately rather than let a flat or accidental structure spread both thin.

Weight internal linking toward priority pages, so the URLs you most want crawled and ranked sit closest to the high-authority, high-crawl parts of the site (typically the homepage and major hubs) and receive the most internal links. Segment XML sitemaps by priority tier and page type. Segmentation does not command Google to crawl, and a sitemap is not an instruction; it is a tracking and diagnostic aid. Its real value at scale is that when you split sitemaps by section, GSC’s index-coverage reporting per sitemap tells you which segments are getting indexed and which are not, turning a single opaque number into an actionable per-section signal.

Tier your content investment

You cannot hand-write a million pages, and you should not pretend to. Decide the investment level by template and by the page’s value.

A workable tiering: fully human-crafted content for the highest-value, highest-competition pages; enhanced-template pages for the mid-tier, where a strong template is enriched with genuinely page-specific data, surrounding context, and any user-generated content available; and a minimum-viable tier for the long tail that still must clear the indexing threshold or be excluded. Programmatic enhancement (pulling in real, distinct data per page) and authentic UGC are the levers that give template pages enough uniqueness to clear the bar. The failure mode is a million pages built from the same template with only a swapped noun, which is exactly the population that lands in “Crawled – currently not indexed.”

Get new pages crawled quickly, then let them settle

Large sites that publish constantly have a second crawl problem: new pages sitting undiscovered. Crawl velocity for fresh URLs is driven by structural signals more than by pinging. Link new pages from sections Google already crawls often, so a high-frequency hub surfaces them fast. Keep lastmod in your sitemaps accurate, because Google does use it as a freshness hint and abusing it (bumping every date on every deploy) trains Google to distrust it. Expect this attention to taper: a page gets crawled aggressively when new and discovered, then settles into a lower steady-state frequency as it ages, which is normal and correct. The job is to make sure the initial discovery is fast and the steady-state pages worth keeping stay in the index.

Frequently Asked Questions

Should I try to get all million-plus pages indexed?
No. Total indexation is the wrong goal at this scale. Aim to get the pages that meet a quality and demand threshold crawled, indexed, and kept fresh, and deliberately exclude the thin long tail. A leaner, higher-quality index typically outperforms a bloated one because it stops the thin pages from dragging perceived site quality and stops them from burning the crawl budget the valuable pages need.

Do I really need log files, or is Search Console enough?
GSC Crawl Stats gives you host-level trends and totals, which are useful, but only server logs show crawl distribution per URL and per template, which is what you need to find waste and prove where Googlebot actually spends its budget. At a million-plus pages, log analysis is not optional.

Sources

Google, Large site owner’s guide to managing your crawl budget: https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
Google Search Console Help, Page indexing report (“Crawled – currently not indexed”): https://support.google.com/webmasters/answer/7440203

Related posts:

  1. The Infinite Scroll SEO Problem Nobody Talks About Infinite scroll breaks indexing because Googlebot's renderer does not fire scroll events, so anything your page loads in response to scrolling is invisible to Google. The Web Rendering Service executes......
  2. Rankings Crash Every Time You Publish New Content When your established pages dip right after you publish something unrelated, you are usually seeing the interaction of three mechanisms rather than one mysterious penalty: cannibalization between the new page......
  3. What Is Semantic SEO and How to Implement It Semantic SEO is optimizing for meaning rather than exact-match keywords. Google no longer reads a page as a bag of keyword strings; its natural-language systems understand topics, the entities involved,......
  4. How to Do an SEO Competitor Analysis Start by identifying your real SEO competitors, which are whoever ranks for your target queries, not your business rivals. The companies your sales team loses deals to often have a......
  5. How to Do Local SEO for Multiple Locations Every location you operate is a separate local-SEO entity that Google ranks on its own signals: its own Google Business Profile, its own reviews, its own citations, its own local......
  6. Forum SEO After Google’s Parasite Update If you run a forum and you have been told that adding editorial oversight to sponsored or placed content keeps you safe under Google's site reputation abuse policy, that advice......
  7. JavaScript Redirects Are Leaking PageRank to Nowhere If you moved a page months ago and the old URL still ranks on its own, sometimes competing with the page you redirected it to, the cause is almost never......
  8. SEO Process Documentation: Playbooks, SOPs, and Knowledge Management Documentation earns its keep only when it lowers the cost of doing the work correctly below the cost of doing it from memory. Most SEO teams get this backwards: they......
  9. Website Merger SEO: Two Domains, One Survivor When you fold two domains into one, you do not add two audiences together. You preserve part of the redirected domain's equity, decay another part during the handoff, and inherit......
  10. How to Do SEO During a Website Redesign A redesign destroys organic traffic through a small set of preventable mistakes: changing URLs without one-to-one redirects, shipping a staging noindex to production, removing pages that hold backlinks, and bulk-redirecting......
  11. How to Do SEO When Rebranding (Same Domain) A same-domain rebrand leaves your non-branded rankings essentially intact, because the things that earn them, domain authority, backlinks, URLs, and on-page topical content, do not change when only the name......
  12. How to Do SEO for API Documentation Developer documentation is an acquisition channel, not just a support cost, and treating it that way is the whole strategy. Developers search precise technical strings: exact error codes, method names,......
  13. How to Do SEO for Franchise Businesses Franchise SEO breaks down as a governance problem, not a technical one. When a corporate brand site and dozens of independently operated franchisee sites all chase the same brand-plus-city queries,......
  14. How to Do SEO for Programmatic Pages at Scale The line between legitimate programmatic SEO and doorway spam is one question: does each generated page carry genuinely unique data value, or is it a template with swapped variables? A......
  15. Your WordPress Theme Update Removed All Your Structured Data If your rich results vanished and the only recent change was a theme update, the theme was almost certainly emitting your structured data and the update dropped or refactored it.......