TL;DR
Crawl budget is the number of pages Google will crawl on your site within a given timeframe, determined by your server’s capacity and Google’s perceived value of your content. For most small sites under 10,000 pages, crawl budget is not a limiting factor. Google can easily crawl a few hundred or few thousand pages. Crawl budget becomes a real constraint only when you have massive page counts, severe technical inefficiencies, or low-quality content that Google deprioritizes. If you have a small site and pages aren’t getting indexed, the problem is almost certainly content quality or discoverability, not crawl budget.
Small Site Indexing Checklist (Before Blaming Crawl Budget)
If your pages aren’t getting indexed, check these common causes first:
- “Crawled – currently not indexed” in GSC? → Google crawled it. Crawl budget isn’t the issue. Content quality is.
- “Discovered – currently not indexed”? → Google knows the URL but hasn’t crawled. Check internal linking and sitemap submission.
- Noindex tag present? → View page source, search for “noindex”. Remove if unintentional.
- Canonical pointing elsewhere? → Check if
rel="canonical"points to a different URL. Google may be consolidating to that URL instead.
- Robots.txt blocking? → Check
yourdomain.com/robots.txtfor unintentional Disallow rules.
- Duplicate URL versions? → Are both www/non-www or http/https live? Consolidate with redirects.
- Orphan pages? → Pages with no internal links are hard for Google to find. Add links from relevant pages.
- Server errors? → Check GSC Coverage report for 5xx or 4xx errors blocking crawling.
If all checks pass and pages still aren’t indexed, then (and only then) consider whether content quality meets Google’s threshold for inclusion.
The Conversation
Nina, boutique hotel booking site, 3 employees, ~800 pages, WordPress with custom theme, basic SEO knowledge, read some articles about crawl budget, no urgency, skeptical: I keep reading about crawl budget optimization and I’m confused. We have about 800 pages on our site. Is crawl budget something I need to worry about? Some articles make it sound critical, others say it’s irrelevant for small sites.
SDC SEO Brain: For 800 pages, crawl budget is almost certainly not your problem. Google can crawl 800 pages in minutes if your server responds properly. The articles you’re reading are probably written for enterprise sites with millions of pages. What made you start worrying about crawl budget specifically?
Nina: Some of our newer pages aren’t getting indexed. I thought maybe Google isn’t crawling them because we’re using up our crawl budget on other pages.
SDC SEO Brain: That’s a reasonable hypothesis but unlikely at your scale. When pages don’t get indexed, people often blame crawl budget because it sounds technical and fixable. The real reasons are usually simpler: Google found the pages but chose not to index them, or Google hasn’t discovered the pages at all. What does Google Search Console show for these pages?
Nina: I checked URL Inspection for a few. They say “Crawled – currently not indexed.”
SDC SEO Brain: “Crawled” means Google visited those pages. If Google visited them, crawl budget wasn’t the issue. Google spent crawl resources on them. The issue is that Google looked at the content and decided not to include it in the index. What are these unindexed pages?
Nina: They’re individual hotel pages. We have about 150 hotel listings.
SDC SEO Brain: Are these unique hotels with original content, or are they pulling data from somewhere else that might appear on other sites?
Nina: We write our own descriptions, but the basic info like address, amenities, and star ratings is similar to what you’d find on TripAdvisor or Booking.com.
SDC SEO Brain: That’s likely your issue. Google doesn’t need another page with the same hotel’s address and amenity list that exists on a hundred other sites. Your pages aren’t offering something unique that makes them worth indexing. This is a content differentiation problem, not a crawl budget problem.
Nina: So crawl budget is completely irrelevant for us?
SDC SEO Brain: Let me clarify something important. Crawl budget and indexing are separate processes that people often conflate. Crawl budget determines how many pages Google will visit. Indexing determines whether Google keeps what it visited. You can have unlimited crawl budget and still have zero pages indexed if the content doesn’t meet quality thresholds. Your “Crawled – currently not indexed” status proves Google spent crawl resources on your pages. The decision not to index is separate from crawl availability. But to answer your question more directly: crawl budget is not your limiting factor. Let me explain what it actually is so you understand when it matters. Crawl budget has two components: crawl rate limit and crawl demand. Crawl rate limit is how fast Google can crawl without overloading your server. Crawl demand is how much Google wants to crawl based on your site’s perceived value and freshness.
Nina: How does Google decide the crawl demand part?
SDC SEO Brain: Google allocates crawl resources based on signals like site authority, content freshness, historical quality, and how often your content changes. A major news site that publishes hundreds of articles daily gets crawled constantly because Google knows the content changes frequently and has high value. A static site that hasn’t changed in months gets crawled rarely because there’s no urgency.
Nina: Our site changes when we add new hotels or update pricing. Is that enough to keep Google interested?
SDC SEO Brain: Changes help, but the scale matters. Adding 5 new hotels per month is not the same as publishing 50 news articles per day. Google’s crawler isn’t measuring “did anything change,” it’s measuring “how much value can I find here relative to other sites I could crawl instead.” Your 800 pages compete with billions of other pages for Google’s attention.
Nina: That’s actually scary when you put it that way.
SDC SEO Brain: It’s the reality of how search works. But here’s the good news: for a site your size, the competition for crawl resources isn’t your bottleneck. Google can crawl your entire site in a single crawl session. The question is whether Google decides those pages are worth indexing after crawling, which brings us back to the content quality issue.
Nina: Okay, so when does crawl budget actually matter?
SDC SEO Brain: Crawl budget becomes a real constraint in three scenarios. First, massive scale: sites with millions of pages, like large e-commerce catalogs or user-generated content platforms. Google literally can’t crawl everything, so you need to prioritize what gets crawled. Second, severe technical problems: if your server responds slowly, has lots of errors, or wastes crawl on redirect chains, Google may reduce crawl rate. Third, massive low-quality content: if Google crawls thousands of pages and finds them all thin or duplicate, it learns your site isn’t worth prioritizing.
Nina: We might have some technical issues. Our site is kind of slow.
SDC SEO Brain: Site speed affects crawl rate limit. Google won’t hammer a slow server because that creates a bad experience for real users. How slow is slow? What does PageSpeed Insights say about your Time to First Byte?
Nina: I’m not sure what that is.
SDC SEO Brain: Run your homepage through PageSpeed Insights. Look for TTFB, Time to First Byte. That’s how long your server takes to start responding. If it’s over 500ms, your server is slow by modern standards. If it’s over 1-2 seconds, you have significant server issues that could limit crawl rate.
Nina: It says 1.2 seconds for our homepage.
SDC SEO Brain: That’s slow. Your server is taking over a second just to start responding before any content loads. For crawling purposes, Google might limit how aggressively it crawls to avoid overloading your server. But at 800 pages, even a slower crawl rate completes quickly. Your pages are getting crawled, as the “Crawled – currently not indexed” status confirms. The speed issue is more relevant for user experience and conversions than crawl budget.
Nina: So I should fix the speed anyway but not because of crawl budget?
SDC SEO Brain: Correct. Fix speed because users abandon slow sites and conversion suffers. Fix speed because Core Web Vitals are a ranking factor. Fix speed because it’s just good practice. But don’t fix speed expecting it to suddenly get your hotel pages indexed. Different problems, different solutions.
Nina: What’s actually causing the hotel pages not to get indexed then?
SDC SEO Brain: Go back to the content question. Pull up one of the unindexed hotel pages. What unique value does it offer that a searcher couldn’t get from TripAdvisor, Booking.com, or the hotel’s own website?
Nina: We have our own user reviews, some original photos, and pricing.
SDC SEO Brain: How many reviews per hotel typically?
Nina: Most have zero. A few popular ones have like 3-5 reviews.
SDC SEO Brain: There’s your answer. A hotel page with the same basic info as major aggregators, zero original reviews, and photos that may or may not be unique provides almost no differentiated value. Google asks “why would I send searchers here instead of TripAdvisor?” and doesn’t have a good answer.
Nina: But we have better pricing sometimes.
SDC SEO Brain: Pricing isn’t something Google can easily evaluate or surface. And pricing pages often don’t rank well because prices change constantly, making the indexed content quickly stale. More importantly, someone searching “Hotel XYZ” wants information first, then maybe books through whichever channel works. Your pages need to be the best information source, not just another place with the same info plus a booking option.
Nina: How do I differentiate 150 hotel pages?
SDC SEO Brain: You probably can’t equally differentiate all 150. That’s the hard truth of indexation competition. Google doesn’t owe your pages a spot in the index just because they exist. Some strategies: create genuinely unique content like neighborhood guides, insider tips, or detailed amenity breakdowns that aggregators don’t bother with. Focus on getting real user reviews that create unique value. Target long-tail queries that major sites don’t optimize for.
Nina: That’s a lot of content work for 150 hotels.
SDC SEO Brain: It is. This is why many small travel sites struggle to compete with aggregators. The aggregators have millions of reviews, comprehensive data, and massive authority. Competing page-by-page is nearly impossible. An alternative strategy is to not compete on hotel pages at all.
Nina: What do you mean?
SDC SEO Brain: Instead of 150 thin hotel pages trying to rank for hotel names, create destination guides, travel content, and booking guides that link to your hotels. Rank for informational queries like “best neighborhoods to stay in Barcelona” or “boutique hotels vs chain hotels” and funnel that traffic to your booking pages. The booking pages don’t need to rank; they just need to convert traffic from pages that do rank.
Nina: That’s a completely different strategy than what we’ve been doing.
SDC SEO Brain: It is. And it sidesteps the indexation problem entirely. Informational content with genuine value gets indexed because it helps searchers. Thin commercial pages competing with giants don’t get indexed because they don’t add value to Google’s index. You’re fighting the wrong battle.
Nina: But some of our hotel pages do rank. Like 20-30 of them show up in Search Console with impressions.
SDC SEO Brain: Which ones? What’s different about them?
Nina: They’re mostly smaller boutique hotels. Not ones you’d find on big booking sites. And they have more reviews than average, like 8-15 reviews each.
SDC SEO Brain: That pattern makes sense. Smaller boutique hotels have less competition in search. More reviews means more unique content. These pages offer something Google can’t get elsewhere: the only comprehensive page about an obscure boutique hotel. Your 150 hotel strategy might become a 30 boutique hotel strategy, focusing resources on properties where you can actually win.
Nina: That’s interesting. So for small sites, it’s not about crawl budget at all. It’s about whether Google thinks your content is worth indexing.
SDC SEO Brain: Exactly. Crawl budget is the number of pages Google will look at. Content quality determines whether Google keeps what it looked at. For a small site, Google looks at everything. The question is whether it keeps any of it. You could optimize crawl budget to perfection and still have nothing indexed if the content doesn’t provide value.
Nina: Is there any crawl budget thing I should check just to make sure?
SDC SEO Brain: Check your robots.txt to ensure you’re not accidentally blocking important pages. Check for crawl errors in Search Console’s Coverage report. Make sure you don’t have duplicate URL versions like http/https or www/non-www both live. But these are basic hygiene items, not crawl budget optimization. At your scale, basic hygiene is sufficient.
Nina: Our robots.txt just has the default WordPress stuff.
SDC SEO Brain: That’s fine. Default WordPress robots.txt is reasonable. The bigger issue for WordPress sites is often plugin-generated pages: archives, tag pages, author pages, search results pages. These can create crawl waste where Google spends time on low-value pages. But again, at 800 pages, this is minor. If you had 800,000 pages with 700,000 of them being useless parameter variations, we’d have a different conversation.
Nina: So the summary is: stop worrying about crawl budget and focus on making content Google actually wants to index?
SDC SEO Brain: For your site, yes. Crawl budget is a real concept that matters at scale, but it’s been oversold to small site owners as an explanation for indexation problems. The real explanation is almost always: Google crawled your pages, evaluated them, and decided the index is better without them. Changing that decision requires changing the content, not optimizing how fast Google crawls.
Nina: One last question. If our site grows to like 5,000 or 10,000 pages, when should I start thinking about crawl budget?
SDC SEO Brain: Even at 10,000 pages, crawl budget probably won’t be your constraint unless you have significant technical problems or tons of low-quality pages. The sites that genuinely worry about crawl budget are typically 100,000+ pages, or sites with complex JavaScript rendering issues, or sites with severe duplicate content at scale. For a 10,000-page site that’s technically sound and maintains content quality, Google handles it fine.
Nina: That’s reassuring actually. One less thing to worry about.
SDC SEO Brain: Focus on problems in the order that matters. Content value first, technical health second, crawl budget a distant third unless you’re operating at enterprise scale or have massive technical debt. The internet is full of small site owners optimizing crawl budget while their content doesn’t deserve to be indexed regardless.
FAQ
Q: What is crawl budget?
A: Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. It’s determined by two factors: crawl rate limit (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl based on your site’s perceived value, authority, and freshness). Google allocates more crawl resources to sites it considers valuable and updated frequently.
Q: Does crawl budget matter for small websites?
A: For most sites under 10,000 pages, crawl budget is not a meaningful constraint. Google can crawl thousands of pages quickly if your server responds properly. If small sites have indexation problems, the cause is almost always content quality (Google chose not to index after crawling) or discoverability (Google hasn’t found the pages yet), not crawl budget exhaustion.
Q: Why are my pages “Crawled – currently not indexed”?
A: This status means Google visited and evaluated your pages but decided not to include them in the index. Common causes: thin content that doesn’t add unique value, duplicate content that exists elsewhere, content that doesn’t serve a clear search intent, or pages on sites with low overall authority. The solution is improving content quality, not crawl budget optimization.
Q: What page count requires crawl budget optimization?
A: Crawl budget typically becomes relevant for sites with 100,000+ pages, sites with severe technical problems (slow servers, redirect chains, JavaScript rendering issues), or sites with massive duplicate content. Even at 50,000 pages, most technically sound sites don’t face crawl budget constraints. Focus on content quality until scale genuinely forces crawl budget considerations.
Q: How can I check my crawl budget?
A: Google doesn’t provide a specific “crawl budget” number, but you can see crawl activity in Google Search Console under Settings → Crawl Stats. This shows how many pages Googlebot crawled, average response time, and crawl trends. If Google is crawling hundreds of pages daily and your site only has thousands of pages, crawl budget isn’t your constraint.
Summary
Crawl budget represents the number of pages Google will crawl on your site within a given period, controlled by two factors: your server’s capacity to handle requests (crawl rate limit) and Google’s interest in your content (crawl demand). For enterprise sites with millions of pages, optimizing which pages receive crawl resources is genuinely important. For small to medium sites, crawl budget is almost never the limiting factor.
The pattern for small site indexation problems is consistent: pages show “Crawled – currently not indexed” in Search Console, indicating Google visited but chose not to index. This confirms crawl budget wasn’t exhausted; Google spent resources crawling those pages. The decision not to index is a content quality judgment, not a resource constraint.
Content must provide unique value for Google to include it in the index. Pages that duplicate information available on higher-authority sites, contain thin content without genuine depth, or serve no clear search intent typically won’t be indexed regardless of crawl budget availability. Google’s index has limited space and prioritizes pages that help searchers.
Technical factors affect crawl rate, not crawl demand. Slow server response (high TTFB), excessive redirect chains, and server errors can limit how fast Google crawls, but for small sites, even slower crawling completes quickly enough. Fix technical issues for user experience and ranking benefits, not crawl budget.
Scale thresholds for crawl budget concern: Sites under 10,000 pages rarely face constraints. Sites with 10,000-100,000 pages need basic technical hygiene but usually aren’t limited. Sites over 100,000 pages, or sites with severe technical debt, or sites with massive programmatic duplicate content may need strategic crawl budget optimization.
Priority order for small sites: Content quality and uniqueness first, technical health second, crawl budget a distant third. Optimizing crawl budget for a site with content Google doesn’t want to index solves the wrong problem.
Sources
- Google Search Central: Crawl budget management – https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
- Google Search Central: Googlebot – https://developers.google.com/search/docs/crawling-indexing/googlebot
- Google Search Console Help: Crawl stats report – https://support.google.com/webmasters/answer/9679690