"Crawl budget" is one of those SEO phrases that sounds scientific and gets treated like a crisis. In reality, for most UK business websites it barely matters — and for the sites where it does matter, it's rarely the actual problem. This guide gives you the honest version: what crawl budget really is, how to tell whether your site is big enough to care, how to diagnose indexation problems properly, and the fixes that actually move the needle.

What crawl budget actually is

Google doesn't crawl every site equally. For each website it decides, roughly, how many pages it's willing to fetch and how often — based on the site's size, its update frequency, its server speed, and how valuable Google judges the site to be. That allocation is your crawl budget.

Here's the part most guides bury: crawl budget is only a constraint on very large sites. Google's own guidance puts the threshold around hundreds of thousands of URLs. If your site has 200 service pages and a blog, Google can crawl the lot without breaking a sweat. Your problem is not budget; it's something else.

What people usually mean when they say "crawl budget" is really indexation: pages that exist but don't appear in Google. The two are related but different, and mixing them up leads to the wrong fixes.

Diagnose before you optimise: is it crawling or indexing?

Open Google Search Console and work through these in order:

  1. Pages → "Why pages aren't indexed". This report tells you the real story. "Discovered – currently not indexed" means Google knows the URL exists but hasn't bothered crawling it — the classic large-site symptom. "Crawled – currently not indexed" means Google looked at it and decided it wasn't worth indexing — that's a content quality problem, not a crawl problem.
  2. Check the ratio. A site with 5,000 URLs and 4,800 indexed is healthy. A site with 500 URLs and 300 indexed has a quality or architecture problem, not a budget problem.
  3. Look at crawl stats. Settings → Crawl stats shows Google's actual crawl activity. If it's crawling thousands of pages a day, you don't have a crawl shortage — you have an indexing decision problem.
  4. Test a sample URL. Use URL Inspection on an unindexed page. It tells you exactly what Google sees: blocked by robots? noindex tag? canonical pointing elsewhere? duplicate of another page? Most "indexation mysteries" resolve here in thirty seconds.

The six fixes that actually help

Once you know what you're dealing with:

  1. Kill the dead weight. Thin tag pages, empty categories, ancient press releases nobody links to — every low-value URL is attention Google could spend on pages that earn revenue. Noindex or remove them. This is the single highest-impact move on bloated sites.
  2. Fix your internal linking. Pages Google can't reach from your navigation or contextual links are pages Google won't prioritise. Orphan pages (no internal links at all) are the most common cause of "Discovered – not indexed" on small sites.
  3. Clean up parameter URLs. Faceted navigation, sort orders and session parameters can generate thousands of near-duplicate URLs that eat crawl attention. Canonical them, noindex them, or block the worst offenders in robots.txt — in that order of preference.
  4. Keep the sitemap honest. Your XML sitemap should contain only indexable, canonical, 200-status URLs. A sitemap stuffed with redirects, 404s and noindexed pages teaches Google to distrust it.
  5. Speed up the server. Slow response times genuinely reduce how much Google crawls — it's polite like that. Time to first byte under a second is a reasonable target for most business sites.
  6. Stop publishing filler. "Crawled – currently not indexed" is Google telling you the content isn't worth it. Fewer, better pages beat a content treadmill every time.

Log-file analysis: when you need the real data

Search Console shows you Google's conclusions; server log files show you Google's behaviour. For large sites, log analysis answers questions nothing else can: which pages does Googlebot actually visit, how often, how deep does it go, and where does it waste time? The pattern to look for is crawl waste — a high share of bot hits going to parameter URLs, internal search results, paginated pages or legacy sections while money pages get visited rarely.

You don't need enterprise software to start: a month of logs, a parser (Screaming Frog's log analyser has a free tier), and a spreadsheet. Compare crawl frequency against page value — your top revenue pages should be among the most-crawled. If they're not, the fixes in the previous section (internal linking, pruning, parameter control) are where the evidence points. For most SMEs this is overkill; for sites with 50,000+ URLs it's the only honest way to manage crawl.

JavaScript rendering and crawl waste

Modern sites increasingly render content with JavaScript, and Google handles it — but in two waves. The first wave indexes the raw HTML; the second wave, days or weeks later, renders the JavaScript. Content that only exists after rendering waits for wave two, and on large sites wave two may never fully arrive. The crawl-budget implication: every JS-dependent page costs Google extra rendering resources, which means fewer of your pages get fully processed. The fix isn't to abandon JavaScript — it's server-side rendering (or pre-rendering) for critical content: the text, links and metadata Google needs should be in the initial HTML. Test with URL Inspection's "view crawled page" — if your key content is missing from the raw HTML, you're paying a crawl tax on every page.

The honest limits

No technical trick forces Google to index a page it doesn't want. You can't "spend" your way to indexation with sitemap submissions or fetch requests — URL Inspection helps discovery, not decisions. And on a genuinely large site (marketplaces, publishers, aggregators), crawl management is a permanent discipline, not a one-off fix: log-file analysis, crawl-priority tiers, and ongoing pruning.

For everyone else, the uncomfortable truth is kinder: fix your internal links, cut the thin content, and keep the sitemap clean, and indexation usually sorts itself out within a few weeks.

Not sure whether you're dealing with a crawl problem, a content problem, or both? Our technical SEO team diagnoses indexation properly — log data, Search Console forensics, and a fix list in priority order, not guesswork. Also worth reading: our guide to fixing broken links, the most common crawl-waster we find. Get a free consultation.

SEO Visuosofts Get a Quote