Crawl budget for local sites: stop wasting it on thin location pages

Crawl budget rarely matters for small sites, but a local site that publishes hundreds of thin area pages can choke its own crawl queue. The fix is pruning and strengthening pages and improving server response time, not crawl tricks. Google says crawl budget is generally a concern only for large or frequently changing sites, so most small local sites do not need to worry about it at all.

This is a supporting note under our indexation work. If your area pages are not getting indexed in the first place, start with the parent guide on why location pages are not indexed and treat this page as the technical follow-up for sites that have already published a large batch.

By Stephen Evans. Published 4 September 2026. Figures and policy quotes current as of that date.

What is crawl budget, briefly?

Crawl budget is the number of URLs Googlebot is willing and able to fetch from your site in a given period. It is shaped by two things: how much crawling your server can handle without slowing down (the crawl capacity limit), and how much Google wants to crawl your site based on its popularity and how often it changes (crawl demand). You do not set it directly. Google adjusts it as it watches how your server responds.

For a plumber with twelve pages, or an estate agent with forty, this number is effectively unlimited relative to the site. Google can fetch everything you have many times over before the budget runs out. Crawl budget only becomes a real constraint when the count of URLs grows large enough that Google starts making choices about what to fetch and what to leave in the queue.

Does crawl budget actually matter for a local site?

For most local sites, no. Google’s own crawl-budget documentation (developers.google.com, accessed 4 September 2026) opens by telling small sites to stop reading: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” That is the honest position. A site with a few dozen pages, crawled the day they go live, has nothing to optimise here.

The same documentation names the sites that do need to pay attention. Crawl budget is a concern for large sites with over one million unique pages whose content changes moderately often, for medium or larger sites with more than 10,000 unique pages whose content changes daily, and for sites where Search Console classifies a large share of URLs as “Discovered – currently not indexed”. That last category is where a local site can end up unexpectedly.

A local site rarely crosses the page-count thresholds. It crosses into trouble a different way. The moment you bulk-publish hundreds of near-identical area pages, you create exactly the third condition: a pile of low-value URLs that Google discovers, queues, and then declines to index. You have not run out of budget in the literal sense. You have given Google a strong reason to slow down and reprioritise, because much of what it is being asked to crawl turns out to be thin.

When a thin-page glut becomes a crawl problem

Here is the chain, in order, from the moment you publish a large batch of weak area pages:

Google discovers hundreds of new URLs from your sitemap and internal links in a short window.
It starts fetching them. Many turn out to be near-duplicates of each other, differing only by a town name.
It indexes a handful, leaves the rest as “Discovered – currently not indexed” or “Crawled – currently not indexed”, and visits them less and less often.
Because the site now signals a lot of low-value content, crawl demand for the whole site can soften, so even your good pages get fetched less promptly.

Google’s documentation puts the consequence plainly: “If Google spends too much time crawling URLs that it shouldn’t, Google’s crawlers might decide that it’s not worth the time to look at the rest of your site” (developers.google.com, accessed 4 September 2026). That is the real cost of a thin-page glut. It is not just that the thin pages fail to rank. It is that they can drag on how the rest of the site is treated.

We see this from the engine side. The Townsmith quality score exists precisely to stop this batch from ever being published. Pages that score below the threshold are held back rather than pushed live, so the glut that triggers the crawl problem never reaches Google in the first place. If you want the full picture of what happens to those pages once they are live and not indexed, the indexation pillar covers the not-indexed pile and how to clear it.

How do you fix it without crawl tricks?

There is no crawl hack. You cannot inflate budget by editing robots.txt, adding crawl-delay, or pinging Google harder. The fix is to give Google less low-value work and a faster server, so it can read more of your good content. In short: prune thin pages, improve server response time, and segment your sitemap so coverage is visible.

Prune and strengthen thin pages

Pruning is the single biggest lever. Every near-duplicate area page you remove or merge is crawl work Google no longer has to do, and one fewer signal that your site is full of thin content. Google’s advice is direct: “Eliminate duplicate content to focus crawling on unique content rather than unique URLs” (developers.google.com, accessed 4 September 2026).

Identify the area pages that differ only by a town name. A page for Uckfield and a page for Lewes that say the same thing about boiler repair with the place name swapped are duplicates in everything but the URL.
Keep the pages that earn real visits and carry genuinely local detail. Merge or remove the rest rather than leaving them to sit unindexed.
For pages you keep, add the things only a real local business knows: which jobs are common in that town, response times to that area, named landmarks or estates you actually cover.
Decide how many area pages you can realistically support before you publish more. Our guide on how many location pages you should have works through that number.

Pruning and quality control are the same job. The Townsmith quality score grades each page 0 to 100 and holds back the thin ones, so a pruned, strong set is what ships. You can read how that scoring works in the quality score documentation.

Improve server response time (TTFB)

The second lever is speed. Crawl capacity moves with how your server responds. Google’s documentation describes the mechanism: “If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down” (developers.google.com, accessed 4 September 2026). A slow time to first byte (TTFB) directly throttles how much Google will fetch.

It also adds: “Make your pages efficient to load. If Google can load and render your pages faster, we might be able to read more content from your site.” Practical steps for a WordPress local site:

Put page caching in front of PHP so Googlebot is served static HTML, not a fresh database query per request.
Check your TTFB on a cold cache as a working rule of thumb. If it is routinely up around a second or more, your host or your stack is the bottleneck, not your content.
Fix server errors and long redirect chains. A run of 5xx responses pushes the crawl limit down quickly.

The Townsmith engine builds real, editable location pages and outputs their schema server-side, so the rendered page is straightforward HTML that caches cleanly. There is no per-request external call to wait on, which keeps response time predictable for the crawler.

Segment the sitemap so coverage is visible

Sitemap segmentation does not save crawl budget by itself. What it buys you is visibility: put your location pages in their own sitemap, and Search Console’s coverage report will show you what share of just those pages is indexed, instead of burying them among posts, products and category archives.

That single number, indexed location pages over total submitted, is the metric that tells you whether pruning is working. If you submit 200 area pages in their own sitemap and Search Console shows 40 indexed, you know four out of five are dead weight and the crawl queue is being spent on pages that never make it in.

Keep one sitemap dedicated to location pages, separate from your blog and core pages.
In Search Console, filter the coverage report by that sitemap to read the indexed share for area pages alone.
Watch the indexed share move as you prune. A rising share means your remaining pages are being treated as worth crawling.

This is part of measuring the programme properly. For how to read coverage and the wider set of signals over time, see the guide on how to measure location pages.

What about robots.txt and noindex for the thin pages?

Blocking thin pages in robots.txt stops Google fetching them, which does free crawl capacity, but it does not remove them from the index if they are already indexed, and it hides them from your own coverage reporting. Adding noindex lets Google crawl them, see the directive, and drop them, but Google still has to crawl them to read the tag, so it is not a crawl saving on its own.

Neither is a substitute for the real decision: if a page is too thin to index, it is usually too thin to keep. Prune it or strengthen it. Use noindex as a transitional step while you decide, not as a permanent home for pages you could not bring yourself to delete.

Common questions about crawl budget for local sites

How many pages before crawl budget matters?

Google’s named thresholds are over one million unique pages for sites that change moderately often, and over 10,000 for sites that change daily (developers.google.com, accessed 4 September 2026). Almost no local site reaches those. The practical trigger for a local site is not a page count at all. It is a large share of URLs sitting in “Discovered – currently not indexed”, which a bulk publish of thin area pages can cause well below 10,000 pages.

Will deleting pages hurt my rankings?

Deleting pages that were never indexed cannot cost you rankings, because they were not ranking. Deleting a thin page that pulled the occasional visit is a trade: you lose a little traffic from a weak page and reduce the low-value signal across the site. Merge such a page into a stronger one and redirect it rather than deleting outright, so any value it held is consolidated rather than dropped.

Does adding more internal links increase crawl budget?

Internal links help Google discover and prioritise pages, but they do not manufacture extra crawl capacity. Pointing more links at a thin page just gets it crawled sooner so it can be declined for indexing sooner. Spend internal links on pages worth ranking, not on propping up the glut.

Is there a crawl-budget setting in the Townsmith plugin?

No, and there should not be, because crawl budget is not something a plugin can set. What the free plugin does is prevent the cause: it builds real, editable location pages, scores each one 0 to 100, holds the thin ones back, and outputs schema, all server-side with no external calls. By stopping a thin-page glut from publishing, it removes the main reason a local site ever runs into a crawl problem. You can see what the engine produces on the features page.

The short version

Most local sites do not need to worry about crawl budget. Google says small sites whose pages are crawled the same day they publish can ignore the topic.
The exception is a thin-page glut. Bulk-publishing hundreds of near-identical area pages creates a pile of unindexed URLs that can deprioritise the whole crawl queue.
Prune and strengthen the thin pages. Fewer duplicates means crawling focuses on unique content rather than unique URLs.
Improve server response time. Faster TTFB raises the crawl limit, slow responses and errors lower it.
Segment the sitemap so Search Console shows the indexed share of your location pages, which is how you tell whether pruning is working.
There is no crawl trick. The optimisation is good people-first content and a fast server.

Source: Google Search Central, “Large site owner’s guide to managing your crawl budget”, developers.google.com, accessed 4 September 2026.