Definition
Crawl budget — Crawl budget is the number of URLs a crawler can and wants to crawl on a site in a given period.
Quick answer
It is a real concern mostly for large sites (hundreds of thousands of pages) or sites with many filter-generated URLs. Remove useless URLs, errors and redirect chains.
What it consists of
Google describes crawl budget as the result of two factors: the crawl capacity limit (how much it can crawl without overloading the server) and crawl demand (how popular and fresh the pages are).
What wastes it
- parameters and filter combinations;
- soft 404s and duplicates;
- redirect chains;
- a slow server.
Sources
- Large site owner’s guide to managing your crawl budget — Google Search Central