Crawl budget in the AI era: how GPTBot, ClaudeBot and PerplexityBot spend it
AI crawlers have their own budgets and priorities. Here is how to make sure the bots that feed answer engines reach your most important pages first.
Crawl budget used to be a concern only for very large sites. In the AI era it matters more broadly, because there are now many crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot and more — each with its own budget and revisit cadence. If they spend that budget on junk URLs, your citation-worthy pages get crawled less often and drift out of date in answers.
Eliminate crawl traps
- Faceted navigation that generates near-infinite filter combinations.
- Session IDs or tracking parameters that fork every URL.
- Calendar and pagination loops with no end.
- Soft 404s that return 200 for pages that are really empty.
Each trap is budget spent learning nothing. Use robots.txt disallows, canonical tags and parameter handling to steer crawlers away from them.
Shorten the path to value
Fix redirect chains — every hop costs a fetch. Keep important pages within a few clicks of the home page. Make sure your internal links point to canonical URLs, not to redirects or parameterized variants. A shallow, clean architecture lets a limited budget reach more of what matters.
Read the logs
Server access logs are the ground truth for crawl behavior. Filter by user agent to see which AI bots visit, how often, and which sections they favor or ignore. If PerplexityBot never reaches your product pages, no amount of on-page optimization will get you cited there. LumenEntity's crawler analytics turn these logs into a bot-by-day view so the gaps are obvious.
Direct the budget you can
You cannot command an AI crawler, but you can nudge the Bing-side budget with IndexNow: when a high-value page changes, push it so Bing prioritizes the recrawl instead of waiting. Combined with an honest sitemap lastmod, this concentrates scarce budget on exactly the URLs you care about.
Frequently asked questions
- Do AI crawlers respect robots.txt?
- The major, named agents generally do. Enforce boundaries at the server or WAF level for anything that misbehaves, but do not block the answer bots you want citations from.
- How do I see which AI bots crawl my site?
- Analyze server access logs by user agent, or use a crawler-analytics tool that classifies bots for you. Logs are the only fully reliable source.
- Does blocking training bots hurt AI visibility?
- It can. Some retrieval and training agents overlap, and blocking broadly may remove you from tomorrow's answers. Decide per-bot, based on the channel value.