Don’t use 403s or 404s for rate limiting

Over the last few months we noticed an uptick in website owners and some content delivery
networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to
reduce Googlebot’s crawl rate.