Google Retry-After and Crawl Rate: What Changed on 6 Oct

On 6 October 2026 Google restructured the emergency crawl-rate section of its documentation and spelled out the Retry-After HTTP header. If your server is overloaded, you can return a 503 or 429 with Retry-After, either as a number of seconds or an absolute UTC date, and Google's crawlers will slow down. It is an emergency brake for a day or two, not a crawl-budget tuning tool.

Key Takeaways

• The change is documentation, dated 6 October 2026 in Google's crawling changelog: Retry-After detail and examples were added to the reduce-crawl-rate guide.
• The header was already mentioned in Google's guidance on temporarily pausing a site; this puts it where people look during an incident.
• Use 503 or 429, optionally with Retry-After (for example Retry-After: 120). Google also treats 500 as a slow-down signal.
• Keep it to roughly 1-2 days. Several days of errors on the same URL can get it dropped from the index.
• Errors slow crawling for the whole hostname, and Google Ads may pause or cancel campaigns during the window.
• If you cannot serve errors, Search Console has a special request route, but Google says it cannot raise crawl rate on request.

What exactly changed

Search Engine Roundtable reported on 6 October that Google "restructured the emergency crawl rate reduction section", with most other edits just moving content around. The new part is the explicit Retry-After guidance, referencing RFC 9110, plus example code (Search Engine Roundtable). Google's crawling changelog lists the same entry: "Added Retry-After HTTP header details to the reduce crawl rate documentation" (Google changelog).

Important framing: this is not a new mechanism. Retry-After was already covered in Google's page on pausing or disabling a site, as the Roundtable piece notes. The edit simply puts it in the crawl-rate guide.

What Retry-After does

Retry-After is a standard HTTP response header. It tells a client how long to wait before another request. Per MDN, it can carry a delay in seconds or an HTTP date (MDN reference). Google's guide confirms both forms for crawlers: seconds (for instance Retry-After: 120) or an absolute UTC date and time (Google crawl rate guide).

Which status codes pair with it

Google's page lists 500, 503 and 429 as ways to slow crawling. Retry-After is documented as an extra signal for 503 and 429. My reading: use 503 for planned maintenance or overload, 429 when you are rate limiting.

An example response:

HTTP/1.1 503 Service Unavailable
Retry-After: 3600

I have not tested how precisely Google honours the interval, and the documentation does not promise an exact schedule, so do not build an SLA on it.

When to use it (and when not to)

Use it when:

• A traffic spike, bad deploy or database problem makes your server unable to serve pages.
• A migration or batch job is saturating the origin.
• A bot storm hits and Googlebot is part of the load.

Do not use it when:

• You want to "optimise crawl budget" on a healthy site. Fix internal links, parameters and sitemaps instead.
• You want to stop indexing. Use noindex or robots.txt properly.
• The problem is a slow page template. Fix the template.

How long is too long

Google says to keep this to a short window, such as "a couple of hours, or 1-2 days". After that, if Googlebot keeps seeing these codes on the same URL across several days, the URL may be dropped from the index. Crawling speeds back up automatically after the errors stop (Google crawl rate guide).

The trade-off is real: new pages get discovered later, price and availability updates lag, and removed pages may remain indexed longer.

The Google Ads side people forget

The same page warns that during the error window, Google Ads campaigns may be paused or cancelled and ads may stop serving. If you run Shopping or Search ads to landing pages on the same hostname, a site-wide 503 is expensive. For ecommerce or lead-gen sites, coordinate with the paid team before flipping the switch.

Step-by-step incident playbook

1. Confirm the cause. Check server load and Search Console Crawl Stats. If the load is coming from bots, verify the user agents are real (Google verification docs).
2. Decide scope. The errors affect the whole hostname, even pages still returning content. A subdomain for heavy content can contain the blast radius.
3. Return 503 or 429 at the edge or load balancer, with a sensible Retry-After.
4. Set a calendar reminder for removing it. Write the time down in the incident log.
5. Fix the cause while the brake is on.
6. Remove the errors, test with Search Console URL Inspection (Search Console Help), and watch Crawl Stats recover.

What it does not fix

• It does not increase crawl rate. Google says it cannot raise crawl rate on request.
• It does not target one URL. The effect is host-level.
• It does not help with AI crawlers that ignore it. Other bots follow their own rules.

If your slowdown comes from CDN rules blocking bots rather than overload, that is a different problem; see Cloudflare, Googlebot and default blocks.

The alternative: the Search Console request

If serving errors is not feasible, Google's guide describes a special request in Search Console describing the optimal crawl rate for your site. Evaluation can take several days, which makes it a poor emergency tool but fine for chronic issues.

What this means for marketers, not just engineers

You may never touch a header, but you should know it exists. Ask your developer three questions this week: Can we return 503 with Retry-After at the edge? Who decides when to turn it on? Who tells paid media? Write the answers in a one-page runbook. The same page of the runbook can hold your contact for spam-update recovery questions, because both are "something changed on Google's side" moments.

Verified status and unverified items

As of 9 October 2026 the details above come from Google's documentation and changelog plus Search Engine Roundtable. I did not test Google's response timing, and I did not read the example code, which Roundtable shows as an image. Treat any claim of an exact honour interval as unverified.

Reading more

For the HTTP semantics beyond Google, http.dev's Retry-After reference is a clear primer, and PPC Land has summarised Google's wider crawling documentation updates (PPC Land) and the archived pre-change version of the page (Internet Archive snapshot dated 5 October 2026), which I have not diffed line by line. I link them because I saw them via search results or the Roundtable article, not because I audited every claim in them. Also see Search Engine Journal on how CDN bot rules can create their own crawl failures (Search Engine Journal).

A sample runbook entry

Trigger: origin CPU above threshold for 10 minutes with Googlebot traffic over normal. Action: edge returns 503 with Retry-After 1800 on non-checkout paths. Owner: on-call engineer. Notify: SEO lead and paid media lead. Maximum duration: 24 hours before escalation. Exit: error rate and CPU normal for 30 minutes, then remove rule and test a sample of URLs. Review: check Crawl Stats within 48 hours. Writing it down before an incident is the whole point; during an incident nobody remembers whether it was 429 or 503.

FAQ

What is the Retry-After header?

It is an HTTP response header that tells a client how long to wait before retrying. It can be a number of seconds or an absolute date. Google's crawl documentation now details it for 503 and 429 responses.

Did Google change how Googlebot behaves?

No evidence of that. The 6 October 2026 change is a documentation restructure that adds Retry-After detail and examples. Google already treated 500, 503 and 429 as slow-down signals.

Should I use 503 or 429?

Use 503 for maintenance or overload and 429 for rate limiting. Both can carry Retry-After. Either way, keep the window short.

How long can I leave it on?

Google suggests a couple of hours to one or two days. Longer, and URLs that keep returning these codes may drop from the index.

Will it lower my rankings?

A short window should not, but it delays discovery and updates. Persistent errors across several days put index inclusion at risk.

Does it affect Google Ads?

Yes. Google's guide says ads may stop serving and campaigns may be paused or cancelled while your site returns the errors.

Can I use it for only one section of my site?

Google says errors affect crawling for the whole hostname. If you need isolation, host heavy content on a separate subdomain beforehand.

Can I ask Google to crawl more?

Not through this mechanism. Google says it cannot increase crawl rate on request, though you can submit a special request to lower it.

Do other crawlers respect Retry-After?

Many well-behaved crawlers do, but I can't speak for every one. Check your logs.

Need a crawl and indexing health check?

I'm Younus Fardeen, an organic growth and SEO/AEO strategist with 4+ years of marketing experience in edtech and startups. If you'd like help turning technical updates like this into a plain-English runbook for your team, see my work and send a note through the contact form at https://younusfardeen.in.