Why Google Ignores Your Last-Modified Headers

Published Read 5 min read

On this page

If your server sends both an ETag and a Last-Modified header, Google’s crawlers use the ETag. The documentation states this outright, and the rule follows the HTTP standard. When there is no ETag, Google’s overview of its crawlers says the infrastructure supports Last-Modified and If-Modified-Since for HTTP caching. Google’s documentation describes the header as a caching tool; it doesn’t describe it as a signal for when to crawl. It lets Google skip downloading a page that hasn’t changed. If an updated page isn’t being recrawled, the header is the wrong lever. The sitemap date, links and the server are better levers.

What Last-Modified is for

Last-Modified is an HTTP caching header. When a crawler has fetched a page before, it can send the stored date back in an If-Modified-Since request. If the page hasn’t changed, the server can answer with a 304 Not Modified status instead of sending the page again. Google’s crawl budget guide recommends supporting 304 responses: a 304 tells Google to reuse the cached version, which saves your server bandwidth and resources.

Google’s crawler documentation sets out how it treats the caching headers:

  • Both headers are supported. Google’s crawling infrastructure supports ETag with If-None-Match, and Last-Modified with If-Modified-Since, as the HTTP caching standard defines them. Other HTTP caching directives aren’t supported.
  • ETag wins when both are present. If a response carries both, Google’s crawlers use the ETag value, as the HTTP standard requires.
  • Google prefers ETag. It recommends ETag over Last-Modified for its crawlers because ETag doesn’t have date formatting issues. It also suggests setting both anyway, regardless of its own preference.
  • The date format matters. The date in Last-Modified must follow the HTTP standard. To avoid parsing issues, Google recommends the format “Weekday, DD Mon YYYY HH:MM:SS Timezone”, for example “Fri, 4 Sep 1998 19:15:56 GMT”.

So there are two documented ways your Last-Modified header can end up with no effect on Google: an ETag in the same response takes precedence, or a date Google cannot parse.

One header setting Google says can help recrawl timing

Google’s crawler documentation adds a suggestion alongside Last-Modified. While not required, it says, consider setting the max-age field of the Cache-Control header to help crawlers determine when to recrawl the specific URL. Set it to the number of seconds you expect the content to stay unchanged, for example Cache-Control: max-age=94043.

That is a hint about how long a page is expected to stay the same. It is not a command to crawl.

What decides when Google recrawls

Google’s crawl budget guide describes crawl demand in terms of the site, not headers. Three factors it names bear on recrawl timing:

  • Perceived inventory. Without guidance, Google tries to crawl all or most of the URLs it knows on your site.
  • Popularity. URLs that are more popular on the internet tend to be crawled more often, to keep them fresher in Google’s systems.
  • Staleness. Google’s systems want to recrawl documents often enough to pick up changes.

The other half is capacity. Google adjusts how fast it crawls to how your server responds. When Google announced that the Search Console crawl rate limiter tool would be retired on January 8, 2024, it said that if a server persistently returns HTTP 500 errors for a range of URLs, Googlebot will slow down crawling automatically and almost immediately. That tool let you limit the rate, and it no longer exists.

The sitemap date Google does use, on one condition

The date input Google documents for recrawling is lastmod in your XML sitemap. Google’s guide to building a sitemap says Google uses the lastmod value if it is consistently and verifiably accurate, for example when compared with the page’s last modification. The value should reflect the date and time of the last significant update.

Google also defines significant. An update to the main content, the structured data or the links on the page is generally significant. An update to the copyright date is not. The same guide says Google ignores the priority and changefreq values.

The practical rule follows: move lastmod only when the page changes in one of those ways. A site that stamps every URL with today’s date on every build publishes dates that don’t match its pages. It has no reason to expect Google to treat them as accurate.

What to do after a real update

For a page you have rewritten:

  1. Set lastmod to the real edit date in the sitemap, and keep the rest of the sitemap’s dates honest so the file stays one Google can rely on.
  2. Link to the page from pages Google revisits, such as the home page, section hubs and recent articles. That gives Googlebot a link to the updated URL on its next visit to those pages.
  3. For a single urgent page, request indexing in the URL Inspection tool. Google says a request does not guarantee indexing, and there is a daily limit, so keep it for a handful of priority URLs.
  4. Keep the server fast and the response clean. Return a 200 on the canonical URL, avoid redirect chains, and support 304 responses so unchanged pages cost little to recheck.

And keep sending caching headers. A correct ETag, a correctly formatted Last-Modified and a sensible Cache-Control: max-age all do their documented jobs. None of them is documented as scheduling a crawl.

Frequently asked questions

Does Google use the Last-Modified header at all?

Yes, for HTTP caching. Google’s crawlers support Last-Modified with If-Modified-Since. If the response also has an ETag, the crawlers use the ETag, and Google recommends ETag because it avoids date formatting issues.

Will updating Last-Modified make Google recrawl a page sooner?

No. Google documents the header for reusing a cached copy of an unchanged page and doesn’t describe it as scheduling crawls. Google does say a Cache-Control max-age value can help crawlers determine when to recrawl a URL.

Is a new sitemap date enough to force a recrawl?

No. Google uses lastmod when it is consistently and verifiably accurate. Change it only on a significant update to the main content, structured data or links.

Can I make Google crawl my site faster?

Not by a setting. The crawl rate limiter tool was retired on January 8, 2024, and it only ever set a ceiling. Google adjusts crawling to how your server responds, and crawl demand depends on factors such as a URL’s popularity and staleness.

Related posts:

  1. How to Fix a Site Hit by Multiple Algorithm Updates A site that lost ground in several updates can have more than one problem, and a costly mistake is treating them as one and applying one fix. Recovery starts with......
  2. What Is Search Intent and How to Optimize for It Search intent is the goal behind a query: what the person wants when they type or speak it. Much of the SEO industry sorts it into four types, informational, navigational,......
  3. How to Write Meta Descriptions That Increase Click-Through Rate Start with how Google builds the text under your result. Its documentation on snippets says Google primarily uses the content on the page to determine the snippet automatically, and may......
  4. SEO Team Capacity Planning: Workload Modeling and Resource Allocation Capacity planning is a constraint that pushes a team toward real prioritization. SEO work has no natural end: there is always another page to improve, another competitor to study, another......
  5. How to Use Google Search Console for SEO Search Console is Google's own record of how it crawls, indexes and shows your site, and its value comes from running repeatable routines on a handful of reports. The routine......
  6. SEO Competitive Response Playbook: Reacting to Competitor Moves Competitive SEO response is triage, not reflex. When a competitor publishes a new hub, ships a technical change or picks up a burst of links, the first action is not......
  7. Why Query Parameters After Hash Symbols Don’t Exist to Google Everything after the # in a URL is a fragment, and the browser doesn't send it to the server. So when a filter lives after the hash, /properties#bedrooms=3 and /properties#city=seattle......
  8. SEO Process Documentation: Playbooks, SOPs, and Knowledge Management Documentation earns its keep when following it is cheaper and safer than doing the work from memory. Building for completeness can work against that test. Pages nobody opens still need......
  9. How to Optimize for Voice Search Google documents very little that is specific to voice, and that tells you where the work is. The voice-specific feature in its structured data guidance, speakable, covers news articles read......
  10. How to Recover from Site-Wide Quality Demotion When a whole domain loses rankings at once, Search Console lists no manual action, and no technical or security cause explains it, the cause sits in Google's ranking systems, not......
  11. How to Optimize for People Also Ask Boxes Google says little about People Also Ask, and what it says is useful. Search Console's help on how clicks, impressions and position are counted describes "People also ask" sections as......
  12. Why Is My Google Business Profile Suspended and How to Fix It A suspended Business Profile is a moderation action, and the first job is to find out which kind: a restriction on the profile itself, or on the Google Account that......
  13. SEO Traffic Forecasting: Building Predictive Models with Historical Data A defensible SEO forecast is a decomposition problem, not a prediction stunt. You separate three things in your history, the underlying trend, the repeating seasonal pattern and the unexplained residual,......
  14. Template Libraries as Organic Traffic Machines A template library can grow organic traffic by meeting a search that blog content can't: "I need this now," where the searcher wants a ready-to-use file rather than an explanation.......
  15. How to Optimize for Google SGE / AI Overviews SGE, the Search Generative Experience, is what Google called its generative search before AI Overviews replaced it. The features you optimize for today are AI Overviews and AI Mode. Google's......

Leave a comment

Your email address will not be published. Required fields are marked *