Why Google Ignores Your Last-Modified Headers

Published Read 7 min read

On this page

Google does not meaningfully use your server’s Last-Modified header to decide when to recrawl a page, because the header is trivially gamed and frequently misconfigured. A CMS can stamp it with the current time on every request, and many servers default it to the last server-restart time, so the value carries no reliable information about whether your content actually changed. Instead, Google detects freshness by comparing the rendered content of a page across crawls, treats the lastmod date in your sitemap as a hint weighted by how accurate your past lastmod values have been, and prioritizes what to crawl by page-importance signals like links, traffic, and historical change frequency. The real levers for getting an updated page recrawled are site architecture and accurate sitemaps, not headers.

The mental model to adopt: headers are claims, content comparison is evidence, and Google prefers evidence. This is a crawl-scheduling and change-detection post. It is not about deleted URLs being recrawled, nor about a cache serving stale content to the crawler.

Why the header is distrusted

The Last-Modified response header was designed for HTTP caching, telling a browser or proxy whether its cached copy is still current, not for telling a search engine that your article got better. Two failure modes make it useless as a freshness signal.

First, it is gameable. A site can set Last-Modified to “now” on every request, which would let anyone claim constant freshness without changing anything. A signal that can be set arbitrarily by the site is a signal Google cannot trust at face value.

Second, it is often simply wrong. Many server and CMS configurations set Last-Modified to the time the server process last started, or to a build timestamp, so it changes on every deploy regardless of whether any individual page’s content changed. Faced with a header that is either gameable or misconfigured most of the time, Google leans on something it can verify itself.

What Google actually uses

The primary mechanism is content comparison between crawls. Google keeps a representation of what a page contained last time it fetched it and compares the new fetch against it. A substantive change to the main content, the structured data, or the links registers as a real update. A cosmetic change, a rotating ad, a “last updated” date in the footer with no body changes, or an auto-updating copyright year, does not register as significant, because the meaningful content is identical. This is why bumping a visible date without editing the page rarely triggers anything.

Alongside content comparison, Google uses historical change frequency (pages that change often get checked more often) and importance signals (heavily linked, high-traffic pages get crawled more aggressively than obscure ones). Crawl scheduling is a budget allocated by these factors, not a queue you can jump with a header.

Sitemap lastmod is a hint, and trust is earned

The one freshness input you control that Google does use is the lastmod value in your XML sitemap, and it is a hint, not a directive. Its weight depends on whether your lastmod values have historically been accurate. If you only move lastmod when a page’s main content meaningfully changes, Google learns your dates are reliable and treats them as a useful prioritization signal. If you bump lastmod on every page on every build, or set them all to today, Google learns your dates are noise and starts to discount them.

The practical rule follows directly: only update sitemap lastmod on a substantive change. An update to the main content, the structured data, or the links is significant; a copyright-year bump is not. Treating lastmod as an honest record is what keeps it valuable.

You cannot force a higher crawl rate

There is no lever that makes Google crawl a page more often on demand. The Search Console crawl-rate limiter tool, which only ever let you cap the maximum rate, was retired on January 8, 2024; Google now automatically manages crawl rate, slowing down when your server returns errors or responds slowly and speeding up when it can. So even the old tool only set a ceiling, never a floor, and it no longer exists.

For an individual urgent update, the Request Indexing button in the URL Inspection tool asks Google to recrawl that one URL. It works per URL and does not scale to a whole site, so treat it as a tool for a handful of priority pages, not a freshness strategy.

What counts as a change worth signaling

Since content comparison drives detection and sitemap trust depends on accurate lastmod, it helps to be precise about what Google treats as a substantive change. An edit to the main content (rewriting sections, adding new information, materially changing the answer), a change to the structured data, or a change to the links on the page generally registers as significant. These alter what the page means or how it connects to the rest of the site, which is what a freshness signal is supposed to capture.

What does not count: an auto-updating copyright year, a rotating testimonial or ad slot, a “last reviewed” stamp with no body edits, or a build that changes a timestamp without touching the content. If you move sitemap lastmod for these, you are telling Google the page is fresh when its meaning is unchanged, and the content comparison on the next crawl will contradict your claim. Do that across many pages and Google learns your dates do not correspond to real changes, which is how a site trains Google to discount its lastmod entirely. The discipline is simple: treat lastmod as a record of meaningful edits, and your dates stay worth something.

The levers that actually work

Internal linking is the strongest controllable input. Linking to an important updated page from your high-traffic, frequently-crawled pages (the homepage, hub pages, recent posts) puts that updated URL in front of Googlebot sooner, because those source pages are crawled often and the renderer follows their links.

Accurate, segmented sitemaps are the second lever. Keep lastmod honest, and split sitemaps by type so freshness-sensitive sections are easy for Google to prioritize. Publishers should maintain a dedicated news sitemap with real publication dates, which is the one place timely lastmod-style signaling carries weight, separate from the evergreen sitemap. Strong external links to updated content help too, by raising the page’s overall importance and therefore its baseline crawl frequency. None of these is a header.

A third lever is keeping the page reachable and fast. Crawl scheduling responds to how your server performs: Google speeds up when the site answers quickly and without errors, and backs off when responses are slow or error out, so a flaky host caps the crawl rate you could otherwise earn. Returning a clean 200 on the canonical URL and avoiding redirect chains that make Googlebot follow several requests to reach the final page both remove friction from the recrawl. The throughline across every real lever is the same: you influence recrawl by making a page more important and easier to reach, not by asserting a date the crawler has no reason to believe.

A worked example

Picture a guide you substantially rewrote this week. Bumping the footer’s “last updated” line and pushing a new Last-Modified header changes nothing Google can act on, because neither survives the trust test. What moves the needle is concrete: link to the rewritten page from your homepage so Googlebot encounters it on its next frequent crawl, set the sitemap lastmod to the actual edit date because the edit was real, and, if the update is time-critical, submit the single URL through URL Inspection. Each gives Google verifiable evidence or a stronger reason to fetch.

Frequently Asked Questions

If I update my Last-Modified header, will Google recrawl the page faster?

No. Google does not treat Last-Modified as a reliable freshness signal because it is easily gamed and commonly misconfigured to the server-restart time. Keep the header for HTTP caching, and influence recrawl through internal links from frequently-crawled pages and an accurate sitemap lastmod instead.

Does updating the date in my sitemap force Google to recrawl?

Sitemap lastmod is a hint, not a directive, and its influence depends on whether your past lastmod values were accurate. Only move it on a substantive content change; if you bump it without real changes, Google learns to ignore your dates, which weakens the one freshness signal you control.

Sources

Google Search Central, “Build and submit a sitemap” (lastmod guidance): https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
Google Search Central Blog, “Upcoming deprecation of the Crawl Rate Limiter Tool in Search Console”: https://developers.google.com/search/blog/2023/11/sc-crawl-limiter-byebye

Related posts:

  1. Original Research as an SEO Moat Original research is defensible for one reason: the asset is data your competitors cannot replicate. Informational content gets reverse-engineered within weeks, because anyone can read the same sources and rewrite......
  2. SEO and Engineering Collaboration: Communication Protocols for Technical Requests SEO requests stall not because engineers undervalue search but because the requests arrive in a form engineering cannot consume. An email saying "we need to fix our canonical tags for......
  3. How to Recover from a Google Penalty Without Knowing the Cause "A penalty I can't explain" is a diagnosis problem before it is a recovery problem, and the first branch settles most of it: open Security and Manual Actions in Search......
  4. How to Build a Multilingual SEO Strategy The strategic decisions come before any hreflang tag, and getting them right is most of the work. Multilingual SEO is a sequence of investment choices: which languages to build for,......
  5. Old Blog Posts Are Cannibalizing New Content Your new post is objectively better and it still cannot outrank the old one you wrote three years ago, because Google ranks the URL it already trusts. Accumulated authority signals......
  6. How to Do SEO for a New Website with Zero Budget Zero-budget SEO is not a cheaper version of funded SEO; it is a different strategy that trades money for time and scope. Money buys speed, scale, and tooling. Without it,......
  7. Third-Party Scripts Are Blocking Your Core Content from Rendering Synchronous, render-blocking third-party scripts, analytics, chat widgets, recommendation engines, personalization tags, loaded in the head per the vendor's default snippet, occupy the single JavaScript thread and consume the finite rendering......
  8. Forum SEO After Google’s Parasite Update If you run a forum and you have been told that adding editorial oversight to sponsored or placed content keeps you safe under Google's site reputation abuse policy, that advice......
  9. Google Crawls Daily But Your New Pages Take Months to Index Crawl frequency and index inclusion are two decoupled systems, and the daily hits you see in your logs are almost entirely Google rechecking established pages it already trusts. That activity......
  10. Your Site Got Caught in a Neighborhood Penalty There is no discrete "neighborhood penalty" in Google's documented systems. What practitioners describe with that phrase is almost always one of two different things: Google's link-spam systems algorithmically neutralizing the......
  11. Core Web Vitals Green But Rankings Still Dropping All-green Core Web Vitals and falling rankings is not a contradiction; it is a diagnosis. Passing CWV is a low-weight page-experience input that keeps you out of a hole, not......
  12. SEO Content Briefs: Templates That Produce First-Draft-Ready Articles A brief that yields first-draft-ready work occupies a deliberate middle ground: enough direction that a competent writer can execute autonomously, never so much prescription that it flattens their voice into......
  13. Your Inline CSS Is Making Pages Look Identical to Googlebot The inlined critical CSS is not the cause of your duplicate-canonical problem. The cause is that you serve a near-identical template shell, the same header, nav, hero, and inlined CSS......
  14. Google Demoted Your Page for a Schema Error You Can’t Find Your schema passes every validator, the Rich Results Test reports no errors, and yet the rich results are gone and there is no manual action in Search Console to appeal.......
  15. How to Do SEO When Rebranding (Same Domain) A same-domain rebrand leaves your non-branded rankings essentially intact, because the things that earn them, domain authority, backlinks, URLs, and on-page topical content, do not change when only the name......