Google will no longer back up the Internet: Cached webpages are dead
Google will no longer back up the Internet: Cached webpages are dead
Google Search will no longer make site backups while crawling the web.
I think we can safely say that "web page don't go down anymore" is not the real reason. Two possibilities I see for the real reason:
- They don't feel it's necessary anymore to cache the actual pages. It's not productive to have them to track and analyze SEO abuse, or even just to tune their algorithm by rescanning things if someone notices there's a problem, even given that it costs them a fraction of a fraction of a fraction of the income they make from search, and it makes life easier for anyone who's working on it.
- They used to make it available just because it was a neat thing to be able to offer and there was no reason not to, since they were caching the pages anyway. But, the current Google management has wandered so far afield of the original mindset that made them successful in the first place that that means nothing to them, more like a bizarre confusion of concepts than any kind of statement that makes coherent sense. So fuck the users. Yes we're still caching them, because the search engine needs them. No you can't have them for free, you fucking hippie. Now get out.
I know which explanation I favor.
Definitely B
This is probably an attempt to save money on storage costs. Expect cloud storage pricing from Google to continue to rise as they reallocate spending towards ML hardware accelerators.
Never been happier to have a proper NAS setup with offsite backup 🙃
I don't think they are going to stop storing it somewhere, just stop delivering it.
Idk… in theory they probably don’t need to store a full copy of the page for indexing, and could move to a more data-efficient format if they do. Also, not serving it means they don’t need to replicate the data to as many serving regions.
But I’m just speculating here. Don’t know how the indexing/crawling process works at Google’s scale.
Seems unlikely they'll deleted it. If they're started deleting data that's quite a change. They might save from bandwidth costs of delivering it to people I suppose.
Maybe something to do with users filling the AIs from the google cache? Google wanting to ensure only they can train from the google-cache.
@kubica@kbin.social @Powderhorn@beehaw.org @rho50@lemmy.nz
I think it's more about the web visitor cost. Handling traffic and API calls becomes a financial problem when there are a growing number of companies using bots to scrape data. Larger companies are moving their content behind paywalls, which acts as a bot filter, and have also identified that they can generate a revenue stream from subscriptions and API connections. Old content on the web is not deemed to have much business value, so it's a decision of either charging for it or scrapping it.
This is probably an attempt to save money on storage costs.
That’s in fact what the article claims as Google’s reason. But seems irrational. Google still needs to index websites for the search engine. So the storage is still needed since the data collection is still needed. The only difference (AFAICT) is Google is simply not sharing that data. Also, there are bigger pots of money in play than piddly storage costs.
Google never backed up the internet, sure they did cache pages, but that isn't even close to backing up the internet
Google "Search Liaison" Danny Sullivan confirmed the feature removal in an X post, saying the feature "was meant for helping people access pages when way back, you often couldn't depend on a page loading. These days, things have greatly improved. So, it was decided to retire it."
okay but... has it? this seems like an unfounded premise, intuitively speaking
"What excuse could we use for this cost-cutting measure?"
"Uh, we could just say that people don't need it anymore."
"Johnson, get that man a promotion!"
"Are you sure? A promotion will cost us more money."
"Oh right, fire that man!"
Yea I've been using it more and more recently, although part of that is sites like Twitter or Reddit randomly hiding content
Bingo. When I read that part of the article, I felt insulted. People see the web getting increasingly enshitified and less accessible. The increased need for cached pages has justified the existence of 12ft.io.
~40% of my web access is now dependant on archive.org and 12ft.io.
So yes, Google is obviously bullshitting. Clearly there is a real reason for nixing cached pages and Google is concealing that reason.
Not dead, just non-public.
AIs are hungry, the value of stored shitposts skyrocketed.
There's ways to rate limit, like increasing response time per IP address per hour to make rapid, massed requests slower and easier to handle. Taking them all down at once is an extreme move.
"Enshittification" isn't just company did bad thing, you know
It isn't, but I think this probably fits. Enshittification is when a company provides useful, good services to gain users, then once those users are locked in they start degrading those service or removing features to cut costs, right? That seems like a pretty close analogy to what's going on here, I'd think.
google be like stop bypassing paywalls and ads using our cache
Wait, how does Google make money off of paywalled contents?
Maybe they don't want to give rival AI devs data access? It's not typical for Google to give up data.
As far as we know, Google is not giving up any data. The crawler still must store a copy of the text for the index. The only certainty we have is that Google is no longer sharing it.
From the article:
"was meant for helping people access pages when way back, you often couldn't depend on a page loading. These days, things have greatly improved. So, it was decided to retire it." (emphasis added)
Bullshit! The web gets increasingly enshitified and content is less accessible every day.
For now, you can still build your own cache links even without the button, just by going to "https://webcache.googleusercontent.com/search?q=cache:" plus a website URL, or by typing "cache:" plus a URL into Google Search.
You can also use 12ft.io.
Cached links were great if the website was down or quickly changed, but they also gave some insight over the years about how the "Google Bot" web crawler views the web. … A lot of Google Bot details are shrouded in secrecy to hide from SEO spammers, but you could learn a lot by investigating what cached pages look like.
Okay, so there’s a more plausible theory about the real reason for this move. Google may be trying to increase the secrecy of how its crawler functions.
The pages aren't necessarily rendered like how you would expect.
More importantly, they don’t render the way authors expect. And that’s a fucking good thing! It’s how caching helps give us some escape from enshification. From the 12ft.io faq:
“Prepend 12ft.io/ to the URL webpage, and we'll try our best to remove the popups, ads, and other visual distractions.”
It also circumvents #paywalls. No doubt there must be legal pressure on Google from angry website owners who want to force their content to come with garbage.
The death of cached sites will mean the Internet Archive has a larger burden of archiving and tracking changes on the world's webpages.
The possibly good news is that Google’s role shrinks a bit. Any Google shrinkage is a good outcome overall. But there is a concerning relationship between archive.org and Cloudflare. I depend heavily on archive.org largely because Cloudflare has broken ~25% of the web. The day #InternetArchive becomes Cloudflared itself, we’re fucked.
We need several non-profits to archive the web in parallel redundancy with archive.org.
Is there any sort of way to self host a limited version of this? L
I’d love to be able to have my own Searx also cache everything I visit as I go to it, it’d at least let me refind information I’ve previously found.
You want to self host... the Internet?
You could use something like archivebox as that saves the whole page, or you could use Waybackmachine and force it to save the page via an add-on.
You could also setup your own yacy index and everytime you find an interesting site you could add it to yacy.
But this is kind of not what you are asking for. Archivebox is probably the closest, or using squidcache and literally caching every url you go to. 😅
Here’s the heart of the not-so-obvious problem:
Websites treat the Google crawler like a 1st class citizen. Paywalls give Google unpaid junk-free access. Then Google search results direct people to a website that treats humans differently (worse). So Google users are led to sites they cannot access. The heart of the problem is access inequality. Google effectively serves to refer people to sites that are not publicly accessible.
I do not want to see search results I cannot access. Google cache was the equalizer that neutralizes that problem. Now that problem is back in our face.
🤖 I'm a bot that provides automatic summaries for articles:
Click here to see the summary
Google Search's "cached" links have long been an alternative way to load a website that was down or had changed, but now the company is killing them off.
The feature has been appearing and disappearing for some people since December, and currently, we don't see any cache links in Google Search.
Cached links used to live under the drop-down menu next to every search result on Google's page.
As the Google web crawler scoured the Internet for new and updated webpages, it would also save a copy of whatever it was seeing.
That quickly led to Google having a backup of basically the entire Internet, using what was probably an uncountable number of petabytes of data.
In 2020, Google switched to mobile-by-default, so for instance, if you visit that cached Ars link from earlier, you get the mobile site.
Saved 68% of original text.
Never seemed feasible to begin with.