AI crawlers cause Wikimedia(The umbrella organization of Wikipedia and a dozen or so other crowdsourced knowledge projects) Commons bandwidth demands to surge 50%.
How crawlers impact the operations of the Wikimedia projects โ Diff
Doesn't make any sense. Why would you crawl wikipedia when you can just download a dump as a torrent ?
AI bros aren't that smart.
Apparently the dump doesn't include media, though there's ongoing discussion within wikimedia about changing that. It also seems likely to me that AI scrapers don't care about externalizing costs onto others if it might mean a competitive advantage (e.g. most recent data, not having to spend time and resources developing dedicated ingestion systems for specific sites).
I want to stress this: it's not that "tech bros" are just stupidโeven though a lot of them are revoltingly unappreciative of the giants whose sholders they stand onโit's that they don't care.
To have the most recent data?
To just have the most recent data within reasonable time frame is one thing. AI companies are like "I must have every single article within 5 minutes they get updated, or I'll throw my pacifier out of the pram". No regard for the considerations of the source sites.
There's a chance this isn't being done by someone who only wants Wikipedia's data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.
Feel like this belongs in !fuck_ai@lemmy.world
Think I should cross-post?
Go on, my brother.
Done.
what assholes .. just fucking download the full package and quit hitting the URL
Scraper bots don't read instructions, they just follow links
Rightโฝ This is ridiculously stupid when you can download the entirety of Wikipedia in a single package and parse it to your hearts desire
Not only that, but we make it goddamn trivial for not just Wikipedia but for other Wikimedia projects. Doing this is just stealing without attribution and share-alike like the CC BY-SA 4.0 license demands and then on top of that kicking down the ladder for people who actually want to use Wikimedia and not the hallucinatory slop they're trying to supplant it with. LLM companies have caused incalculable damage to critical thinking, the open web, the copyleft movement, and the climate.
Yay interrobang :D
The amount of stupid AI scraping behavior I see even on my small websites is ridiculous, they'll endlessly pound identical pages as fast as possible over an entire week, apparently not even checking if the contents changed. Probably some vibe coded shit that barely functions.
If I was running infra for them, Iโd just start blacklisting abusive IPs without warning
These fucking companies.. downing a torrent of annas archive but crawling wikipedia scourge of mankind
wikipedia should install ai mazes on their servers
Are there alternatives besides Cloudflare's solution?
Nepenthes does about the same thing but isn't managed by a corp.
Laws should be passed in all countries that AI crawlers should request permission before crawling whatever target site. I haver no pity to AI "thiefs" that get their models poisoned. F...ing plague, wasn't enough the adware and spyware...
An HTTP request is a request. Servers are free to rate limit or deny access
And Wikimedia, in particular, is all about publishing data under open licenses. They want the data to be downloaded and used by others. That's what it's for.
Rate limiting in itself requires resources that are not always available. For one thing you can only rate limit individuals you can identify so you need to keep data about past requests in memory and attach counters to them and even then that won't help if the requests come from IPs that are easily changed.
Bots lie about who they are, ignore robots.txt, and come from a gazillion different IPs.
i doubt the recent uptick in traffic is from โstealing dataโ for training but rather from agents scraping them for context, eg Edge Copilot, Googleโs AI search, SearchGPT, etc.
poisoning the data will likely not help in this situation since thereโs a human on the other side that will just do the same search again given unsatisfactory results. like how retries and timeouts can cause huge outages for web scale companies, poisoning search results will likely cause this type of traffic to increase and further increase the chances of DoS and higher bandwidth usage.
When I imagine a future with AI ruining the world, I always thought it was going to be some Skynet/CABAL/HAL9000 type of thing
Not this sad, boring, depressing type shit
We should stop using ai
I still struggle with a use case for artificial intelligence in my own life. I play around with it all and I'm just like, it doesn't do a good job. Also, I think humanity is missing the plot, you know? Like, we don't need government. If government isn't going to do government. Government serves the people, not corporations. Or at least it should. I don't know, I think we're entering in times. At some point, I think people will pray for nuclear war, because life will be so miserable. That it would be better than just to end it all.
AI has niches but they're exactly that: Niches. Small duct tape tasks for fudging over "hard problems" where manual code would result in a worse outcome and take far more time. Little esoteric problem spaces, which notably don't actually require you to use several states worth of electrical power training on a 50PB dataset of anime titties.
An example: I have a name generator in my game that strings together several consonant+vowel phoneme pairs into a name. This means that the names are always pronounceable, but often the spelling looks really unintuitive. Eg Joosiffe, which the player would likely pronounce as Joseph. However, the leap we do in our head between those two spellings is a process of declassifying phonemes and then re-classifying phonemes, and is actually a "hard problem" from a coding perspective due to the unintituive, multifarious complexities of written, spoken, and conceptualized human language. Adding this step to my name generator in code would be a project of it's own, larger than the game itself, and wouldn't ever work nearly as well as it needed to. But relatively small (30MB) AI models that do this with something like 99.8% satisfaction already exist. They didn't require a data center's worth of resources to train, and since they're academic projects they have licenses that allow them to be used for free in a game.
Iโm dyslexic and basically a terrible writer. It has helped my professional communication develop. It really helps me speed up my issues with my disability and feel confident in my communications.
Like, we don't need government.
Welcome to the anarchist. Now you have to pick your flavor! Social Anarcho-syndicalism, Anarcho comunist, anarcho-capitalism, anarcho christianis, and the list goes on!
I found LLMs helpfuls to develop some scripts and answer some simple trivial questions (like how does house property work in China). I could have looked for that in a regular search engine though. But that's it, I am still happy looking for things myself and investigating since you can't really trust their answers.
At some point, I think people will pray for nuclear war, because life will be so miserable.
Reminds me of Roll out the Fallout by The Chalkeaters
Actual AI?
Imagine your phone knows that you have a business meeting downtown today. It's already reserved a parking space for you, set your car to warm up before you leave and looped your contact in on your ETA, along with automatically notifying you of any delays. Then, your kid wakes up this morning in with a horrible toothache, you ask your phone what to do and it rings up your family dentist, who has a full schedule today, but makes you a referral nearby. You agree to try that other dentist today, and your AI books an appointment, checks your meeting today, coordinates with their AIs and approves a 15 minute delay so you can get to the dentist. It also notifies your kid's school of their absence and has their teachers AI automatically queued up to send transcripts, notes and homework assignmenta from today's classes.
That's the kind of stuff actual AI can do. Overgrown autocorrect? It's basically a multi-billion dollar Magic Eightball.
I don't know about stopping entirely. I built a pretty cool RAG system for internal use in my company, it very much facilitates navigating very large amounts of text data.
even better stop joining major platforms like social media and then they won't be able to create data sets. Be a leach, especially when they give it away for free, but don't contribute to the project. Understand how it works, sure. But it seems like most of humanity says they don't want something, yet they do the contrary. It's like we choose to comply before we even ask to comply for the fear of missing out. But if you look at what is today, what are you really missing out on?
So, uh. What about Lemmy?
They can also crawl this publically-accessible social media source for their data sets.
I'm on board with abandoning mainstream social media, but my point is that your suggestion would not solve the problem just relocate it. A better solution to the AI conglomerates stealing everyone's data from the open Internet is legislation and regulations - ie tackling the whole 'stealing data' component, along with stronger privacy regulations for everyone to make it harder for them to do the same in the future. It's nice seeing the EU taking some positive steps, but we will not see the US take any steps in that direction anytime soon, due to corporate capture of their politicians and the AI companies all being in the top 10 most wealthy companies in the US.
And the quality of the AI output sucks. I was recently looking for information about positive convention for yaw, pitch, and roll in aircraft. I was looking at az and yaw and got reasonable results from the AI, but when I looked at pitch and el all of the results were about elevator pitches. Even when I spelled out elevation it insisted on elevator pitches. I scroll past the AI results as a matter of principle, but I usually look at them so I have something specific to complain about when people ask why I am so virulently anti-AI.
The other day I tried to have it help me with a programming task on a personal project. I am an experienced programmer, but I only "get by" in Python (typically just by looking up the documentation for the standard library). I thought, "OK. This is it. I will ask Llama 3.3 and GPT4 for help."
That shit literally set me back a weekend. It gave me such bad approaches and answers, that I could tell were bad (aforementioned experience in programming, degree in comp sci, etc) that I got confused about writing Python. Had I just done what I usually do, which is to look up the documentation and use my brain, I would have gotten my weekend task done a whole weekend sooner.
It scares me to think what people are doing to themselves by relying on this, especially if they're novices.
It scares me to think what people are doing to themselves by relying on this, especially if they're novices.
Same here. There's a lot of denial going on but, LLMs are not good for anything that requires factual information. They likely will never be on account of just being statistical models for language. Summarizing long text where correctness isn't an issue is really one of the only places where I still think that they are good.
Search? Not if you want anything factual with citations.
Code? Fuck no. They constantly produce code of poor quality that may depend on non-existent libraries or functionality. More time it's spent debugging than writing code and it leaves the dev with a poor understanding of what the code actually does and ways to optimize/extend/etc.
Generating literary smut? Well, it's not going to do as good of a job as a person who can create something completely novel but can be passable without likely harm to authors (I'd classify it as a tier below erotic fan fiction).
AI is useful for basic, mundane tasks and that's about it. Trying to force it to be some sort of Uber search engine is such a bad idea.
I recently started as a graphic designer despite knowing absolutely nothing about it, so i am constantly searching how to do stuff in Adobe suite at work. Half the time Google's AI can't even keep "Cmnd" and "ctrl" straight, telling me to use' "cmnd+shift+H" on Windows or "ctrl+shift+H" on Mac'. I don't even know how it botches that, but it does it about 25% of the time.
Yea thatโs a bad example of what to use ai for at least right now. Youโre going to get bad results with that question.
Itโs good for things, if you pay.
This is an example of corporate terrorism sponsored by our own government. Elon Musk loves to see himself as the villain in Ready Player One. And this is not a joke you can look it up. Big tech is waging war against American citizens, and no longer do we have any control of our government, and the Democrats will not save us. The electoral processes will not save us. This is just hard for some people to accept, that's why things have to fall apart before they get a clue. Unfortunately, those that are wiser are going to feel the flames first.
Life is "simpler" when you realize governments (and hence the State) are just bullies that exercise their power through physical violence or economical violence.
Governments aren't fundamentally bad. Having a governing body along with laws and regulations is a good thing when done beneficently. For example, government is responsible for access to public education, libraries, banking, worker safety, and hospitals - all of which are objectively good things to have as a society. The problems usually occur when some individuals have more power/influence than others to choose what the government does, which is what's happening in much of the world right now.
AI: The "pen that can write in zero gravity" when pencils exist.
Well I get the analogy, but also I think they didn't use pencils because of the graphite and complications with filtering air or something.
Graphite is conductive. A short circuit and fire are Very Bad.
You may be right. It's just easier to get the sentiment across that way than expound about how it's ridiculously complex and overbuilt to achieve menial results.
Also, all pens work in zero gravity. They don't rely on gravity to feed ink to the point, try writing on a piece of paper that's held against the ceiling and it works just fine.
This pen / pencil thing has been corrected so many times for so many decades that it's ludicrous people are still bringing it up.
https://www.scientificamerican.com/article/fact-or-fiction-nasa-spen/
Random bits of pencil lead floating around in a high tech environment is such a poor idea that even the Soviet's quit using pencils once Fisher's Space Pen was available. A pen which Fisher itself paid to develop and then sold to both NASA and the Soviet Space Program.
Yeah, I know it's not precisely correct, but it's a fable that's commonly understood as an example of over-engineering. I'm open to better and more factual examples, if you have any!
Pen, pencil. Both are dangerous in the wrong hands.
Support Wikipedia! They're awesome and are our backbone.
Artificial intelligence is really just a Swiss army, knife, version of a search engine. It serves the interest of the ruling elite and is a way the uber wealthy can make America stupid again or even dumber than dumb. Like we move forward fast but we always miss the point and we're always putting out fires that once were easily manageable. The code base is growing we're all saying a lot of nothing. It's really just drowning out voices and good ideas just to centralize everything. It's about power. It's about control. It's about the man behind the curtain in the wizard of Oz. I like technology a lot and I would like it to be in the hands of working people and not the uber wealthy. So I am no techno optimist at all, not until we fix that issue.
To be clear, network costs represent a tiny fraction of WMF's expenses. Much of WMF's budget goes to social programs, not technical upkeep.