Do you believe anything coming out of OpenAI when it's abundantly clear that they'll say anything to protect their bottom line.
OpenAI are not the only people harvesting data and selling it to interested parties.
There is no legal requirement to adhere to the standard and I'd be shocked if any court in the USA could understand the issue, let alone enforce a voluntary standard.
The amount of automated data collection online is staggering. On my own services it accounts for 50% of the hits. Good luck with policing that.
It's not about relying on it, it's about changing the behaviour of web crawlers that respect 'em, which, as someone who has adminned a couple scarily popular sites over the years, is a surprisingly high percentage of them.
If someone wants to get around it, they obviously can, but this is true of basically all protective measures ever. Doesn't make them pointless.
I too am a fan of privacy, but in the other hand… I kind of like the idea that we’re training AIs to convince people to become communists and/or change their gender
No, robots.txt doesn’t solve this problem. Scrapers just ignore it. The idea behind robots.txt was to be nice to the poor google web crawlers and direct them away from useless stuff that it was a waste to index.
They could still be fastidious and follow every link, they’d just be ignoring the “nothing to see here” signs.
You beat scrapers with recursive loops of links that start from 4pt black on black divs whose page content isn’t easily told apart from useful human created content.