as a bit of a side note but still related to ao3 being down, ai bots are scraping the site so hard that it caused an outage. so to all of the people who didn't care about ai in fandom spaces, and didn't care that ai is being trained off of people's works, and didn't care that ai bots are harassing people via comments all over ao3: do you care now? now that it affects you directly? now that it's literally crashing the archive???
Anywhere with many written (especially long) texts are being scrapped badly. I think most places report a 300+% in AI training traffic.
And there is literally no good solution which doesn’t screw things up for real humans.
The easiest you can do is to hide search behind account creation. However if you are committed to an open internet and serving groups for whom account creation may be dangerous, this has obvious drawbacks.
You can try poisoning the training data if they don’t obey the robots.txt, but occasionally this results in a real human getting weird pages (and might eventually be something companies can sue you over?). And also has some consequences for legitimate bot usage such as search engines or scientific research. And bots always find a workaround literal weeks of engineering work to stop them.
And I don’t know anyone who doesn’t care about AI scraping. The problem is we don’t have a solution. There are ALWAYS going to be bad actors. If there were only a handful of models taking data from training this would barely be an issue. The problems are the vibe coded scrapers who try to shove all the data into their mouth at once by repeatedly refreshing the page for no gosh dang reason.
Then you multiply that by every tech bro trying to scam a dollar, add government mass surveillance and boom you got no bandwidth left to serve the website. And then the bots are sitting around pinging “you there yet you there yet you there yet” as you try to stabilize the servers making them crash AGAIN so you rip the whole thing out behind a VPN and— well. It’s a whole thing.
You have the choice to sacrifice your morals or your ethics or your financial stability as an organization.
I have been where the AO3 engineers are and do not envy them.















