#ai.txt

[mocha-tapioca]

drdt dev doing a banger job of making us all collectively go “?!?!?!?!?!?2!!£!2{3{2<382]{2!1>29§]£[~.@:2.3:$,$8#)1’wm?????????” everybody say thank you drdt dev

[gojoest]

[gojoest-main]

'i am hungry' girlfriend 🤝 'pulls out his dick' boyfriend

[guzsdaily]

robots.txt

Day 227 - Jun 19th, 12.024

Nowadays, with things like AI crawling the internet to collect training data, I think it would be good to give this tip I found/applied today to my website, and probably my future websites.

Context

If you ever adventure yourself on web development, you will probably one day probably find about web crawlers. In summary, they are bots that browses through the internet automatically, normally collecting and indexing data and the visited website's content. They are normally used by search engines, to better find and index results.

Nowadays, they are also used for AI training, and probably were being used before the AI hype. Said crawlers normally are used to retrieve the websites' contents so it can be used for AI training.

(You can see more about web crawlers on Wikipedia.)

The robots.txt File

One thing to note is that said crawlers visit sites randomly and without prior warning or notice. However, you can block a lot of these crawlers using a robots.txt file. This is a simple plaintext file that lists what paths in your website said crawlers can and cannot visit, so you can prevent parts of your website from being indexed or stored.

You can actually see any websites' robots.txt file easily, just add /robots.txt after the domain and there's it. Here is Tumblr's robots.txt file, as you can see, it is mostly to block search engine crawlers from indexing internal and non-content paths, but it also has thing like blocking the Google's Gemini (formerly Bard) crawler for training data.

Something to notice, is that the crawler has to respect the robots.txt file to the blocking to work. Unfortunately, most of them do not respect it, that's why a lot of websites such as Twitter have things like anti-bot and HTML obfuscation or blocking using JavaScript (with most crawlers do not support) to block said crawler.

(You can see more about robots.txt on Wikipedia.)

The Tip

Well, the whole post was to give context to something which I added to my website, a robots.txt file to block AI crawlers. Personally, I do not like the whole opt-out nature of data collection used by most AI companies, even though most of my content is licensed under Creative Commons licenses. Maybe one day I will change opinion, I really am questioning myself on this whole AI business that we are right now.

But, if you would also like to block AI crawlers in your website, I would suggest doing the same and use this repository which has a ready-to-go robots.txt file that blocks the currently known crawlers. It is well documented and also lists which crawlers respect or not said file. It also provides an ai.txt file, which you can also add, it is a similar format but for just AI model training, and it was created by Spawning.

In my website, I made it automatically fetch the raw version of the file, but you can just copy-paste on your static directory and you should be fine.

Again, these files are, in some sense, a polite ask, and do not actively block the crawlers. But I would say it's better than nothing and doesn't hurt to add y'know.

Today's artists & creative things Video: The Mind Electric (no glitch + original ending) (lyrics) - by MONO

© 2024 Gustavo "Guz" L. de Mello. Licensed under CC BY-SA 4.0

[saintobio]

[dasakuryo]

No matter how I say this, invariably some people will misconstrue it as me being rude, so whatever, I am just going to say it. Some USAmerican authors need to stop co-opting spaces that aren’t meant for them, and likewise be straightforward about that ethical responsibility, so their fans don’t enact that behaviour either. If I have to see yet another Reading Challenge in which the works of USAmerican authors are hailed as stories from/about X country, to detriment of works by writers from those actual countries, I am going to blow a fuse.

[kageyuji]

8.5k... golly thats like a tiny nation HEHDF all hail president lore ily babe kisses u im rly proud of u!! <333

omg i’m calling my followers kageyuji nation now 😋 ilyt bubs :(

[mafuki]

Happy new year, angels! I know I'm a little late, but I still wanted to wish you all the best. I wish for this new year to be full with lots of happiness and love to us all. Don't forget that I love you all so much! ♥️