#web crawler

[tulipterrace]

WHITE KNIGHT (they/them) and WEB CRAWLER (they/she) (just born today) say hi to them

[hjbender]

The AO3 kudos bot has been hitting my thorki fics, particularly my In Side Out series, pretty consistently for a while now—I don’t know for how long; it’s 3-4 kudos daily on average, which didn’t grab my attention like some people who have reported guest kudos in the double or triple digits.

I hate it, but I went ahead and locked those fics (and a couple new ones whose kudos counts seemed disproportionate). Hopefully it’ll break the cycle.

Kind of annoyed that the ratio on my most popular works are now meaningless, but I didn’t even know about the AO3 bot—or crawler, or whatever it is—until last week, so consider this a PSA to all AO3 authors who may not already know. Check your kudos, or make sure you've got the "Turn off emails about kudos" unselected in your preferences. If you’ve been getting small but daily kudos from guests over a long period of time, like me (I had to check my AO3 kudos notification emails over the last month to verify), you might be a victim.

Anyway, I've got some spare invites if anyone needs/wants an AO3 account. Just send me an ask 👍

[shironezuninja]

I’ve always wanted a role model who endures the burden of responsibility on their shoulders. Because I’ve been running away from that unknown future all my life.

[between-awesome-and-awkward]

Your friendly classic spider suit

[impeterperez]

Spider-Man: Homecoming (2017)

[somenerdthing]

[hjbender]

It’s really sad. Before the dreaded kudos web crawler began plaguing AO3 (and my works), I used to welcome guest readers, their comments and kudos—and I still do, when they’re real people.

But now whenever I get kudos notification emails from AO3, I follow the link for every work that has a guest kudos, and if the ratio of guest to user kudos looks suspect, I lock that work. I really hate to do that, but it’s the only way to shield myself from this web crawler.

And just when I think I’ve outsmarted the nasty thing, a fresh crop of guest kudos appear on my unlocked works. This is happening on old works as well as new works. And it’s really insidious, too. One to 3 kudos every day, which I didn’t notice was happening for over a year. That adds up to a lot of fake kudos over time.

My problem is: I don’t want to lock every work I’ve got. I guess I just need to stop caring about kudos—and I hate that because, as a writer, I use the kudos-to-hits metric as a gauge for how well I nailed a certain character or theme or trope (after factoring in things like fandom size and ship popularity). I’m sure a lot of us do that as a measure for our writing skills. So this crawler situation just… sucks in the worst way.

[danielslegacycomputercollections]

"At Dataprovider.com we transform the internet into a structured database of web data."

"Our crawlers are transparent by design: they identify themselves, respect robots.txt, and follow strict ethical guidelines."

...or so B.V. Dataprovider says...

Balderdash. Poppycock. Hogwash. Fiddle-faddle...

Their "crawlers" weren't respecting anything when they tried accessing the items in the picture above. Their attempts netted them a big nothing, because I took down my DLCC website's content last Halloween. 😁

And, "Canada???" What respect are they showing by trying to conceal their true location? Naughty, naughty... 😆

😁