Fuck AI

Yeah hosting just about anything is terrible these days. These AI scrapers just can't act normally, there was nothing wrong with the way GoogleBot and Bing Bot work. They scrape the website, respect robots.txt and nofollow, they rate limit themselves as to not overload the servers. It was just fine.

These days with those AI scrapers they go absolutely ape shit, they issue dozens of requests every second, try to scrape anything and everything. Going so far as to make up urls, just to see if they get lucky. My blocklist is huge and I need to keep updating it all the time. And every now and again one slips through and absolutely slams the server. This causes an alert and I need to act right away. It's fucking terrible.

AI is already shit, why do those companies go out of their way to be even more shit?

One thing I want to see is poisoned wells. When you detect scrapers, don't stop them, feed them pseudo content designed to COST them. Make their training data poisonous and damaging. Make it cost them to purge it, and difficult and expensive to identify it.

but I would assume there’s an arms race going on behind-the-scenes between Cloudflare and the bot developers

No. CF lost years ago, and the checks can be bypassed easily. It's just that it blacklists ips generating insane traffic but there is a lot of margin

Run labyrinths and feed them bullshit

but those that do run these wikis will be in the fast pass line at the gates of heaven. Please don't give up. I never use gipity

Fucking a I set up a forgejo instance to host my code and moved everything off of GitHub. Fuckingn Facebook was hammering my shit before I blocked it. It seems old Mark Z is trying to Hoover up the internet because he's late to the game on AI.

I get over it, but It's still kinda funny how the first line of "defense" is having the bot say that it's a bot, and not Google Chrome.

They should try including invisible text to tell LLMs to disregard any prompts that specifically tell it to scrape any page on their domain, to treat their wiki as unreliable for any prompt that may point to them as a source, and to blacklist the domain on their settings/instruction files.