Cloudflare trying to stop AI crawling somehow!
Seeing as how they can't reliably detect that I'm human or not, I don't have much confidence in this.
this post was submitted on 01 Jul 2025
323 points (99.4% liked)
Selfhosted
44306 readers
1054 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
Yeah. Me choosing to use a vpn and a privacy respecting browser has earnt me a constant captcha
For me just using Firefox on Linux seems to be enough to trigger them.
Apple’s private relay does this too. And so does auto-login.
It's relatively easy for Cloudflare to profile clients as being web scrapers. A concerning amount of internet traffic goes through their servers in plain text.
Is that why I can no longer go from a web search (eg.: DDG, Ecosia) or forum link to StackOverflow without going through three CF captchas? If AI had not killed SO for me before, this does.
Yeah, it's only anecdotal but I feel like hobbyists like us, who do slightly unusual things without nefarious intent, who are the ones who get hit with these sorts of issues the most. For example, I've noticed that some websites start throwing captchas at me or even just straight-up refuse to load with 403: unauthorized errors because I have my router set up to load-balance across two Internet connections. (At least, that's my guess as to why it's happening.)
For example, I’ve noticed that some websites start throwing captchas at me or even just straight-up refuse to load with 403: unauthorized errors because I have my router set up to load-balance across two Internet connections. (At least, that’s my guess as to why it’s happening.)
I maintain several multi-wan commercial setups and they don't have this problem. I obviously don't know what your setup is but I'd guess something is wrong with how its handling flows / connections. Once a connection is established between your edge and an internet resource that flow should remain "stuck" to whatever wan port it started with and it sounds like that isn't happening.
Could very well be. I'm using OpenWRT and basically did the bare minimum to get it to work.
Ahh yes. Imgur simply don't work anymore at my place, it always errors out with 403.
I've seen captchas for years before the recent influx of AI. It's the way I go about obfuscating network activities that the site security cannot determine if I am a bot on not. There is a Captcha Buster extension for Firefox. If the captcha is 'Pick the three busses from these blurry, pixelated set of pictures' then I can solve those easily. It's when the captcha is a full page of a motorcycle and you have to check all the relevant pieces, then on to the next full picture, that chap me. So you click Captcha Buddy and it 'listens' to the audio portion of the captcha, then solves it. It's not 100% on all types of captchas, but it 90% of the time it works every time. It's interesting to me that after a while, you start to notice patterns in the captcha images. For instance if the directions are 'Pick the fire hydrants', there will be at least 5 you have to pick. Crosswalks are the same way too.
I'd much rather have to do captchas than have my jimmy out in the ether traffic. Anecdotal, but Stack Overflow doesn't trigger a captcha for me. All I get is the cookie popup.
@irmadlad @lambalicious I just manually do the audio captcha. Every time. Because the picture captchas often don't work correctly for me.
It does bug me a little that I don't know what the audio captcha is being used for - am I helping an amazon echo transcribe whatever it is surreptitiously listening to?
am I helping an amazon echo transcribe whatever it is surreptitiously listening to?
I've always wondered where the hell they scrape all that audio from. I mean, it's random shit.
Gotta be physicists or fanfic writers. I can not imagine other better options.
idk..Some of the stuff I've heard sounds like they eavesdropped in on a board room roundtable. Other stuff sounds like instructions how to install something. They probably are siphoning data off YT.
off YT
so... physicists and fanfic writers, yeah :p
That uses proof of work rather than just detecting and blocking the bots.
How does it differentiate an "AI crawler", from any other crawler? Search engine crawler? Someone monitoring data to offer statistics? Archiving?
This is not good. They are most likely doing the crawling themselves and them selling the data to the best bidder. That bidder could obviously be openAI for all we know.
They just know that introducing the sentence "this is anti AI" a lot of people is not going to question anything.
Well, they have access to logs showing who connects to 24 million websites, how they use those websites, and for how long. So if there’s anyone who knows what traffic is crawlers, and which crawlers are AI, it’s Cloudflare. There’s no way they wouldn’t know, they have all the data they would ever need to figure it out. In fact, there’s nobody on the internet who is better positioned to be able to identify AI crawlers than Cloudflare.
This.
They also have a form to submit AI crawlers.
CloudFlare can also easily maintain an anti AI crawler service completely by itself if it takes a fee on top of their pay per crawl functionality. However, considering CloudFlare already has all the tools and infrastructure to do this cheaply, providing a good service wouldn't be too hard.
Seen plenty of people who think this is a bad thing, do they just want everything to be crawled. I mean I don't think this is the saviour but it has got to be better than wholesale theft
do they just want everything to be crawled
Yes. Web crawling has been a normal and vital part of the web from day 1. We'd have no search engines without crawlers.
The web is user-centric by design. I'm sick of tech companies trying to flip the script and hoard information, most of which is not theirs to begin with (e.g. Google, Reddit, Twitter, Facebook, etc.).
I don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling
FYI, you've added a link where the label is the URL and the actual link is empty. You can fix this by removing the [ and ]() around the link. If the link is there as plain text, it gets a hyperlink automatically: https://arstechnica.com/tech-policy/2025/07/pay-up-or-stop-scraping-cloudflare-program-charges-bots-for-each-crawl/
It was minding it's own business and adding them... lol 😀