this post was submitted on 09 Nov 2026
2 points (75.0% liked)

Web Development

5942 readers
21 users here now

Welcome to the web development community! This is a place to post, discuss, get help about, etc. anything related to web development

What is web development?

Web development is the process of creating websites or web applications

Rules/Guidelines

Related Communities

Wormhole

Some webdev blogsNot sure what to post in here? Want some web development related things to read?

Heres a couple blogs that have web development related content

CreditsIcon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 3 years ago
MODERATORS
snowe@programming.dev
erlingur@programming.dev
Ategon@programming.dev
Vacant@programming.dev
2
Worth checking: your Cloudflare AI-training block may also cover mixed-purpose crawlers like Googlebot (programming.dev)
submitted 4 hours ago by proxymint@programming.dev to c/webdev@programming.dev
0 comments fedilink hide all child comments
 

If you ticked a "block AI training" option on Cloudflare at some point, this is worth five minutes. Cloudflare's docs (updated Nov 2) say training actions also apply to mixed-purpose crawlers used for both search and training. Their July blog post names Googlebot, Applebot and BingBot as examples.

There were reports in August of Googlebot getting 403s on sitemaps. Nobody has confirmed the cause as far as I can find, so don't panic over it. It could be another setting entirely, Bot Fight Mode came up in the same thread.

Still, checking is cheap. Look at crawl stats in Search Console for a jump in 403s since mid-September, then read which Cloudflare setting is on. I'd rather spend five minutes on that than learn about it from a traffic graph.

Has anyone here actually seen this on their own site?

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here