this post was submitted on 02 Jun 2025
32 points (100.0% liked)

TechTakes

1911 readers
83 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago
MODERATORS
dgerard@awful.systems
32
Fighting the AI scraper bots at Pivot to AI and RationalWiki (pivot-to-ai.com)
submitted 5 days ago* (last edited 5 days ago) by dgerard@awful.systems to c/techtakes@awful.systems
6 comments fedilink hide all child comments
 

video version

that s3kr1t method: https://www.jwz.org/blog/2025/05/user-agent-blocking/#comment-259266

you are viewing a single comment's thread
view the rest of the comments
[–] BlueMonday1984@awful.systems 15 points 5 days ago

Edit: But also - why do AI scrapers request pages that show differences between versions of wiki pages (or perform other similarly complex requests)? What’s the point of that anyway?

Gonna take a shot in the dark here and guess that the people behind these AI scrapers are incredibly stupid, and think Scrape More = More Data = More Growth = More Good^TM^. Given this bubble's already given us the goddamn scourge that is "vibe coding", and is headed by growth-obsessed fuckwits and True Believers^TM^, chances are I'm completely correct on this.