this post was submitted on 20 Mar 2025
380 points (99.7% liked)
Open Source
35511 readers
60 users here now
All about open source! Feel free to ask questions, and share news, and interesting stuff!
Useful Links
- Open Source Initiative
- Free Software Foundation
- Electronic Frontier Foundation
- Software Freedom Conservancy
- It's FOSS
- Android FOSS Apps Megathread
Rules
- Posts must be relevant to the open source ideology
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon from opensource.org, but we are not affiliated with them.
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
While AI crawlers are a problem I'm also kind of astonished why so many projects don't use tools like ratelimiters or IP-blocklists. These are pretty simple to setup, cause no/very little additional load and don't cause collateral damage for legitimate users that just happend to use a different browser.
IP based blocking is complicated once you are big enough or providing service to users is critical.
For example, if you are providing some critical service such as health care, you cannot have a situation where a user cannot access health care info without hard proof that they are causing an issue and that you did your best to not block the user.
Let's say you have a household of 5 people with 20 devices in the LAN, one can be infected and running some bot, you do not want to block 5 people and 20 devices.
Another example, double NAT, you could have literally hundreds or even thousands of people behind one IP.
It's literally as simple as importing an ipset into iptables and refreshing it from time to time. There is even predefined tools for that.