Web Development

5450 readers

1 users here now

Welcome to the web development community! This is a place to post, discuss, get help about, etc. anything related to web development

What is web development?

Web development is the process of creating websites or web applications

Rules/Guidelines

Follow the programming.dev site rules
Keep content related to web development
If what you're posting relates to one of the related communities, crosspost it into there to help them grow
If youre posting an article older than two years put the year it was made in brackets after the title

Related Communities

Wormhole

!cool_github_projects@programming.dev

Some webdev blogs

Not sure what to post in here? Want some web development related things to read?

Heres a couple blogs that have web development related content

https://frontendfoc.us/ - [RSS]
https://wesbos.com/blog
https://davidwalsh.name/ - [RSS]
https://www.nngroup.com/articles/
https://sia.codes/posts/ - [RSS]
https://www.smashingmagazine.com/ - [RSS]
https://www.bennadel.com/ - [RSS]
https://web.dev/ - [RSS]

Credits

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 2 years ago

MODERATORS

snowe@programming.dev

erlingur@programming.dev

Ategon@programming.dev

Introducing pay per crawl: Enabling content owners to charge AI crawlers for access (blog.cloudflare.com)

submitted 3 days ago by punrca@piefed.world to c/webdev@programming.dev

5 comments fedilink hide all child comments

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:

Allow: Grant the crawler free access to content.
Charge: Require payment at the configured, domain-wide price.
Block: Deny access entirely, with no option to pay.

Although this is old news, I still found it interesting. Also, I like the "AI Labyrinth" feature of Cloudflare to block AI bots.

top 5 comments

sorted by: hot top controversial new old

[–] rimu@piefed.social 13 points 3 days ago (2 children)

Cloudflare seems to live in a world where AI scrapers are ethical and lawful.

[–] AudaciousArmadillo@piefed.blahaj.zone 5 points 3 days ago

Cloudflare's business model is dealing with unethical and unlawful bots. They can pour money into the arms race with "AI" companies and win. If they can't their service isn't valuable.

[–] Dojan@pawb.social 5 points 3 days ago

And apparently profitable. These companies are essentially buying their shit based on clout, not actual possessed monetary value. I’m struggling to see how that could translate into whatever system Cloudflare thinks they will engage with.

[–] cockmushroom@reddthat.com 2 points 3 days ago (1 children)

what if youlre not trying to train ai? What if you just want to update a personal dashboard/archive?

[–] Hippy@piefed.social 2 points 3 days ago

Well, cockmushroom, then you're building a pretty expensive dashboard. I still don't see how they are going to differentiate between normal browsing and bot crawling. I don't think that's an exact science.