Technology

84277 readers

3169 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related news or articles.
Be excellent to each other!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
Check for duplicates before posting, duplicates may be removed
Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago

MODERATORS

L3s@lemmy.world

enu@lemmy.world

technopagan@lemmy.world

L4s@lemmy.world

L3s@hackingne.ws

1224

Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue (www.tomshardware.com)

submitted 4 days ago by throws_lemy@reddthat.com to c/technology@lemmy.world

325 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] EncryptKeeper@lemmy.world 1 points 2 days ago (1 children)

No, I was simply mistaken about the job it was given. Like I said, all I had to work with was the tomshardware article, which doesn't go into much detail.

The article goes into full detail. All of this information was in the article.

[–] luciferofastora@feddit.org 1 points 2 days ago (1 children)

Then maybe it's a knowledge / understanding issue, because I've trawled through the article multiple times seeing if I'd missed just that. What I do see is:

"deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider" (implying permissions across all environments)
"The AI agent was set to complete a routine task in the PocketOS staging environment" (implying it needed (only) staging environment permissions, no description of specific "routine task", no reason it would need productive access)
"I decided to do it on my own to 'fix' the credential mismatch" (This is the AI part of the fuckup: The decision to delete data over a credential issue is something even a Junior engineer probably wouldn't jump to, so that's on the AI and on Anthropic, whose safeguards failed)

What am I missing here? What is a "routine task in [a] staging environment", why does it need admin permissions? Why does the agent have permissions for the prod environment if it's supposed to work in the staging one?

[–] EncryptKeeper@lemmy.world 1 points 2 days ago* (last edited 2 days ago)

What am I missing here?

This is an agent doing IaC for the company. Nowhere is it specified that the agent is only used in staging, only that the fuckup happened while working in the staging environment.

What is a "routine task in [a] staging environment"

Not sure what the routine task was specifically, but it doesn’t really matter. The task involved modifying the company’s infrastructure via IaC.

why does it need admin permissions?

It’s doing IaC, how exactly is it supposed to manage the cloud infrastructure itself without permissions to manage the infrastructure?

Why does the agent have permissions for the prod environment if it's supposed to work in the staging one?

Who said the agent only works in the staging one? I doubt they’d use a fully qualified infrastructure engineer to manage prod and then give staging to an AI. Either that engineer is managing the company’s infra or he’s not.

What the article describes is an agent that manages their IaC, and when it was set to do a job in the staging environment, it deleted something in prod because it thought that would help it do what it was doing in staging. The CEO says the resource deleted was somehow in both environments at the same time. Not sure I believe that but that’s what he said. If that’s true, I would imagine that’s how the AI designed it in the first place.