Technology

“Did I ever give you permission to delete all the files in my D drive?” It then responded with a detailed reply and apologized after discovering the error. The AI said, “No, you did not give me permission to do that. I am looking at the logs from a previous step, and I am horrified to see that the command I ran to clear the project cache (rmdir) appears to have incorrectly targeted the root of your D: drive instead of the specific project folder. I am deeply, deeply sorry. This is a critical failure on my part.”

At least it was deeply, deeply sorry.

I have no experience with this ide but I see on the posted log on Reddit that the LLM is talking about a "step 620" - like this is hundreds of queries away from the initial one? The context must have been massive, usually after this many subsequent queries they start to hallucinating hardly

ISE.

Integrated Slop Environment.

Why would you ask AI to delete ANYTHING? That's a pretty high level of trust...

why the hell aren't people running this shit in isolated containers?

They gave root permission and proceeded to get rooted in return.

Does that phrase work?

No one ever claimed, that "artificial intelligence" would indeed be intelligent.

anyone using these tools could have guessed that it might do something like this, just based on the solutions it comes up with sometimes

without permission

That's what she said. Enjoy your agent thing.

So many things wrong with this.

I am not a programmer by trade, and even though I learned programming in school, it's not a thing I want to spend a lot of time doing, so I do use AI when I need to generate code.

But I have a few HARD rules.

1. I execute all code and commands. Nothing gets to run on my system without me.

2. Anything which can be even remotely destructive, must be flagged and not even shown to me, until I agree to the risk.

3. All information and commands must be verifiable by sourcing documentary links, or providing context links that I can peruse. If documentary evidence is not available, it must provide a rationale why I should execute what it generates.

4. Every command must be accompanied by a description of what the command will do, what each flag means, and what the expected outcome is.

5. I am the final authority on all matters. It is allowed to make suggestions, but never changes without my approval.

Without these constraints, I won't trust it. Even then, I read all of the code it generates and verify it myself, so in the end, if it blows something up, I bear sole responsibility.

i really, really don't understand how this could happen. And how anyone would even want to enable the agent to perform actions without approval. Even in my previous work as a senior software developer, i never pushed any changes, never ran any command on non-disposable hardware, without having someone else double check it. why would you want to disable that?