TechTakes

Alas, foiled again! Nobody said they had to be leading 9s!

That, and plenty of FOSS software's been infected with AI-extruded "code". And plenty of software engineers got one-tapped by the slop bots.

That was great, thank you! Full respect to this absolute maniac for tracing some of the spaghetti, I was definitely not going to try that on my phone.

They've validated most gut feelings I had about how Claude works (and doesn't), based on my experience having to use it. I'm feeling pretty smug that my hunches now have definitive code attributions.

But the one unfortunate part about all of this is that this leak and the ensuing justified sneers about specific bits are going to be fed back in to their codebase to fix some of the gaping holes. It's an embarrassing indictment of the product, but it's also free pre-IPO pentesting. Sort of like their open source pull request slop spam "undercover mode" was probably used as a way to extract free labor in the form of reviews from actually competent developers. This doesn't seem as planned though.

A Korean developer named Sigrid Jin—featured in the Wall Street Journal earlier this month for having consumed 25 billion Claude Code tokens—woke up at 4 a.m. to the news. He sat down, ported the core architecture to Python from scratch using an AI orchestration tool called oh-my-codex, and pushed claw-code before sunrise. The repo hit 30,000 GitHub stars faster than any repository in history.

Considering how one of the major use cases of llm coding agents is laundering open source and copy left, this is some well deserved payback to Anthropic imho.

According to the claude code leak the state of the art is to be, like, really stern and authoritative when you are begging it to do its job:

I don't work at one of those companies, just somewhere mainlining AI, so this answer might not satisfy your requirements. But the answer is very simple. The first thing anyone working in AI will tell you (maybe only internally?) is that the output is probabilistic not deterministic. By definition, that means it's not entirely consistent or reproducible, just... maybe close enough. I'm sure you already knew that though.

However, from my perspective, even if it was deterministic, it wouldn't make a substantial difference here.

For example, this file says I can't ask it to build a DoS script. Fine. But if I ask it to write a script that sends a request to a server, and then later I ask it to add a loop... I get a DoS script. It's a trivial hurdle at best, and doesn't even approach basic risk mitigation.

I'm sure these English instructions work because they feel like they work. Look, these LLMs feel really great for coding. If they don't work, that's because you didn't pay $200/month for the pro version and you didn't put enough boldface and all-caps words in the prompt. Also, I really feel like these homeopathic sugar pills cured my cold. I got better after I started taking them!

No joke, I watched a talk once where some people used an LLM to model how certain users would behave in their scenario given their socioeconomic backgrounds. But they had a slight problem, which was that LLMs are nondeterministic and would of course often give different answers when prompted twice. Their solution was to literally use an automated tool that would try a bunch of different prompts until they happened to get one that would give consistent answers (at least on their dataset). I would call this the xkcd green jelly bean effect, but I guess if you call it "finetuning" then suddenly it sounds very proper and serious. (The cherry on top was that they never actually evaluated the output of the LLM, e.g. by seeing how consistent it was with actual user responses. They just had an LLM generate fiction and called it a day.)

This is all just JavaScript, so yes. As a tissue-thin defense, had they not left their source maps wide open, it would have been much harder to know this string existed and how to edit it. Not impossible, but much harder.

Yeah, letting the intrinsically insecure RNG recursively rewrite its own security instructions definitely can't go wrong. I mean they limited it to only so so when the users asked nicely!

Edit to add:

The more I think about it the more it speaks to Anthropic having an absolute nonsense threat model that is more concerned with the science fiction doomsday AI "FOOM" than it is with any of the harms that these systems (or indeed any information system) can and will do in the real world. The current crop of AI technologies, while operating at a terrifying scale, are not unique in their capacity to waste resources, reify bias and inequality, misinform, justify bad and evil decisions, etc. What is unique, in my estimation, is both the massive scale that these things operate despite the incredible costs of doing so and their seeming immunity to being reality checked on this. No matter how many times the warning bells about these systems' vulnerability to exploitation, the destructive capacity of AI sycophancy and psychosis, or the simple inability of the electrical infrastructure to support their intended power consumption (or at least their declared intent; in a bubble we shouldn't assume they actually expect to build that much), the people behind these systems continue to focus their efforts on "how do we prevent skynet" over any of it.

Thinking in the context of Charlie Stross' old talk about corporations as "slow AI," I wonder if some of the concern comes either explicitly or implicitly from an awareness that "keep growing and consuming more resources until there's nothing left for anything else, including human survival" isn't actually a deviation from how these organizations are building these systems. It's just the natural conclusion of the same structures and decision-making processes that leads them to build these things in the first place and ignore all the incredibly obvious problems. They could try and address these concerns at a foundational or structural level instead of just appending increasingly complex forms of "please don't murder everyone or ignore the instructions to not murder everyone" to the prompt, but doing that would imply that they need to radically change their entire course up to this point and increasingly that doesn't appear likely to happen unless something forces it to.

@Soyweiser @fiat_lux

So many of these people, as with the NFT clowns, have "Twelve Year Old First Day On The Internet" Energy

A chatbot interface offers no meaningful advantages for interrogating Washington's ethical stance, over and above the documents that are already available. Instead, it offers a pleasant sheen of false certainty. So in that way, it's dragging a guy who's been dead for two centuries into the social media era. Huzzah!

In my experience "all hands" meetings are very much CEOs and their sycophants cosplaying at podcast hosts for an hour whilst forcing their employees to watch/listen. They are almost never useful and a colossal waste of money - especially in corporation's with 10k+ employees. Like the salary cost for 10k people for 1 hour would probably pay off my mortgage.