TechTakes

Can't believe I'm nerd-sniped this easily. Very technically, the point at which a service should be considered unreliable or down is at γ nines, where γ = 0.9030899869919434… is a transcendental constant. γ nines is exactly 87.5% availability, or 7/8 availability, and it's the point at which a service's availability might as well be random. (Another one of the local complexity theorists can explain why it's 7/8 and not 1/2.)

Alas, foiled again! Nobody said they had to be leading 9s!

Even the most dysfunctional team would have spent time developing some kind of standard here from my (admittedly limited) experience.

My experience has been vastly different. Prior to LLMs I have seen all sorts of horrors of this sort and others writ large across many codebases. It's so awesome that LLMs offer the ability to make the same sorts of code but at a much faster speed. In times past it used to take devs years to build up the kind of tech debt that LLMs can give you in days.

I'm sure these English instructions work because they feel like they work. Look, these LLMs feel really great for coding. If they don't work, that's because you didn't pay $200/month for the pro version and you didn't put enough boldface and all-caps words in the prompt. Also, I really feel like these homeopathic sugar pills cured my cold. I got better after I started taking them!

No joke, I watched a talk once where some people used an LLM to model how certain users would behave in their scenario given their socioeconomic backgrounds. But they had a slight problem, which was that LLMs are nondeterministic and would of course often give different answers when prompted twice. Their solution was to literally use an automated tool that would try a bunch of different prompts until they happened to get one that would give consistent answers (at least on their dataset). I would call this the xkcd green jelly bean effect, but I guess if you call it "finetuning" then suddenly it sounds very proper and serious. (The cherry on top was that they never actually evaluated the output of the LLM, e.g. by seeing how consistent it was with actual user responses. They just had an LLM generate fiction and called it a day.)

Yeah, letting the intrinsically insecure RNG recursively rewrite its own security instructions definitely can't go wrong. I mean they limited it to only so so when the users asked nicely!

Edit to add:

The more I think about it the more it speaks to Anthropic having an absolute nonsense threat model that is more concerned with the science fiction doomsday AI "FOOM" than it is with any of the harms that these systems (or indeed any information system) can and will do in the real world. The current crop of AI technologies, while operating at a terrifying scale, are not unique in their capacity to waste resources, reify bias and inequality, misinform, justify bad and evil decisions, etc. What is unique, in my estimation, is both the massive scale that these things operate despite the incredible costs of doing so and their seeming immunity to being reality checked on this. No matter how many times the warning bells about these systems' vulnerability to exploitation, the destructive capacity of AI sycophancy and psychosis, or the simple inability of the electrical infrastructure to support their intended power consumption (or at least their declared intent; in a bubble we shouldn't assume they actually expect to build that much), the people behind these systems continue to focus their efforts on "how do we prevent skynet" over any of it.

Thinking in the context of Charlie Stross' old talk about corporations as "slow AI," I wonder if some of the concern comes either explicitly or implicitly from an awareness that "keep growing and consuming more resources until there's nothing left for anything else, including human survival" isn't actually a deviation from how these organizations are building these systems. It's just the natural conclusion of the same structures and decision-making processes that leads them to build these things in the first place and ignore all the incredibly obvious problems. They could try and address these concerns at a foundational or structural level instead of just appending increasingly complex forms of "please don't murder everyone or ignore the instructions to not murder everyone" to the prompt, but doing that would imply that they need to radically change their entire course up to this point and increasingly that doesn't appear likely to happen unless something forces it to.

@Soyweiser @fiat_lux

So many of these people, as with the NFT clowns, have "Twelve Year Old First Day On The Internet" Energy

This is all just JavaScript, so yes. As a tissue-thin defense, had they not left their source maps wide open, it would have been much harder to know this string existed and how to edit it. Not impossible, but much harder.

what they actually offered wasn’t speedy compliance as a service to get you certified, it was speedy certification as a service by bypassing actual compliance.

I mean... Yeah. I think if you read it any other way you're a massive rube. Like it's obviously not possible to do the former in "days" as they advertise.

On the one hand, Yud's vision of AI doomsday is specifically "AI turns sentient/superintelligent and kills us all because reasons", not "Humanity wipes itself out because they trusted lying machines".

On the other hand, the absence of sentience/superintelligence hasn't stopped AI from causing untold damage anyways, as the past two to three years can attest.

Eliezer is trying to get around that with some weird conditions and game on the prediction market question:

This market resolves N/A on Jan 1st, 2027. All trades on this market will be rolled back on Jan 1st, 2027. However, up until that point, any profit or loss you make on this market will be reflected in your current wealth; which means that purely profit-interested traders can make temporary profits on this market, and use them to fund other permanent bets that may be profitable; via correctly anticipating future shifts in prices among people who do bet their beliefs on this important question, buying low from them and selling high to them.

I don't think that actually helps. But Eliezer is committed to prediction markets being useful on a nearly ideological level, so he has to try to come up with weird complicated strategies to try to get around their fundamental limits.