TechTakes

Look, im def on team Murderbot, but when ~~we~~ the AI's start building them I really hope Martha Wells gets some kickbacks at least.

I just looked at the first PR out of curiosity, and wow...

this isn't integrated with tests

That's the part that surprised me the most. It failed the existing automation. Even after prompted to fix the failing tests, it proudly added a commit "fixing" it (it still didn't pass...something that copilot should really be able to check). Then the dev had to step in and say why the test was failing and how to fix the code to make it pass. With this much handholding all of this could have been done much faster and cleaner without any AI involvement at all.

For LLMs specifically? Code is not text, aside from the most clinical, dictionary definition of "text".

But even then, it also fails at writing coherent short or longform, so even if code was "just text" it'd fail equally badly.

To elaborate on the other answers about alphaevolve. the LLM portion is only a component of alphaevolve, the LLM is the generator of random mutations in the evolutionary process. The LLM promoters like to emphasize the involvement of LLMs, but separate from the evolutionary algorithm guiding the process through repeated generations, LLM is as likely to write good code as a dose of radiation is likely to spontaneously mutate you to be able to breathe underwater.

And the evolutionary aspect requires a lot of compute, they don't specify in their whitepaper how big their population is or the number of generations, but it might be hundreds or thousands of attempted solutions repeated for dozens or hundreds of generations, so that means you are running the LLM for thousands or tens of thousands of attempted solutions and testing that code against the evaluation function everytime to generate one piece of optimized code. This isn't an approach that is remotely affordable or even feasible for software development, even if you reworked your entire software development process to something like test driven development on steroids in order to try to write enough tests to use them in the evaluation function (and you would probably get stuck on this step, because it outright isn't possible for most practical real world software).

Alphaevolve's successes are all very specific very well defined and constrained problems, finding specific algorithms as opposed to general software development

zbyte64 gave a great answer. I visualize it like this:

Writing software that does a thing correctly within well defined time and space constraints is nothing like climbing a smooth gradient to a cozy global maximum.

On a good day, it's like hopping on a pogo stick around a spiky, discontinuous, weirdly-connected n-dimensional manifold filled with landmines (for large values of n).

The landmines don't just explode. Sometimes they have unpredictable comedic effects, such as ruining your weekend two months from now.

Evolution is simply the wrong tool for the job.

Talking about Alpha Evolve https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ ?

First, Microsoft isn't using this yet but even if they were it doesn't work in this context. What Google did was they wrote a fitness function to tune the Generative process. Why not have some rubric that scores the code as our fitness function? Because the function needs to be continuous for this to work well, no sudden cliffs. But also they didn't address how this would work in a multi-objective space, this technique doesn't let the LLM make reasonable trade offs between complexity and speed.