Gemini actually has a really interesting architecture, hence it has fast responses, and it’s easily the best long context model out there.
And outside of bechmaxxing or pure coding, Gemma is very good for its size. 12B is an incredible multimodal LLm, the only one natively trained for image/text ingestion without a mmproj hacked on at the end.
…But it sure feels like executive meddling kills it.
The pattern I see is:
-
Gemini preview is released.
-
It’s genuinely good! It’s smart, it’s straight.
-
Then they “refine” it, it’s gets more and more sycophantic, more deep fried. Long context performance degrades… benchmark scores go up, but anyone who actually uses it can immediately tell it’s gotten worse.
-
Only then, is it released for mass use.
It’s obvious they took a good model, then enshittified it to make their bosses happy and tech bros in Twitter excited.
Gemma has the same pattern. Researchers tease the local community, delay it, and then when a new Gemma finally comes out, it turns out to be using some old SWA architecture. And the biggest model is cut. And only a smaller one uses the multimodal training.
It’s obvious it was neutered to not “threaten” Gemma API or be too “unsafe.”
Another thing I’ve noticed is that both Gemini and Gemma are awful with their default 1.0 temperature/top-p 0.95. Sampling completely screws them up. But they like low temperature + minp, and Gemma loves constrained sampling.
But 99% of users don’t know anything about sampling, so that’s going to leave a bad impression.
