Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
On the bar chart I included what I think are the most popular models right now (not objectively, based on my anecdotal observations, link to include/exclude what you want).
What do you think? Is it fair? There was, and to some digree still is, a bit of controversy about AA score indexes after they updated their benchmarks to V5.0. I am mainly interested in your thoughts about QFN in relation to other models.
I don't know what they are testing but I use QFN for everything.
Assuming it doesn't have to rely on internal knowledge too much by giving it tools to retrieve information, it does everything I ask it to.
What sort of setup are you rocking? Which quant? I'm running Qwen 3.6 35B A3B Q2 for speed on my 16GB T4060 Ti and Qwen 3.8 27b Q5 on my R9700 32GB for coding. Unsure if I can run a half decent QFN quant on my setup along with my 128GB of sys RAM.
I'm running Gufo on a Strix Halo PC with 128GB shared memory.
The quant is unsloth's UD-Q4_K_XL. Speed is really good with this configuration (1600 t/s prefill and 100 t/s decode). I wanted to try some higher quants as well but I'm waiting for Qwen 4 for now to do more testing.
You should be able to get QFN running quite well on your system as well when offloading unused experts to RAM.
Amazing, will give that a shot. Thanks!
I think Gufo is optimized only for the Strix Halo but llama.cpp should get you running with expert offloading no problem.
i reccomend strata instead
Gemini 3.8 does not match its benchmarks.
It forgets system instructions, spends too much time thinking, and has this weird bug with reasoning token reprocessing that totally throws off the model.
Mistral Large should be higher up, it’s far more reliable than its benchmarks shows.
What is this bug you described? First time hearing about it... Mistral seems weirdly low, it's a very big model, maybe they didn't finetune it for benchmarks?
Gemini encrypts their thinking traces, so if you inject a thought in an early thinking trace it always gets added after user context on any later message in the conversation.
That means if the thought trace disagrees with a user message it goes with the original thought trace.
E.g imagine you start a conversation with the date, then come back the next day. The thinking trace will make the model always think it’s the same date, no matter how much you argue.
They were working on fixing this, so not sure if the new models today fixes that.
Interesting, maybe caused by design focused on overprotection against distillation of the thinking process, so "it's a feature, not a bug" mentality. I don't use gemini models so I can't confirm by myself, thanks for pointing that out!
That is part of it.
The thinking trace is a base64 encoded proton if with an encrypted string inside