Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
On the bar chart I included what I think are the most popular models right now (not objectively, based on my anecdotal observations, link to include/exclude what you want).
What do you think? Is it fair? There was, and to some digree still is, a bit of controversy about AA score indexes after they updated their benchmarks to V5.0. I am mainly interested in your thoughts about QFN in relation to other models.
I don't know what they are testing but I use QFN for everything.
Assuming it doesn't have to rely on internal knowledge too much by giving it tools to retrieve information, it does everything I ask it to.
What sort of setup are you rocking? Which quant? I'm running Qwen 3.6 35B A3B Q2 for speed on my 16GB T4060 Ti and Qwen 3.8 27b Q5 on my R9700 32GB for coding. Unsure if I can run a half decent QFN quant on my setup along with my 128GB of sys RAM.
I'm running Gufo on a Strix Halo PC with 128GB shared memory.
The quant is unsloth's UD-Q4_K_XL. Speed is really good with this configuration (1600 t/s prefill and 100 t/s decode). I wanted to try some higher quants as well but I'm waiting for Qwen 4 for now to do more testing.
You should be able to get QFN running quite well on your system as well when offloading unused experts to RAM.
Amazing, will give that a shot. Thanks!
I think Gufo is optimized only for the Strix Halo but llama.cpp should get you running with expert offloading no problem.
i reccomend strata instead