this post was submitted on 20 Oct 2026
1055 points (99.4% liked)
Technology
89641 readers
1759 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I dont think a local model will do 99% of what the big ones can, and even if it could, it would require an enormous amount of memory and cpu to do it.
People pay for a good user interface and convenience, that's well known. Netflix and Spotify are built on that idea and people starting using them even over free pirated stuff :)
Qwen3.8-flash-next broke the vram requirement with its new architecture (which is a preview of their 4.0 architecture). You can run a 177b parameter model on 12gb VRAM, 64gb ram and 80gb SSD at 30tok/s and the speed scales almost linearly with increases to vram.
Incidentally, this was proven at almost the exact time all the calls for "regulating local models because they'll spread into the toasters" started.
You absolutely can have local models that run on consumer grade hardware, and the proprietary companies know it and are scared to death.
i would not recommend using the SSD to run local LLMS,its slow
best: VRAM only
good: VRAM + RAM
bad: Swapping
I don't think they're insinuating running inferences from the SSD. But you need somewhere to store the weights for the GPU to load.
i am trying to say: you need the SSD to store it but not swap to it due to its slowness
I won't pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven't run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I've understood those are modest losses in the worst case.