Fuck AI

That is like arguing that climate change could have been addressed. Yes, but it was never going to be. Zero chance from day one. Zero chance from twenty six years back when I learned about it in more depth.

Yes it could be, but it never was going to be. No chance it is theoretical.

The problem is being “anti AI” smothers open weights ML, doing basically nothing against corporate AI.

The big players do not care. They’re going to shove it down your throats either way. And this whole fuss is convenient, as it crushes support for open weights AI and draws attention away from it.

What I’m saying is people need to be advocating for open weights stuff instead of “just don’t use AI” in the same way one would advocate for Lemmy/Piefed instead of “just don’t use Reddit”

The Fediverse could murder trillions of dollars in corporate profit with enough critical mass. AI is the same, but it’s much closer to doing it than people realize.

Open models, running on consumer hardware, will probably be really subpar for a very long time because the AI big boys are heavily subsidizing their AIs with massive models running on massive compute farms in massive datacenters. They are subsidizing these things using your environment, your electricity dollars, your tax dollars and your retirement fund's dollars to subsidize it with, since they're all losing money all the time.

But they're not going to do that forever (and you can't afford to do this forever either), but that's okay because they just want to get you onboard. It's a trap, and you can play around with that alluring trap if you want, but they're going to make it really easy to fall in when you do.

Once they've got you in the trap, they're eventually going to tighten the screws, lock you into their ecosystem (if they haven't already) and start extracting money from you directly. That's coming, they'll lock it down and enshittify until you pay them, and then they'll enshittify that until you pay them more, because eventually they have to reach the real economics of providing these AI models, and then they have to start actually turning a profit, and you can't even imagine how far they'll have to eventually go to get to that point, you might as well just set your wallet on fire now.

So, if you're expecting, open, free, local models to compete directly with that, you aren't understanding their business model. The results that big AI providers are giving you are an absolute illusion, built out of a collage of countless different lies and thefts, those results won't continue they way they are now, they can't continue, they're not economically viable to continue. They're only that good so they can lure you into the trap.

Open models, on the other hand, operate in the concrete reality of the here and now. What they're giving you is the real quality of existing models on sensible contemporary hardware using the real economics of providing these things. If you find that underwhelming, well, maybe it is, but it doesn't mean the lies of big AI are real or ever will be. You can try and take advantage of the constant, delightful lies they're currently showing you while you still can, but chances are they'll end up taking advantage of you instead, because that's what their goal is.

Yeah, accessibility is the big problem.

What I used depends.

For “chat” and creativity, I use my own version of GLM 4.6 350B quantized to just barely fit in 128GB RAM/24GB VRAM, with a fork of llama.cop called ik_llama.cpp:

https://huggingface.co/Downtown-Case/GLM-4.6-128GB-RAM-IK-GGUF

It’s complicated, but in a nutshell, the degradation vs the full model is reasonable even though it’s like 3 bits instead of 16, and it runs at 6-7 tokens/sec even with so much in CPU.

For the UI, it varies, but I tend to use mikupad so I can manipulate the chat syntax. LMStudio works pretty well though.

Now, for STEM stuff or papers? I tend to use Nemotron 49B quantized with exllamav3, or sometimes Seed-OSS 36B, as both are good at that and at long context stuff.

For coding, automation? It… depends. Sometimes I used Qwen VL 32B or 30B, in various runtimes, but it seems that GLM 4.7 Flash and GLM 4.6V will be better once I set them up.

Minimax is pretty good at making quick scripts, while being faster than GLM on my desktop.

For a front end, I’ve been switching around.

I also use custom sampling. I basically always use n-gram sampling in ik_llama.cpp where I can, with DRY at modest temperatures (0.6?). Or low or even zero temperature for more “objective” things. This is massively important, as default sampling is where so many LLM errors come from.

And TBH, I also use GLM 4.7 over API a lot, in situations where privacy does not matter. It’s so cheap it’s basically free.

So… Yeah. That’s the problem. If you just load up LMStudio with its default Llama 8B Q4KM, it’s really dumb and awful and slow. You almost have to be an enthusiast following the space to get usable results.