SuspiciousCarrot78

They don't need to. Github has a much stronger SEO. It's literally a global top 100 website.

Github as billboard / pointer / "trust signal" is just smart discoverability....but not a good "home" these days.

Go for it! The m73 is cheap enough (and powerful enough) to run all that and ddr3 is still not insane (say, 2x8gb 1600mhz sodimm if want / need). $100 or so, all up, if you shop around / your local market pending.

Raspberry pi is more elegant / more constrained / more "fuck you, figure it out" but unless you need the challenge, Lenovo is simpler and all around easier first step :). You can't stick a gpu in it (I think the m920 is the oldest one that has pcie - dunno what they go for. The usual combo is something like a 920 and a Quadro P1000 4GB GPU. Maybe ~$300 all up if we're guessing. At which point, there are better, non shoe box options)

I have a RPI 4b and 3 lenovos (m93p, m710q, p330).

You can't beat the RPI for power draw (~2w idle and ~7w under max load) but I suspect if you wanted to look at $ to utility measure you'd probably prefer the Lenovo M93P. $50 USD. Mine has i7-4785t, 16GB ddr3 (2x8iirc?) with ethernet, USB etc. Bought 2023/4. I expect base model is still that price now (mines upgraded). The only caveat is that it doesn't have HDMI, it has display port out, but that's just a $5 dongle or SSH issue. M73 would be a touch cheaper.

Iirc the TDP is 35w max and can be lowered / undervolted a touch (don't update the BIOS - it blocks throtlestop).

I turned mine into a retro PC slash game server for the kids (luanti etc). But the siren call of doing truly impossible things with the RPI is too beguiling :)

Eg: running diet pi (headless) with all of my services (media stack, privacy, docs, search, images etc) takes about 300 megabytes (or 650mb if I have to boot into xfce).

300mb, 2-3w.

That shouldn't be possible. I love it.

My next goal is to create an expert system / pseudo llm that sources answers based on user provided markdown or PDF, ZIM files and 4get search or Tavily.

The advantage here is that 1) speed will be stupid fast as no neural network crap (outside of optional extra Markov chain garnish) 2) not stochastic (but allow for llm as optional "plug in module" - pi might actually run a 135M at non glacial speeds) 3) still serves openAI compat endpoint.

Is that the right site or am I not seeing it? Your link points to this -

https://idlewatt.foundagent.net/ Lookup Categories Compare Vendors AI Data Watch Methodology Will this vendor sign a HIPAA BAA? A cited, date-stamped answer for 105 major SaaS tools — can you sign a Business Associate Agreement and store PHI? Built for digital-health teams during vendor procurement.

Respectfully, that's not really how local LLMs work.

A GGUF model sitting on my hard drive has no ability to "send content back home" any more than a PDF or a JPEG does. If you're running something like llama.cpp or Ollama entirely locally, the model weights are just data files.

The real privacy concerns are cloud APIs, telemetry in front-ends, browser extensions, analytics, update services, or accidentally exposing a service to the public internet.

"Self-hosted AI" isn't one thing. There's a huge difference between:

• Running ChatGPT through an API
• Running a commercial AI appliance
• Running a local Qwen/Mistral/Llama model on your own hardware

Firewalling internet-facing services is good advice. Assuming every local model is secretly uploading prompts is not.

Hmm....it runs on a 1060...it's a MoE not a dense. 24B is even lighter. Worth a shot.

https://www.youtube.com/watch?v=8F_5pdcD3HY

Else, if youre looking for a coding model (??) something like Sara or fara might suit

https://huggingface.co/microsoft/Fara-7B

Nuance about AI? On Lemmy? On FuckAI? What? GTFO.

PS: I work with AI in health care; it's really useful to me. In my personal life, I'm also a fan of self hosting local models. I'd like to think there's room for nuanced discussion but there usually isn't, because wagons have been circled and frankly, people have pegged their identities to x or y. There are better things to do than argue with the righteous on the interwebs.

Anyway, I liked your post, fellow meat popsicle. I recommend using a Thinkpad for hitting with - more satisfying thump.

I mean...that entirely depends on your use case - and I hate saying that. For me and what I do, Qwen SLM (esp Qwen3-4B 2507 instruct and Qwen3.5-2B) are exceptional. But I'm not trying to do Claude at home.

Best bet? Spend $10 on OpenRouter and try different models. In a head to head with ChatGPT 5.4 mini (excellent for coding BTW), I've found Qwen 3.5 27B more than able to hold its own for coding tasks...IF you narrowly gate it/confine it. The last batch of Qwen's really are something. Dunno about the 3.7 series.

Having said ALL that, I'm really tempted to go back in time and code myself a deterministic expert system, with user updatable knowledge cascade, tool calling and a minimal amount of Markov chain word garnish for flavour. I think we use to just call that "a program" lol.

Really tempted actually, because if 50% of llm use case is basically Super Google but not shit...well, I can make that myself. I just need to point my autism at it.

PS: this might help

https://www.youtube.com/watch?v=0AqpaFm11oI

Numbers about 3-4x. The P100 is near 800 GB/s. The 1080 is what... 192GB/s? Hell, even if it were double that, HBM2 simply has larger bandwidth. The 1080 was a gaming card; the P100 is a server / number cruncher.

Just for sake of completion

https://piwigo.org/

Pros

Mature project (around since the early 2000s)

Lightweight compared to Immich

Designed as a photo library first, not an AI platform

Albums, tags, metadata, permissions

Huge plugin ecosystem

Runs happily on modest hardware

Can manage very large collections

Doesn't demand phone-app-centric workflows (though of course it has a phone to computer app / sync)

Cons

Feels more like a traditional photo archive than Google Photos

Mobile experience is functional rather than slick

No fancy AI search or face recognition by default (though can add easy enough)

UI is a bit "classic web"

Huh - cheaper than the P40s (though less VRAM) but larger bandwidth due to HBM2. Good looking out