LocalLLaMA

Oh, I LOVE to talk, so I hope you don't mind if I respond with my own wall of text :) It got really long, so I broke it up with headers.

TLDR: Bifurcation is needed because of how fitting multiple GPUs on one PCIe x16 lane works and consumer CPU PCIe lane management limits. Context offloading is still partial offloading, so you'll still get hit with the same speed penalty—with the exception of one specific advanced partial offloading inference strategy involving MoE models.

CUDA

To be clear about CUDA, it's an API optimized for software to use NVIDIA cards. When you use an NVIDIA card with Kobold or another engine, you tell it to use CUDA as an API to optimally use the GPU for compute tasks. In Kobold's case, you tell it to use cuBLAS for CUDA.

The PCIe bifurcation stuff is a separate issue when trying to run multiple GPUs on limited hardware. However, CUDA has an important place in multi-GPU setups. Using CUDA with multiple NVIDIA GPUs is the gold standard for homelabs because it's the most supported for advanced PyTorch fine-tuning, post-training, and cutting-edge academic work.

But it's not the only way to do things, especially if you just want inference on Kobold. Vulkan is a universal API that works on both NVIDIA and AMD cards, so you can actually combine them (like a 3060 and an AMD RX) to pool their VRAM. The trade-off is some speed compared to a full NVIDIA setup on CUDA/cuBLAS.

PCIe Bifurcation

Bifurcation is necessary in my case mainly because of physical PCIe port limits on the board and consumer CPU lane handling limits. Most consumer desktops only have one x16 PCIe slot on the motherboard, which typically means only one GPU-type device can fit nicely. Most CPUs only have 24 PCIe lanes, which is just enough to manage one x16 slot GPU, a network card, and some M.2 storage.

There are motherboards with multiple physical x16 PCIe slots and multiple CPU sockets for special server-class CPUs like Threadrippers with huge PCIe lane counts. These can handle all those PCIe devices directly at max speeds, but they're purpose-built server-class components that cost $1,000+ USD just for the motherboard. When you see people on homelab forums running dozens of used server-class GPUs, rest assured they have an expensive motherboard with 8+ PCIe x16 slots, two Threadripper CPUs, and lots of bifurcation. (See the bottom for parts examples.)

Information on this stuff and which motherboards support it is spotty—it's incredibly niche hobbyist territory with just a couple of forum posts to reference. To sanity check, really dig into the exact board manufacturer's spec PDF and look for mentions of PCIe features to be sure bifurcation is supported. Don't just trust internet searches. My motherboard is an MSI B450M Bazooka (I'll try to remember to get exact numbers later). It happened to have 4x4x4x4 compatibility—I didn't know any of this going in and got so lucky!

For multiple GPUs (or other PCIe devices!) to work together on a modest consumer desktop motherboard + CPU sharing a single PCIe x16, you have to:

1. Get a motherboard that allows you to intelligently split one x16 PCIe lane address into several smaller-sized addresses in the BIOS
2. Get a bifurcation expansion card meant for the specific splitting (4x4x4x4, 8x8, 8x4x4)
3. Connect it all together cable-wise and figure out mounting/case modification (or live with server parts thrown together on a homelab table)

A secondary reason I'm bifurcating: the used server-class GPU I got for inferencing (Tesla P100 16GB) has no display output, and my old Ryzen CPU has no integrated graphics either. So my desktop refuses to boot with just the server card—I need at least one display-output GPU too. You won't have this problem with the 3060. In my case, I was planning a multi-GPU setup eventually anyway, so going the extra mile to figure this out was an acceptable learning premium.

Bifurcation cuts into bandwidth, but it's actually not that bad. Going from x16 to x4 only results in about 15% speed decrease, which isn't bad IMO. Did you say you're using a x1 riser though? That splits it to a sixteenth of the bandwidth—maybe I'm misunderstanding what you mean by x1.

I wouldn't obsess over multi-GPU setups too hard. You don't need to shoot for a data center at home right away, especially when you're still getting a feel for this stuff. It's a lot of planning, money, and time to get a custom homelab figured out right. Just going from Steam Deck inferencing to a single proper GPU will be night and day. I started with my decade-old ThinkPad inferencing Llama 3.1 8B at about 1 TPS, and it inspired me enough to dig out the old gaming PC sitting in the basement and squeeze every last megabyte of VRAM out of it. My 8GB 1070 Ti held me for over a year until I started doing enough professional-ish work to justify a proper multi-GPU upgrade.

Offloading Context

Offloading context is still partial offloading, so you'll hit the same speed issues. You want to use a model that leaves enough memory for context completely within your GPU VRAM. Let's say you use a quantized 8B model that's around 8GB on your 12GB card—that leaves 4GB for context, which I'd say is easily about 16k tokens. That's what most lower-parameter local models can realistically handle anyway. You could partially offload into RAM, but it's a bad idea—cutting speed to a tenth just to add context capability you don't need. If you're doing really long conversations, handling huge chunks of text, or want to use a higher-parameter model and don't care about speed, it's understandable. But once you get a taste of 15-30 TPS, going back to 1-3 TPS is... difficult.

MoE

Note that if you're dead set on partial offloading, there's a popular way to squeeze performance through Mixture of Experts (MoE) models. It's all a little advanced and nerdy for my taste, but the gist is that you can use clever partial offloading strategies with your inferencing engine. You split up the different expert layers that make up the model between RAM and VRAM to improve performance—the unused experts live in RAM while the active expert layers live in VRAM. Or something like that.

I like to talk (in case you haven't noticed). Feel free to keep the questions coming—I'm happy to help and maybe save you some headaches.

Oh, in case you want to fantasize about parts shopping for a multi-GPU server-class setup, here are some links I have saved for reference. GPUs used for ML can be fine on 8 PCI lanes (https://www.reddit.com/r/MachineLearning/comments/jp4igh/d_does_x8_lanes_instead_of_x16_lanes_worsen_rtx/)

A Threadripper Pro has 128 PCI lanes: (https://www.amazon.com/AMD-Ryzen-Threadripper-PRO-3975WX/dp/B08V5H7GPM)

You can get dual sWRX8 motherboards: (https://www.newegg.com/p/pl?N=100007625+601362102)

You can get a PCIe 4x expansion card on Amazon: (https://www.amazon.com/JMT-PCIe-Bifurcation-x4x4x4x4-Expansion-20-2mm/dp/B0C9WS3MBG)

All together, that's 256 PCI lanes per machine, as many PCIe slots as you need. At that point, all you need to figure out is power delivery.

Yeah it’s basically impossible to keep up with new releases, heh.

Anyway, Gemma 12B is really popular now, and TBH much smarter than Nemo. You can grab a special “QAT” Q4_0 from Google (that works in kobold.cpp, but fits much more context with base llama.cpp) with basically the same performance as unquantized, would highly recommend that.

I'd also highly recommend trying 24B when you get the rig! It’s so much better than Nemo, even more than the size would suggest, so it should still win out even if you have to go down to 2.9 bpw, I’d wager.

Qwen3 30B A3B is also popular now, and would work on your 3770 and kobold.cpp with no changes (though there are speed gains to be had with the right framework, namely ik_llama.cpp)

One other random thing, some of kobold.cpps sampling presets are very funky with new models. I’d recommend resetting everything to off, then start with like 0.4 temp, 0.04 MinP, 0.02/1024 rep penalty and 0.4 DRY, not the crazy high temp sampling they normally use, with newer models than llama2.

I can host specific model/quantization on the kobold.cpp API to try if you want, to save tweaking time. Just ask (or PM me, as replies sometimes don’t send notifications).

Good luck with exams! No worries about response times, /c/localllama is a slow, relaxed community.