LocalLLaMA

5654 readers

178 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago

MODERATORS

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works

PR merged: llama_moe_cache: run missed experts on CPU - Tested on 32GB RAM and 8GB VRAM with Tiel 35b, Gemma 26b, and LFM 24b (github.com)

submitted 1 week ago* (last edited 1 week ago) by BeefAndPoultry@lemmus.org to c/localllama@sh.itjust.works

4 comments fedilink hide all child comments

am17an: Before you guys bash llama.cpp again - I know it's not there yet. I'm working on it.

am17an: I'm working towards an "auto" mode here, thinking about a good UX.

I tried this on my Ryzen 2600, 32GB DDR4 RAM, RTX 2080 8GB, PCIe 3 desktop. The tuning script takes about 5 minutes to run but it gives you recommended settings. I tested Tiel 35b, Gemma 26b, and LFM 2.5 24b, all without mmproj (or with --no-mmproj-offload). If you have PCIe 5, your numbers will probably look different even with the same 8GB VRAM.

My real world numbers seem about 20% slower than what the tune script is predicting, but still I'm getting the fastest speeds I've ever seen in llama.cpp on my computer, and I've tried a few of these different attempts at MoE caching.

Tiel 35b

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 35.3 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 50.1 t/s
4/6 3136 MiB, share 0: 41.1 t/s
5/6 3136 MiB, share 0.5: 47.3 t/s
6/6 3136 MiB, share 0.95: 53.5 t/s

[The script pauses here for a few minutes...]

Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7284 MiB free, cache up to 3188 MiB

  cache MiB  share   t/s  hits
       1856      0  35.3   73%
       1856    0.5  38.6   66%
       1856   0.95  50.1   64%
       3136      0  41.1   81%
       3136    0.5  47.3   79%
       3136   0.95  53.5   73%

machine: uploads 10.9 GiB/s, CPU 40.8 GiB/s, CPU part 0.00 ms, overlap 1.4 ms/GiB, rest 14.14 ms per token, fit within 0.7%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  25.3  32.8  39.6  42.4  43.9  44.6
       1856  32.8  39.8  46.3  48.8  49.9  50.3
       3136  41.6  47.7  52.9  54.7  55.4  55.4

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.8 (52.9 t/s, 41.6 with share 0)

Interesting to note that 1856MB and 0.95 share is almost as fast as 3136MB and 0.8 share. This means it might be worth reducing the cache size in order to use MTP, or maybe I should start using a higher quant.

But looking at the "predicted" table, it seems like the fastest would actually be 3136 MB and 0.98 share for 55.4 tok/sec? Not sure why it doesn't recommend that instead, maybe it's an expected peak not average.

Forcing Lower VRAM Usage

You can use --max-mib to force it to tune your settings with a lower VRAM usage, just make sure you put this argument before the llama-server portion of the arguments. It seems to work in steps, so not every number is respected, and the initial data collection run still seems able to use more than your requested amount, but the recommendation will respect it.

$ python3 scripts/moe-cache-tune.py --max-mib 2500 -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 34.1 t/s
2/6 1856 MiB, share 0.5: 37.4 t/s
3/6 1856 MiB, share 0.95: 49.8 t/s
4/6 3136 MiB, share 0: 41.6 t/s
5/6 3136 MiB, share 0.5: 44.9 t/s
6/6 3136 MiB, share 0.95: 54.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7292 MiB free, cache up to 2500 MiB

  cache MiB  share   t/s  hits
       1856      0  34.1   73%
       1856    0.5  37.4   66%
       1856   0.95  49.8   64%
       3136      0  41.6   81%
       3136    0.5  44.9   79%
       3136   0.95  54.1   73%

machine: uploads 8.7 GiB/s, CPU 47.0 GiB/s, CPU part 3.06 ms, overlap -12.2 ms/GiB, rest 11.50 ms per token, fit within 1.3%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  23.2  31.6  39.4  42.8  44.7  45.7
       1856  31.4  38.3  45.7  48.8  50.3  50.9
       2496  36.9  42.5  49.1  51.8  53.0  53.4

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (51.8 t/s, 36.9 with share 0)

It recommended a slightly higher "share" meaning it allows more to run on the CPU, to reduce movement of data.

Testing with GPU Power Limit

I used sudo nvidia-smi -pl 125 to limit my GPU's power from 225 down to 125:

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 29.5 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 48.9 t/s
4/6 3136 MiB, share 0: 41.4 t/s
5/6 3136 MiB, share 0.5: 46.8 t/s
6/6 3136 MiB, share 0.95: 52.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7288 MiB free, cache up to 3192 MiB

  cache MiB  share   t/s  hits
       1856      0  29.5   73%
       1856    0.5  38.6   66%
       1856   0.95  48.9   64%
       3136      0  41.4   81%
       3136    0.5  46.8   79%
       3136   0.95  52.1   73%

machine: uploads 7.6 GiB/s, CPU 26.4 GiB/s, CPU part 0.00 ms, overlap 0.0 ms/GiB, rest 11.30 ms per token, fit within 7.9%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  21.1  28.8  36.1  39.3  41.0  41.7
       1856  29.0  37.0  44.9  48.0  49.4  49.8
       3136  39.4  47.5  54.6  57.2  58.1  57.8

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.9 (57.2 t/s, 39.4 with share 0)

You can see GPU power limits don't affect token generation much, it has a bigger impact on prefill speed. The cpu share is a little higher because the power limit makes the expert movement slightly less valueable. The predictions seem a bit... unpredictable. This run had slightly slower figures for every test, but ended up predicting a higher speed, who knows why.

Gemma 4 26b

With Gemma 4 26b QAT I got:

recommended: --moe-cache-mib 3328 --moe-cache-cpu-share 0.8 (50.1 t/s, 40.3 with share 0)

And with 131072 context:

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (45.7 t/s, 30.5 with share 0)

LFM 2.5 24b

With LFM 2.5 24b QAD I got:

recommended: --moe-cache-mib 5056 --moe-cache-cpu-share 0.9 (75.7 t/s, 51.5 with share 0)

I wonder why this one suggested a high share when the cache size nearly half of the model size. Maybe it's because the experts that aren't cached are infrequently used. So when your cache is big enough to hold every frequently used expert, they aren't worth moving back and forth.

you are viewing a single comment's thread
view the rest of the comments

[–] BeefAndPoultry@lemmus.org 1 points 17 hours ago* (last edited 17 hours ago)

I'm starting to think it's not worth pushing past a share of 0.95, even 0.95 can have issues. It can take too long to shuffle out experts that aren't being used anymore. I like to look at the predicted table and lean away from high shares. Like in my VRAM limited example:

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  23.2  31.6  39.4  42.8  44.7  45.7
       1856  31.4  38.3  45.7  48.8  50.3  50.9
       2496  36.9  42.5  49.1  51.8  53.0  53.4

A share of 0.9 is more than fast enough, even 0.8 might be good. I was trying 0.98 for a while and it would often get stuck in a rut where it was doing like 10 t/s for a long time as it was only moving 2% of experts per turn.

Now all I need is llama.cpp to support an option similar to Strata's STRATA_ROUTE_PRIOR (like a runtime version of REAP models, it biases towards using cached experts by however much you want)

And KV cache streaming https://github.com/ggml-org/llama.cpp/discussions/28216