Technology

Tool use with Gemma has been hit or miss. I wouldn't rely on it for anything unsupervised.

Tool use for Qwen3.6 has been great lately, but I do remember seeing some issues with it too, a while back. I don't remember when/why the issues cleared up (I have tweaked configs a bit over time), but switching to Pi definitely helped.

I do remember having a lot more problems in OpenCode and it was practically unusable (which is why my recent experience with VS Code was surprising). I'd definitely recommend trying Pi.

A fresh Pi install is very minimal by design. The system prompt is tiny, so it's a pretty good fit for small LLMs like these. It's sort of like Neovim: Nothing fancy out of the box, but you can add lots of fancy things to it. I containerize it because I don't like giving LLMs (especially these small ones) unrestricted access to my host computer – though, I have not seen any signs of it accidentally doing something destructive, which is surprising.

There are similar alternatives to Apple Container for Linux (e.g. Docker Sandboxes, muvm, Firecracker). There's also this thing made specifically for Pi called Gondolin. I haven't tried it yet, but I may end up switching to that if it could simplify my stack.

Here's my current llama-swap/llama.cpp config for Qwen3.6 35B-A3B:

qwen3.6-35b-a3b:
    name: "Qwen3.6 35B-A3B (Coding)"
    proxy: "http://127.0.0.1/:$%7BPORT%7D" # If you're seeing a `/` after `127.0.0.1` here, don't include it. I think something in Lemmy is trying to "sanitize" this input by adding the `/`.
    cmd: |
      llama-server
      --port ${PORT}
      --no-webui
      -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL
      --jinja
      --parallel 1
      --flash-attn on
      --no-mmproj
      --load-mode none
      --reasoning-preserve
      --ctx-size 190000
      --temp 0.6
      --top-p 0.95
      --top-k 20
      --min-p 0.0
      --presence-penalty 0.0
      --repeat-penalty 1.01

A few notes about this config:

• Now that I think of it, --reasoning-preserve might be another thing that helped with tool calls.
• Note the -MTP part of the -hf param. MTP helps speed things up. Here's the Huggingface page for this model
• You can also omit --no-mmproj if you need vision, but it might mean sacrificing speed or context size, so I usually just enable vision in a separate llama-swap model entry to use as needed.
• Unsloth recommends --repeat-penalty 1.0, but I saw the LLM enter a thinking loop in VS Code, so I bumped it up just a tiny bit to 1.01. I have since seen it do something that resembled the same thought loop, but it was able to recover on its own. Not sure if it's a coincidence or if 1.01 was actually the solution, so worth some experimentation.

If your MB has onboard graphics, maybe you could mask the GPU and just pass it off to a container running the LLMs I guess