First local inference experiments on the RTX 5060 Ti
After putting the 16 GB RTX 5060 Ti inside ai-01 via PCI passthrough on Proxmox, I ran some simple llama.cpp tests before thinking about a more robust stack. The chain was Proxmox → VFIO → Ubuntu Server 26.04 → NVIDIA 595-open → CUDA 13.x → llama.cpp → GGUF.
I used the same prompt for every test and kept -ngl 99 to offload almost everything to the GPU:
| Model | Quantization | Prompt (t/s) | Generation (t/s) |
|---|---|---|---|
| Qwen3-4B | Q4_K_M | 797.7 | 125.6 |
| Qwen3-8B | Q4_K_M | 677.9 | 78.9 |
| Qwen3-14B | Q4_K_M | 493.4 | 44.9 |
| Qwen3-14B | Q5_K_M | 459.6 | 39.4 |
The Qwen3-14B in Q5_K_M took about 14.8 GB of the 16.3 GB available, so it seems very close to the card's comfortable limit. Even so, almost 40 tokens per second is more than enough for interactive use. So far this 14B Q5 looks like the most interesting point between model size, quantization quality and speed.
On top of these models I'm also testing coding-agent harnesses. DSH (DeepSeek Harness) is the main one; OpenCode and Pi (coding agent) are queued next.
The next experiment is to deliberately exceed 16 GB of VRAM with larger models, in the 20B to 32B range, to observe the cost of splitting the model between GPU and RAM. The question that started as "which model is fastest?" is already turning into something more useful: what is the best model that still answers faster than I can read?
2026-09-06 — before the VRAM experiment, the first real harness test happened. I ran llama-server on ai-01 with the bartowski Qwen3.8-27B-IQ3_XXS GGUF — a 27B model that lives 100% in VRAM, at 15188 MiB of the 16311 MiB available — and pointed DSH, running on the MacBook, at the VM. First time the harness and the model were on different machines: the laptop stays the interface, the workstation does the work.
One friction point: DSH expects an API key, and llama-server does not issue one. The fix was trivial — the field accepts any value, so it now says local, a placeholder honest about what it is.
The results are satisfactory for interactive use. Before the 20B-to-32B experiment, two things to record: whether the IQ3_XXS quantization holds up under long agent sessions, and how much latency the hop from the MacBook to the VM actually adds.
