Finding
Yes. All three modes give bit-identical output, and switching is refused while a request is running. With every expert layer on the CPU, hoisted VNNI tiles raised prefill by 25–31% over per-row tiles. Decode barely changed. 30cec2f now serves the 30B-A3B in hoisted + VNNI mode.
| Input | Mode | Runs | Prefill | Decode | Throttled ms |
|---|---|---|---|---|---|
| 512 | 0,0 | 4 | 178.59 | 28.47 | 2,468 |
| 512 | 1,0 | 4 | 200.96 | 30.56 | 2,179 |
| 512 | 1,1 | 4 | 234.96 | 30.58 | 1,852 |
| 4,096 | 0,0 | 2 | 184.28 | 25.82 | 14,522 |
| 4,096 | 1,0 | 2 | 194.50 | 26.03 | 14,128 |
| 4,096 | 1,1 | 2 | 229.76 | 25.57 | 10,237 |
Paired prefill ratios against mode 0,0: at 512 tokens, 1.10× for hoisting and 1.31× with VNNI (four pairs). At 4,096 tokens, 1.06× and 1.25× (two pairs). These are small samples, and the ranges shown in the data are observed spreads, not confidence intervals.
This test kept all 48 expert layers on the CPU to leave GPU headroom on a shared host. The serving setup keeps 40 there, so these rates don’t compare directly with the other studies. That gap is likely part of why the serving before/after showed no gain.
What was checked
- 18 CPU operator cases matched bit for bit across all three modes. A debugger breakpoint confirmed native VNNI instructions ran.
- Both GPUs returned 1,024 exact matrix outputs each.
- Switching during an active or queued request returns 409 and changes nothing. Invalid modes, unsupported combinations and non-loopback clients are rejected.
- Every completion response, streamed or not, is stamped with the tile mode and generation that produced it.
- When the control is disabled, its endpoints don’t exist. Multi-model hot-swap processes refuse it with 501.
- All 18 scored real-model outputs matched within each prompt length.
Method
- One process, three modes. Mode order was shuffled within each pair (seed 20261103).
- Each switch was followed by one excluded full-length warmup and one scored request.
- Exact 512- and 4,096-token fixtures, 128 output tokens, temperature zero, seed 42, prompt cache off.
- 8 generation and 20 batch threads, 8,192 context, Q8_0 KV cache, no speculation, process at nice 10.
Excluded attempts
The first attempt stopped when a sampled thread exited mid-read and the kernel returned ESRCH. The three runs it had produced were excluded, the harness was fixed, and the whole schedule restarted with fresh warmups.
Limits
- The host was shared, so neighbouring work and CPU quota throttling affect results. Cgroup counters include other processes.
- Phase boundaries use the arrival of the first streamed content, which includes some transport buffering.
- The CUDA build was incremental, with cached objects verified by hash, rather than a full clean rebuild.