Why it was not adopted
No workload met the predeclared deployment bar: at least 2% lower median paired request latency, a 95% interval wholly below parity, and no material decode or memory regression. Every request-latency interval crossed parity. Prefill showed no repeatable improvement, and no deployable whole-model break-even threshold was established.
Full racecheck also remained an unresolved qualification gate. Its shared-memory warnings occurred in the unchanged routing helper and reproduced in the baseline. They were not attributed to the new traversal, but were not dismissed as false positives. The candidate remains experimental and disabled by default; it was not deployed.
| Input tokens | Baseline s | Candidate s | Paired change | 95% interval |
|---|---|---|---|---|
| 512 | 3.2774 | 3.2483 | -0.47% | -5.36% to +2.09% |
| 768 | 3.2829 | 3.2871 | +0.52% | -1.13% to +7.83% |
| 1024 | 3.4647 | 3.4939 | +0.68% | -2.56% to +1.71% |
| 2048 | 5.0505 | 5.0348 | -0.50% | -1.66% to +1.99% |
| Input tokens | Prefill ms | Decode ms |
|---|---|---|
| 512 | 1488.47 → 1488.89 | 1777.81 → 1757.25 |
| 768 | 1574.24 → 1576.26 | 1704.30 → 1711.72 |
| 1024 | 1675.77 → 1677.80 | 1786.42 → 1813.28 |
| 2048 | 3139.17 → 3145.96 | 1904.31 → 1880.52 |
Separate arm medians need not produce the median paired ratio when divided. Decode covers 64 generated tokens. Decode variability was substantial; these trials do not establish noninferiority against a narrow regression margin.
What changed from the first experiment
The earlier operator study found gains only for large fixtures and regressions on the paired kernel. This follow-up preserves the paired two-token path unchanged. It reuses the existing compact expert grouping and quantization, traversing work in bands of four output-row tiles without a new routing sort. Per-tile arithmetic and accumulation order are preserved.
The opt-in guard admits only SM70 Q4_K/Q6_K down projections with K=768, 2,048 output rows, 128 experts, eight routed experts per token and exactly 768 or 1,024 independent tokens, with compatible compact-mode settings and tile width. Other shapes and decode fall back to the existing implementation. A 2,048-token prefill is split into two qualifying 1,024-token microbatches; 512 tokens is a fallback control.
Operator gains did not establish a deployment threshold
Exploration covered both projection geometries, ten sizes and three expert pools on both GPUs. Up-projection and 512-token results were mixed or regressed, so the final candidate was restricted before model timing. In the final down-projection measurements, including routing and quantization, the median of 12 condition medians was 1.98% faster at 768 tokens and 1.11% faster at 1,024. Individual upper confidence bounds were below parity in 11/12 and 10/12 conditions respectively.
These are condition-dependent operator gains. The lowest selected exploratory size, 768, is not a universal break-even guarantee. Neither the measured operator benefit nor the restricted guard delivered a repeatable whole-model improvement.
Correctness, sanitizers and memory
- Eight baseline-versus-candidate-disabled/enabled comparisons were bit-exact over 65 full vocabulary-logit vectors each (39,503,360 bytes). All 64 generated token IDs per run were compared separately and matched. All timed request pairs also produced identical text.
- 298 Python tests, 240 operator conditions with changed-input CUDA-graph replays, 370 backend operator cases and 12 paired-path regression runs passed. The paired-path source was unchanged.
- Memcheck, initcheck and synccheck each passed eight final fixtures across both GPUs, quantizations and admitted sizes. Full racecheck returned warnings in all eight candidate fixtures; four baseline fixtures reproduced the same routing-helper warning sites. A clean full sanitizer pass is not claimed.
- Sampled aggregate GPU increments were identical between arms: 5.8672 GiB at 512 tokens, 5.8984 GiB at 768, and 5.9336 GiB at 1,024 and 2,048. Per-device peaks also matched. Host peak RSS was about 14.6 GiB, with no material increase. The download includes per-device peaks and worst observed runs.
Method and limits
Both arms used the same model, server executable, placement and settings; only the engine library differed. The control runtime source matched the current expXa implementation. Each arm started a fresh isolated process, loaded and warmed the model, then generated 64 greedy tokens with a fixed seed and EOS ignored. Prompt caching was disabled and scored token counts were asserted. Request latency includes the full streamed response after warmup, excluding model loading.
Settings: context 8,192; batch 2,048; microbatch 1,024; one sequence; equal layer split over two GPUs; 40 CPU expert layers; 8 generation and 20 batch threads; flash attention; Q8_0 KV cache. This two-GPU placement is specific to the experiment. A 512-token microbatch configuration cannot activate this candidate. Activation traces confirmed qualifying work and fallback; tracing was off during scored timing.
Eight pairs per length alternated AB/BA. Approximate per-workload 95% intervals use 20,000 paired bootstrap resamples, without early stopping or favorable-round selection. They are not simultaneous confidence bounds, and deterministic alternation is not randomized assignment. One model and one prompt pattern do not establish results for other models, inputs or placements. Logit checks cover the final prefill vector and successive decode states, not every intermediate prefill token.
Memory was sampled every 0.1 seconds through startup, warmup and scoring. GPU increments subtract the pre-run allocation; RSS includes file-backed model pages. Sampled peaks can miss short transients. Timing alone does not establish cache reuse as the cause of an operator gain.
Excluded attempts
Earlier development attempts included a probe compile error, incorrect incumbent tile-width control, insufficient sanitizer fixtures, graph-reference ordering issues, multi-GPU sanitizer initialization errors, controller contention and an argument-quoting error. Those were corrected before final qualification. A model group interrupted by unrelated maintenance was excluded and rerun after stability was confirmed. Exploratory and single-pair timings were not pooled with the final 32 pairs. The adverse full-racecheck result remains part of the decision.