Consume nearby data.
Count the scheduling cost.
Grouping expert work helps selected larger batches. Small batches and the existing paired path regress. No runtime was promoted.
The measured result
A separate alternating A/B confirmation tested stable expert grouping plus 32-row bands against the same Q6_K dot computation in job-major order. The 255-active-job case confirms a 3.8–4.2% time reduction. The 63-active-job case is about 0.7% slower and does not confirm its discovery-stage gain. The table includes the GPU grouping pass, not just the computation. Negative percentages mean less time.
| GPU | Job slots / active | Control µs | Grouped µs | Time change | Descriptive 95% interval |
|---|---|---|---|---|---|
| 0 | 64 / 63 | 97.28 | 97.98 | +0.66% | -0.60% to +0.79% |
| 0 | 256 / 255 | 353.92 | 339.10 | -4.21% | -4.24% to -4.17% |
| 1 | 64 / 63 | 97.47 | 98.08 | +0.66% | +0.59% to +0.76% |
| 1 | 256 / 255 | 356.58 | 343.10 | -3.76% | -3.81% to -3.66% |
These are standalone synthetic operator results, not model tokens/second. The larger independent-job fixture uses 32 expert IDs, 2,048 output rows and 512 input columns. It does not establish an improvement over the engine’s existing batched MoE implementation.
What changed
The research kernel retains expXa’s Q6_K × Q8_1 dot primitive and output indexing. A parallel GPU rank sort groups job indices by expert ID; work then traverses a small row band across those groups before advancing. This makes consumers of nearby weight tiles run closer together. It changes work assignment, not the hardware warp scheduler.
Six schedules cover job-major control, full-row expert sorting, and sorted bands of 4, 32, 128 and 512 rows. A seventh arm in the two-token family calls the actual native paired incumbent. The paired family preserves the existing matching-expert fusion. Sorting and band traversal are combined in arms 2–5; this study cannot attribute their combined effect to sorting alone.
Retained regressions
The 32-row schedule slows the existing paired kernel by roughly 11–21% including grouping. Smaller independent batches also regress. Even high expert overlap does not guarantee a win. A first serial-sort prototype lost badly once its grouping cost was charged; all of its accepted results remain in the download.
Complete discovery matrix for the 32-row schedule
| GPU | Family | Job slots | Expert pool bound | Inclusive time change |
|---|---|---|---|---|
| 0 | Native paired | 16 | 8 | +11.04% |
| 0 | Native paired | 16 | 32 | +20.49% |
| 0 | Independent jobs | 16 | 8 | +9.40% |
| 0 | Independent jobs | 16 | 32 | +20.29% |
| 0 | Independent jobs | 16 | 256 | +13.93% |
| 0 | Independent jobs | 64 | 8 | +2.01% |
| 0 | Independent jobs | 64 | 32 | -1.82% |
| 0 | Independent jobs | 64 | 256 | +4.60% |
| 0 | Independent jobs | 256 | 8 | +0.55% |
| 0 | Independent jobs | 256 | 32 | -4.87% |
| 0 | Independent jobs | 256 | 256 | +5.38% |
| 1 | Native paired | 16 | 8 | +11.24% |
| 1 | Native paired | 16 | 32 | +21.01% |
| 1 | Independent jobs | 16 | 8 | +9.13% |
| 1 | Independent jobs | 16 | 32 | +13.56% |
| 1 | Independent jobs | 16 | 256 | +12.89% |
| 1 | Independent jobs | 64 | 8 | +2.20% |
| 1 | Independent jobs | 64 | 32 | -1.58% |
| 1 | Independent jobs | 64 | 256 | +4.09% |
| 1 | Independent jobs | 256 | 8 | +1.04% |
| 1 | Independent jobs | 256 | 32 | -4.14% |
| 1 | Independent jobs | 256 | 256 | +5.24% |
Method and correctness
Both V100 SXM2 32GB GPUs, SM70, CUDA 12.8.93. Main sweep: 3,264 accepted timing batches; each averages 16 launches after three warmups. Twelve repetitions rotate and reverse candidate order. That discovery ordering is not fully position-balanced, so a separate confirmation fixes the 32-row arm in advance and alternates A/B versus B/A for twelve fresh repetitions per shape and GPU. Confirmation contributes 192 timing batches. Timings use CUDA events with fixed routes and repeated warm work.
Every normal fixture contains one negative-ID sentinel: 16, 64 and 256 job slots mean 15, 63 and 255 active jobs. Expert pool bounds 8, 32 and 256 are not distinct-count guarantees; actual cardinalities are in the download. A pool bound of 256 is not a no-reuse control. Grouping is charged once per operator invocation; CPU routing, graph execution, serving queues and whole-model latency are outside scope.
All 136 main-sweep candidate/output comparisons and 24 confirmation comparisons passed bitwise against their references. Three rows per active job were also checked against CPU dequantization and dot products. Quick final-binary memcheck and synccheck passed with zero errors. These sanitizer runs cover small fixtures; they are not exhaustive large-batch sanitizer coverage.
Paired ratios compare each repetition with its control. Intervals use 2,000 descriptive bootstrap resamples of those ratios. They do not cover workload diversity, shared-host variation or future requests. Full tables retain every schedule, shape, sample and adverse outcome.
What this does not prove
Nsight Compute returned ERR_NVGPUCTRPERM. The rental does not permit access to GPU performance counters, so DRAM traffic and cache-hit rates were not measured. We can establish an elapsed-time effect for these fixtures; we cannot establish cache residency as its cause. No permission or driver setting was changed.
This is a starting point for selective scheduling, not a general automatic reuse detector. No attention kernel, model output, full-model inference throughput or production scheduler change was qualified. All serving processes stayed resident; guarded test windows drained existing work and queued arrivals, then verified probe cleanup before reopening admission. No serving process was restarted.
Inspect the evidence
The download includes the parallel sweep, independent confirmation, original serial prototype, binary identities, actual expert cardinalities, paired ratios and uncertainty. Prior benchmarks remain available.