Warp sharing,
measured.
Expert IDs, weight fragments and attention scalars were tested across both V100s. The tested sharing changes did not earn promotion. Rust and Kairo keep their verified live runtime.
What was tested
Two V100 SXM2 32GB GPUs on the existing production host, CUDA 12.8.93, driver 580.178.04, SM70. Standalone synthetic operators used the unchanged CUDA source from live expXa 47f0b50. Serving stayed loaded and idle during each bounded test window. No model weights, offload settings, KV formats, context limits or runtime switches changed.
There are 15,408 scored timing batches: 12 balanced repetitions per configuration, 64 expert or 16 attention launches per batch. All final matrices passed bitwise output checks. Quick sanitizer runs and invalid harness attempts are excluded from performance summaries. Negative percentages mean less operator time; positive means more. These are not tokens/second or end-to-end serving gains.
Expert work: existing reuse already matters
Kairo's native two-token Q6_K down projection already broadcasts expert IDs with warp shuffles and groups matching experts. We tested explicit paired weight decoding, lane-zero reductions, block-shared routing, warp-shared scales/high bits and every combination. Coverage spans every overlap count 0–8, padded strides and negative-ID tails.
| Change | Time vs actual incumbent | Time vs matched research control |
|---|---|---|
| Paired Q6 fragment hoist | -0.76% | +0.31% |
| Lane-zero expert reduction | -0.76% | +0.31% |
| Block-shared expert routing | +2.45% | +3.73% |
| Warp-shared Q6 metadata | +1.51% | +2.69% |
The research routing/launch control itself ran -1.06% relative to the actual incumbent. Hoisting and root-only reduction were faster than the incumbent but slower than that matched control. The control differs in routing/host launch structure, so this small result cannot be attributed to warp sharing or treated as a serving gain. No candidate was deployed.
Disassembly showed the same static global-load instruction counts for the incumbent, control and explicit paired hoist. Warp metadata sharing added 16 indexed shuffles to the compiled kernel. Static instruction counts do not measure DRAM transactions or prove a cache-hit rate.
Attention: keep the required broadcasts
Kairo's active 256-wide, eight-head, 4×8 query schedule uses the Volta MMA kernel. Its maxima and rowsums feed multiple output fragments; removing their distribution would be incorrect. We instead tested broadcasting common rescaling exponentials and final reciprocal denominators.
| Change | Active 4×8 schedule | Comparison 8×8 schedule |
|---|---|---|
| Rescaling exponential broadcast | +18.87% | -1.62% |
| Normalization reciprocal broadcast | +1.42% | +0.65% |
| Both broadcasts | +20.35% | -0.93% |
The 4×8 path regressed. A small benefit in some 8×8 cases does not qualify changing the live schedule. Fixtures cover 1/2/4 real queries, 256/4,096/32,768 KV rows and masked/unmasked inputs, with independent dense-reference checks. The probe invokes the real process-tile implementation, but omits serving's Q8-to-F16 conversion and stream-K launch scheduling.
Rust's Q8 vector attention
Head-128 native Q8_0 K/V fixtures tested both accumulation modes. Root-only reductions also restrict their scalar consumers to lane zero; later normalization reductions still broadcast to every lane that writes an output. Tests cover 1/2 queries, 256/4,096/8,192 KV rows, one/four partitions and masks, comparing all outputs and partition metadata.
| Change | FP16 accumulation | FP32 accumulation |
|---|---|---|
| Root reductions + lane-zero consumers | +0.39% | +0.00% |
| Common exponential broadcast | +0.71% | -0.32% |
| Both changes | -0.24% | +0.16% |
Results vary by shape and do not establish a general improvement. These are separate operators from Rust's CPU expert banks; GPU shuffles cannot optimize the CPU portion. Existing CPU hoisting, parallel preparation and VNNI settings remain unchanged.
Correctness, safety and exclusions
Final checks include 1,152 expert, 216 MMA and 288 vector candidate/control output comparisons across both cards. Complete matrices and before/after final binary hashes are required for scoring. Quick memory and synchronization checks report zero errors. Both live process identities, mapped libraries and effective profiles were preserved.
Two early MMA harness mistakes were caught and corrected: an output-head offset and weak CUDA template symbols that collapsed the arms. Those attempts are excluded. The final build verifies eight distinct kernel symbols. Earlier expert/vector sweeps without bracketed probe hashes were rerun and excluded from scoring. All measurements, including adverse and invalid attempts, remain archived.
The benchmark controller tracks an owned remote process group. Both its parent and watchdog require verified cleanup before reopening admission. Injected operator exceptions and abrupt operator death were tested using an owned sleep process; both restored admission without changing live inference identities.
Read the complete record
Tables summarize medians of per-fixture paired time ratios across both GPUs. The download includes every accepted timing batch, shape-specific ratios, 2,000-resample descriptive bootstrap intervals and every combination. Intervals describe these fixtures, not production throughput. No new runtime was promoted and no full-model serving speedup is claimed.