expXa

Research

CUDA kernels

Volta GPU operator experiments: expert dot products, scheduling and warp-level sharing.

  1. Warp sharing in expert and attention kernels

    Testing warp-level sharing of expert IDs, weight fragments and attention scalars in expXa's Volta kernels on two V100s. No change earned adoption.

    Not adopted

  2. Reordering expert work for weight reuse

    Grouping MoE expert jobs by expert and traversing row bands on V100: about 4% less operator time at 255 active jobs, slower for small batches and the paired kernel.

    Not adopted

Other topics: Attention AVX-512 VNNI CPU kernels MoE CPU offload PXA comparison Q4_K_M Q6_K NVIDIA V100