Add BGEMM (BF16 input → BF16 output) for POWER10 by reusing the
existing SBGEMM kernel infrastructure. A -DBGEMM compile flag switches
only the store path; the xvbf16ger2pp MMA instruction and BF16 packing
routines are shared with SBGEMM unchanged.
Changes
-------
kernel/power/KERNEL.POWER10
- Register BGEMM kernel and copy-routine targets, reusing the
sbgemm_ncopy/tcopy sources (packing layout is identical).
kernel/power/sbgemm_kernel_power10.c
- Under BGEMM: force v4sf_t to float so accumulators stay in
float32; add STORE4_BF16/STORE2_BF16 macros (read BF16 C, widen,
apply alpha*acc, convert back via xvcvspbf16, store); add
f32_to_bf16_scalar() for m&1/n&1 tails; add BGEMM variants of
all SAVE_ACC macros covering the full m/n tile hierarchy.
param.h
- Add BGEMM_DEFAULT_UNROLL_M=16, UNROLL_N=8, P/Q/R blocking
parameters for POWER10, matching the 16x8 kernel tile.
Unit test (utest/test_extensions/test_bgemm.c)
- Uses SBGEMM as a trusted reference. 21 test cases cover all four
transpose combinations, all m/n remainder paths, odd-k, alpha=0,
beta=0/1, and the m>=32 fast path. Tolerance 0.01 to account for
the one extra BF16 rounding on the BGEMM store.
Performance (POWER10, single-threaded, Transa=N Transb=N)
----------------------------------------------------------
Step=1 (sizes 1–200):
- Sizes 1–80: baseline 0.1–128 MFlops (scalar fallback);
patch 0.1–39,656 MFlops (up to ~350x faster at M=80).
- Sizes 81–200: patch 3x–24x faster (avg ~10x) with 16×8 MMA tile
engaged.
Step=8 (sizes 8–1024):
- Baseline peaks at ~2,950 MFlops (avg ~1,746 MFlops).
- Patch sustains 75,000–1,009,866 MFlops for sizes 384–960,
exceeding 1 TFlops at large sizes.
- Median speedup ~99x; average speedup ~140x across all sizes.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Extends GEADD to support independent transposition of both A and C,
matching the behavior of cuBLAS's geam and Apple's Accelerate geadd.
Previously only A could be transposed.
- Add transc parameter across cblas.h, common_interface.h,
common_level3.h, common_param.h
- Add transc handling to interface/geadd.c and interface/zgeadd.c
- Extend kernel/generic/geadd.c and kernel/generic/zgeadd.c with
stride logic for transposed C
- Add transpose test coverage (hand-verified 2x2 cases and randomized
large-matrix tests) for sgeadd, dgeadd, cgeadd, zgeadd
Amrita H S