kernel/wasm/KERNEL is included after KERNEL.WASM128_GENERIC and
unconditionally pointed S/D GEMV at RISC-V scalar gemv_{n,t}.c, so a
target-file SIMD kernel would never run.
Add kernel/wasm/gemv_{n,t}.c for both precisions. GEMV_T keeps 8
(float) / 4 (double) independent column accumulators with IEEE mul+add
and delays the horizontal add until after the inner loop. GEMV_N
streams four columns into y with a four-lane v128 unroll. Non-unit
stride stays scalar (no WASM gather/scatter).
Guard SGEMV{N,T}KERNEL/DGEMV{N,T}KERNEL with ifndef in KERNEL so the
target file wins. CGEMV/ZGEMV stay RISC-V scalar.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Vectorize the packed 2x2 CGEMM/ZGEMM inner loop with f32x4/f64x2
complex mul. Do not let kernel/wasm/KERNEL override the target
kernel the way 4x4 real GEMM already guards TRMM.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Generic scalar DTRMM was the remaining Level-3 gap (~15 GFLOPS vs
~23 DGEMM). Keep SIMD for double only; single-precision TRMM already
auto-vectorized and a shared S+D kernel slowed SGEMM.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
kernel/wasm/KERNEL is included after KERNEL.WASM128_GENERIC and
unconditionally pointed S/D AXPY at RISC-V scalar axpy.c, so the
x86_64 V_SIMD sources listed in the target file never ran.
Add kernel/wasm/axpy.c for both precisions. Unit-stride y += da * x
uses eight independent v128 lanes (32 floats / 16 doubles) with IEEE
mul+add; remainder is one vector then scalar. Skip relaxed madd:
AXPY is checked to machine epsilon, and putting it in the generic
V_SIMD path previously slowed Level 1. Non-unit stride stays scalar
(no WASM gather); inc==0 uses that path so y[0] += n * da * x[0].
Guard SAXPYKERNEL/DAXPYKERNEL with ifndef in KERNEL so the target
file wins. CAXPY/ZAXPY stay RISC-V scalar.
On n=512..8192 vs the RISC-V champion, SAXPY is about 1.4–3.0x and
DAXPY about 1.4–2.2x. test.sh is green.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Switch WASM128_GENERIC to unroll 4 with matching ncopy/tcopy and TRMM
4x4 so packing width stays consistent. Use relaxed SIMD madd in the
inner loop when compiled with -mrelaxed-simd.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Keep production STRSM/DTRSM free of always-on increments; enable the
counters only when counting invocations.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
The skylakex/cooperlake d/s/c/z asum and c/z sum microkernels peel leading
elements until the input pointer reaches a 64-byte boundary (a scalar loop in
dasum/sasum, a masked header load in the complex variants) before entering an
aligned-load accumulator loop. The peel count depends on the buffer address
mod 64, so the grouping of the sum into accumulators - and therefore the
rounding of the result - depends on where the caller's buffer happens to sit
in memory. The same data at a different address can give a bitwise-different
sum.
That address dependence surfaced as non-reproducibility in OrcaFlex: LAPACK's
dstein scales each inverse-iteration eigenvector by 1/dasum(...) over a heap
array whose alignment varies with allocation history, so eigenvectors from
identical inputs differed run to run in the last bits, which zero-tolerance
regression comparison flags.
Fix by dropping the alignment peel and using unaligned loads throughout, so
the summation order is a function of the length alone. On AVX-512 hardware
unaligned load instructions on addresses that happen to be aligned cost the
same as aligned loads; only genuinely split cache lines pay a small penalty,
negligible for these level-1 reductions.
DYNAMIC_ARCH builds CORTEXA72 as a separate kernel and pull
dneg_tcopy from generic/neg_tcopy_$(DGEMM_UNROLL_M).c. Width 6 was
missing (only 1/2/4/8/16 existed), which broke the arm64 Graviton
Cirun and Azure DYNAMIC_ARM64 jobs.
Co-authored-by: Cursor <cursoragent@cursor.com>
TARGET=CORTEXA72 previously reused the A57 8x4 path. Add a dedicated
6x8 NEON ukernel, contiguous MR=6 panel packers (stock gemm_*copy_6 is
4+2), and UNROLL_M=6-aware TRSM kernels so HPL/dtrsm does not corrupt
the heap. DTRMM falls back to generic 2x2 until a matching kernel exists.
Co-authored-by: Cursor <cursoragent@cursor.com>
continuous build / neoverse_build (push) Canceled after 0s
continuous build / neoverse_n1_build (push) Canceled after 0s
continuous build / neoverse_n1_omp_build (push) Canceled after 0s
continuous build / neoverse_n1_ilp64_build (push) Canceled after 0s
harmonyos / build (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=GENERIC, DYNAMIC_ARCH, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA264, LA264, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA464, LA464, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA64_GENERIC, LA64_GENERIC, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON2K1000, LOONGSON2K1000, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON3R5, LOONGSON3R5, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSONGENERIC, LOONGSONGENERIC, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=GENERIC, DYNAMIC_ARCH) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA264, LA264) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA464, LA464) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA64_GENERIC, LA64_GENERIC) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON2K1000, LOONGSON2K1000) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON3R5, LOONGSON3R5) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSONGENERIC, LOONGSONGENERIC) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=I6400, I6400, mipsisa64r6el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=I6500, I6500, mipsisa64r6el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=MIPS64_GENERIC, MIPS64_GENERIC, mips64el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=P6600, P6600, mipsisa64r6el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=SICORTEX, SICORTEX, mips64el-linux-gnuabi64) (push) Canceled after 0s
riscv64 zvl256b qemu test / TEST (TARGET=RISCV64_GENERIC BINARY=64 ARCH=riscv64 DYNAMIC_ARCH=1, rv64,g=true,c=true,v=true,vext_spec=v1.0,vlen=256,elen=64, DYNAMIC_ARCH=1) (push) Canceled after 0s
riscv64 zvl256b qemu test / TEST (TARGET=RISCV64_ZVL128B BINARY=64 ARCH=riscv64, rv64,g=true,c=true,v=true,vext_spec=v1.0,vlen=128,elen=64, RISCV64_ZVL128B) (push) Canceled after 0s
riscv64 zvl256b qemu test / TEST (TARGET=RISCV64_ZVL256B BINARY=64 ARCH=riscv64 BUILD_BFLOAT16=1 BUILD_HFLOAT16=1, rv64,g=true,c=true,v=true,vext_spec=v1.0,vlen=256,elen=64,zfh=true,zvfh=true,zvfbfwma=true, RISCV64_ZVL256B) (push) Canceled after 0s
Pure cosmetic cleanup; no functional changes.
Convert all tab-indented code to 4-space indent, align macro
continuation backslashes, switch brace placement to consistent K&R
style, reformat multi-line vector initialisers, and remove trailing
whitespace on SAVE4x2_ACC1_SCALAR.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Add BGEMM (BF16 input → BF16 output) for POWER10 by reusing the
existing SBGEMM kernel infrastructure. A -DBGEMM compile flag switches
only the store path; the xvbf16ger2pp MMA instruction and BF16 packing
routines are shared with SBGEMM unchanged.
Changes
-------
kernel/power/KERNEL.POWER10
- Register BGEMM kernel and copy-routine targets, reusing the
sbgemm_ncopy/tcopy sources (packing layout is identical).
kernel/power/sbgemm_kernel_power10.c
- Under BGEMM: force v4sf_t to float so accumulators stay in
float32; add STORE4_BF16/STORE2_BF16 macros (read BF16 C, widen,
apply alpha*acc, convert back via xvcvspbf16, store); add
f32_to_bf16_scalar() for m&1/n&1 tails; add BGEMM variants of
all SAVE_ACC macros covering the full m/n tile hierarchy.
param.h
- Add BGEMM_DEFAULT_UNROLL_M=16, UNROLL_N=8, P/Q/R blocking
parameters for POWER10, matching the 16x8 kernel tile.
Unit test (utest/test_extensions/test_bgemm.c)
- Uses SBGEMM as a trusted reference. 21 test cases cover all four
transpose combinations, all m/n remainder paths, odd-k, alpha=0,
beta=0/1, and the m>=32 fast path. Tolerance 0.01 to account for
the one extra BF16 rounding on the BGEMM store.
Performance (POWER10, single-threaded, Transa=N Transb=N)
----------------------------------------------------------
Step=1 (sizes 1–200):
- Sizes 1–80: baseline 0.1–128 MFlops (scalar fallback);
patch 0.1–39,656 MFlops (up to ~350x faster at M=80).
- Sizes 81–200: patch 3x–24x faster (avg ~10x) with 16×8 MMA tile
engaged.
Step=8 (sizes 8–1024):
- Baseline peaks at ~2,950 MFlops (avg ~1,746 MFlops).
- Patch sustains 75,000–1,009,866 MFlops for sizes 384–960,
exceeding 1 TFlops at large sizes.
- Median speedup ~99x; average speedup ~140x across all sizes.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Tile packed-A traversal by GEMM_UNROLL_M so the RVV TRSM solver matches the GEMM/itcopy packing contract on non-x280 RISC-V targets. This keeps the RVV solve path while avoiding VLEN-dependent corruption when VSETVL_MAX differs from the target GEMM unroll width.
Signed-off-by: Xiaofei Gong <gongxiaofei24@iscas.ac.cn>
Signed-off-by: YuanSheng <yuansheng@isrc.iscas.ac.cn>
Keep the logical target for dynamic dispatch names while selecting kernel sources and capabilities from the core chosen by getarch after ISA restrictions.
* Add transpose support for C in GEADD (fixes#4646)
Extends GEADD to support independent transposition of both A and C,
matching the behavior of cuBLAS's geam and Apple's Accelerate geadd.
Previously only A could be transposed.
- Add transc parameter across cblas.h, common_interface.h,
common_level3.h, common_param.h
- Add transc handling to interface/geadd.c and interface/zgeadd.c
- Extend kernel/generic/geadd.c and kernel/generic/zgeadd.c with
stride logic for transposed C
- Add transpose test coverage (hand-verified 2x2 cases and randomized
large-matrix tests) for sgeadd, dgeadd, cgeadd, zgeadd
* Add self to CONTRIBUTORS.md
Extends GEADD to support independent transposition of both A and C,
matching the behavior of cuBLAS's geam and Apple's Accelerate geadd.
Previously only A could be transposed.
- Add transc parameter across cblas.h, common_interface.h,
common_level3.h, common_param.h
- Add transc handling to interface/geadd.c and interface/zgeadd.c
- Extend kernel/generic/geadd.c and kernel/generic/zgeadd.c with
stride logic for transposed C
- Add transpose test coverage (hand-verified 2x2 cases and randomized
large-matrix tests) for sgeadd, dgeadd, cgeadd, zgeadd
Per review, the U74-specific DGEMM micro-kernel does not belong under
kernel/generic (reserved for portable, architecture-neutral C kernels).
Move both the hybrid dispatcher and its readable assembly source into
kernel/riscv64, next to KERNEL.U74:
- git mv kernel/generic/gemmkernel_4x4_u74.c -> kernel/riscv64/
- git mv kernel/generic/kern_u74.S -> kernel/riscv64/
- KERNEL.U74: DGEMMKERNEL ../generic/gemmkernel_4x4_u74.c
-> ../riscv64/gemmkernel_4x4_u74.c
- gemmkernel_4x4_u74.c: #include "conversion_macros.h"
-> #include "../generic/conversion_macros.h"
(that header stays under kernel/generic; same form kernel/wasm
already uses for it)
No functional change. The portable gemmkernel_4x4.c (SGEMMKERNEL) is
untouched and stays in kernel/generic. DGEMMKERNEL still resolves through
$(KERNELDIR), so both the Makefile ($(KERNELDIR)/$(DGEMMKERNEL)) and CMake
(GenerateNamedObjects "${KERNELDIR}/${DGEMMKERNEL}") builds pick the kernel
up from its new location.
Full-memory HPL N=27456 (asm hybrid + Q=256) measured at 5.99 GFLOPS
(residual PASSED, ~50% of the 12 GF peak, +10.7% over the tuned C kernel,
1.81x the stock 2x2) - the best clean figure. Doc-only header update.
Adds a hand-written RV64GC scalar 4x4 DGEMM micro-kernel (kern_u74.S) and
wires it as the U74 DGEMMKERNEL via a hybrid dispatcher
(gemmkernel_4x4_u74.c): the fast path (bm,bn multiples of 4, even bk,
non-TRMM) runs the asm; all other shapes, odd bk, and the TRMM builds fall
back to the portable C 4x4 kernel.
The asm uses a 4x4 register tile (16 accumulators), full operand double-
buffering (P/Q ping-pong) with one-iteration lookahead, and load-before-FMA
issue ordering matched to the U74's dual-issue in-order front end. Probes
show this reaches the FP-pipe peak (~16.5 cycles / 16 fmadd.d = 2.9 GF on
L1-resident data); the streaming plateau is memory-latency-bound, not the
schedule.
Measured on a VisionFive 2 (single-core, KC=256): micro-kernel 1.88 vs 1.54
GF (+22%), full blocked DGEMM 1.77 vs 1.48 GF (+20%); the advantage holds
under 4-core contention (+17%). End-to-end HPL N=10000 (4 cores, Q=256):
5.17 vs 4.97 GF (+4.0%), residual PASSED. Correctness validated against the
full BLAS Level-3 test suite (DGEMM 17,496 computational calls, 0 failures).
kern_u74.S is the readable source; the .c embeds it via top-level __asm__ so
it builds as a single OpenBLAS kernel object with no build-system changes.
Julien Jerphanion