Tile packed-A traversal by GEMM_UNROLL_M so the RVV TRSM solver matches the GEMM/itcopy packing contract on non-x280 RISC-V targets. This keeps the RVV solve path while avoiding VLEN-dependent corruption when VSETVL_MAX differs from the target GEMM unroll width.
Signed-off-by: Xiaofei Gong <gongxiaofei24@iscas.ac.cn>
Signed-off-by: YuanSheng <yuansheng@isrc.iscas.ac.cn>
Per review, the U74-specific DGEMM micro-kernel does not belong under
kernel/generic (reserved for portable, architecture-neutral C kernels).
Move both the hybrid dispatcher and its readable assembly source into
kernel/riscv64, next to KERNEL.U74:
- git mv kernel/generic/gemmkernel_4x4_u74.c -> kernel/riscv64/
- git mv kernel/generic/kern_u74.S -> kernel/riscv64/
- KERNEL.U74: DGEMMKERNEL ../generic/gemmkernel_4x4_u74.c
-> ../riscv64/gemmkernel_4x4_u74.c
- gemmkernel_4x4_u74.c: #include "conversion_macros.h"
-> #include "../generic/conversion_macros.h"
(that header stays under kernel/generic; same form kernel/wasm
already uses for it)
No functional change. The portable gemmkernel_4x4.c (SGEMMKERNEL) is
untouched and stays in kernel/generic. DGEMMKERNEL still resolves through
$(KERNELDIR), so both the Makefile ($(KERNELDIR)/$(DGEMMKERNEL)) and CMake
(GenerateNamedObjects "${KERNELDIR}/${DGEMMKERNEL}") builds pick the kernel
up from its new location.
Adds a hand-written RV64GC scalar 4x4 DGEMM micro-kernel (kern_u74.S) and
wires it as the U74 DGEMMKERNEL via a hybrid dispatcher
(gemmkernel_4x4_u74.c): the fast path (bm,bn multiples of 4, even bk,
non-TRMM) runs the asm; all other shapes, odd bk, and the TRMM builds fall
back to the portable C 4x4 kernel.
The asm uses a 4x4 register tile (16 accumulators), full operand double-
buffering (P/Q ping-pong) with one-iteration lookahead, and load-before-FMA
issue ordering matched to the U74's dual-issue in-order front end. Probes
show this reaches the FP-pipe peak (~16.5 cycles / 16 fmadd.d = 2.9 GF on
L1-resident data); the streaming plateau is memory-latency-bound, not the
schedule.
Measured on a VisionFive 2 (single-core, KC=256): micro-kernel 1.88 vs 1.54
GF (+22%), full blocked DGEMM 1.77 vs 1.48 GF (+20%); the advantage holds
under 4-core contention (+17%). End-to-end HPL N=10000 (4 cores, Q=256):
5.17 vs 4.97 GF (+4.0%), residual PASSED. Correctness validated against the
full BLAS Level-3 test suite (DGEMM 17,496 computational calls, 0 failures).
kern_u74.S is the readable source; the .c embeds it via top-level __asm__ so
it builds as a single OpenBLAS kernel object with no build-system changes.
The SiFive U74 (RV64GC; e.g. StarFive JH7110 / VisionFive 2) is a scalar,
in-order core with no RVV, so today it falls back to RISCV64_GENERIC whose
S/D GEMM uses the generic 2x2 C micro-kernel.
Per the U74 Core Complex Manual (Table 169) fmadd.d has a 7-cycle latency
at repeat rate 1 (fully pipelined). A 2x2 tile exposes only 4 independent
accumulator chains -- fewer than the FMA latency -- so the FP pipe stalls
on the accumulator dependency, and the 1:1 load:FMA ratio saturates the
single load/store pipe ("only one outstanding line fill", manual 8.2).
This adds a portable 4x4 GEMM micro-kernel and a dedicated U74 target:
- kernel/generic/gemmkernel_4x4.c: 16-accumulator 4x4 register tile. 16
independent chains exceed the 7-cycle latency, and the load:FMA ratio
drops to 1:2. 16 acc + 4 A + 4 B fit RV64G's 32 FP registers without
spilling. Full 4/2/1 edge handling in both M and N.
- U74 target wiring: getarch.c (FORCE_U74, 32 KiB/64 B L1D, 2 MiB L2),
param.h (S/D UNROLL 4/4; complex stays 2/2), kernel/riscv64/KERNEL.U74
(S/D GEMM -> gemmkernel_4x4 + gemm_[nt]copy_4; S/D TRMM -> existing
trmmkernel_4x4), Makefile.prebuild + Makefile.riscv64 (-mtune=sifive-u74),
TargetList.txt, cpuid_riscv64.c.
The 4x4 kernel was verified numerically against a naive reference GEMM,
driven through the real gemm_tcopy_4 / gemm_ncopy_4 packing routines,
across 27,436 M/N/K x alpha combinations covering every 4/2/1 tail case:
worst absolute error 0.
Build with: make TARGET=U74
The general-stride ROTM path already computes kx/ky with the BLAS negative-increment starting offset. Converting negative increments to positive strides and moving dx/dy again double-adjusted the address and could access the wrong elements. Keep the signed byte strides for RVV strided loads and stores.
ROTM has loop-carried dependencies when incx or incy is zero because the same element is updated repeatedly across iterations. The RVV strided load/store path would compute lanes from the same old value and write them back in parallel, producing results that do not match BLAS ROTM semantics. Use a scalar path for zero-stride cases.
Optimize the RVV implementation of ZROT when inc_x and inc_y are
non-unit strides (inc_x != 1, inc_y != 1).
Reorder several operations to reduce vector register pressure and
avoid unnecessary vector register spill to the stack. This helps GCC
keep vector values in registers and reduces redundant spill/reload
instructions, improving runtime performance.
No functional change.
Signed-off-by: daichengrong <daichengrong@iscas.ac.cn>
Felix-Gong