kernel/wasm/KERNEL is included after KERNEL.WASM128_GENERIC and
unconditionally pointed S/D GEMV at RISC-V scalar gemv_{n,t}.c, so a
target-file SIMD kernel would never run.
Add kernel/wasm/gemv_{n,t}.c for both precisions. GEMV_T keeps 8
(float) / 4 (double) independent column accumulators with IEEE mul+add
and delays the horizontal add until after the inner loop. GEMV_N
streams four columns into y with a four-lane v128 unroll. Non-unit
stride stays scalar (no WASM gather/scatter).
Guard SGEMV{N,T}KERNEL/DGEMV{N,T}KERNEL with ifndef in KERNEL so the
target file wins. CGEMV/ZGEMV stay RISC-V scalar.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
continuous build / neoverse_build (push) Canceled after 0s
continuous build / neoverse_n1_build (push) Canceled after 0s
continuous build / neoverse_n1_omp_build (push) Canceled after 0s
continuous build / neoverse_n1_ilp64_build (push) Canceled after 0s
harmonyos / build (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=GENERIC, DYNAMIC_ARCH, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA264, LA264, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA464, LA464, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA64_GENERIC, LA64_GENERIC, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON2K1000, LOONGSON2K1000, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON3R5, LOONGSON3R5, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSONGENERIC, LOONGSONGENERIC, loongarch64-linux-gnu) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=GENERIC, DYNAMIC_ARCH) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA264, LA264) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA464, LA464) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LA64_GENERIC, LA64_GENERIC) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON2K1000, LOONGSON2K1000) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSON3R5, LOONGSON3R5) (push) Canceled after 0s
loongarch64 clang qemu test / TEST (NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=LOONGSONGENERIC, LOONGSONGENERIC) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=I6400, I6400, mipsisa64r6el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=I6500, I6500, mipsisa64r6el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=MIPS64_GENERIC, MIPS64_GENERIC, mips64el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=P6600, P6600, mipsisa64r6el-linux-gnuabi64) (push) Canceled after 0s
mips64 qemu test / TEST (NO_SHARED=1 TARGET=SICORTEX, SICORTEX, mips64el-linux-gnuabi64) (push) Canceled after 0s
riscv64 zvl256b qemu test / TEST (TARGET=RISCV64_GENERIC BINARY=64 ARCH=riscv64 DYNAMIC_ARCH=1, rv64,g=true,c=true,v=true,vext_spec=v1.0,vlen=256,elen=64, DYNAMIC_ARCH=1) (push) Canceled after 0s
riscv64 zvl256b qemu test / TEST (TARGET=RISCV64_ZVL128B BINARY=64 ARCH=riscv64, rv64,g=true,c=true,v=true,vext_spec=v1.0,vlen=128,elen=64, RISCV64_ZVL128B) (push) Canceled after 0s
riscv64 zvl256b qemu test / TEST (TARGET=RISCV64_ZVL256B BINARY=64 ARCH=riscv64 BUILD_BFLOAT16=1 BUILD_HFLOAT16=1, rv64,g=true,c=true,v=true,vext_spec=v1.0,vlen=256,elen=64,zfh=true,zvfh=true,zvfbfwma=true, RISCV64_ZVL256B) (push) Canceled after 0s
Vectorize the packed 2x2 CGEMM/ZGEMM inner loop with f32x4/f64x2
complex mul. Do not let kernel/wasm/KERNEL override the target
kernel the way 4x4 real GEMM already guards TRMM.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Generic scalar DTRMM was the remaining Level-3 gap (~15 GFLOPS vs
~23 DGEMM). Keep SIMD for double only; single-precision TRMM already
auto-vectorized and a shared S+D kernel slowed SGEMM.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
kernel/wasm/KERNEL is included after KERNEL.WASM128_GENERIC and
unconditionally pointed S/D AXPY at RISC-V scalar axpy.c, so the
x86_64 V_SIMD sources listed in the target file never ran.
Add kernel/wasm/axpy.c for both precisions. Unit-stride y += da * x
uses eight independent v128 lanes (32 floats / 16 doubles) with IEEE
mul+add; remainder is one vector then scalar. Skip relaxed madd:
AXPY is checked to machine epsilon, and putting it in the generic
V_SIMD path previously slowed Level 1. Non-unit stride stays scalar
(no WASM gather); inc==0 uses that path so y[0] += n * da * x[0].
Guard SAXPYKERNEL/DAXPYKERNEL with ifndef in KERNEL so the target
file wins. CAXPY/ZAXPY stay RISC-V scalar.
On n=512..8192 vs the RISC-V champion, SAXPY is about 1.4–3.0x and
DAXPY about 1.4–2.2x. test.sh is green.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Switch WASM128_GENERIC to unroll 4 with matching ncopy/tcopy and TRMM
4x4 so packing width stays consistent. Use relaxed SIMD madd in the
inner loop when compiled with -mrelaxed-simd.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Keep production STRSM/DTRSM free of always-on increments; enable the
counters only when counting invocations.
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
INSTALL/dlamch.c and INSTALL/slamch.c are f2c translations of the deprecated
dlamchf77.f and slamchf77.f, which determine the floating point format at run
time by probing in dlamc1/dlamc2 rather than reading it from the environment.
The current dlamch.f and slamch.f use the Fortran 90 inquiry intrinsics
(EPSILON, TINY, HUGE, DIGITS, MINEXPONENT, MAXEXPONENT, RADIX) instead, but f2c
cannot translate those, so the C LAPACK selected by NOFORTRAN=1 has been left
with the older probing implementation.
The probe is only correct if double intermediates are genuinely rounded to
double. That does not hold on x87. Building 32 bit for a target without SSE2,
so with -mfpmath=387, and with gcc 16, the intermediates stay in registers and
the probe measures the 80 bit register format: it reports emin/emax as
-16381/16384, and a mantissa width that follows the caller's x87 precision
control bits (64 at extended precision, 53 at double). Written back as doubles,
rmin underflows to 0 and rmax overflows to +Inf, so dlamch('S') and dlamch('U')
return 0 and dlamch('O') returns +Inf.
Everything that scales by those values is then wrong, mostly silently. The
first symptom to surface was a floating point exception rather than a wrong
answer: dsbevx computes
safmin = dlamch('S')
eps = dlamch('P')
smlnum = safmin / eps
bignum = 1 / smlnum
so a zero safe minimum makes smlnum zero and the next line divides by zero.
Callers that unmask the divide by zero exception get a hard failure there;
callers that do not get whatever the wrong scaling produces.
Replace the probe with the <float.h> constants, mirroring the values the
current dlamch.f and slamch.f return. This fixes two lesser problems at the
same time: rmach was left uninitialised when cmach matched nothing, where
dlamch.f returns zero; and the cached static results made both routines
unsafe to call concurrently on first use.
dlamc1-dlamc5 and slamc1-slamc5 are left in place. They become unreachable
from dlamch/slamch, but dlamc3 and slamc3 have callers of their own in dlaed3,
dlaed9, dlals0, dlasd3, dlasd8 and their complex equivalents, where they serve
as optimiser barriers.
Martin Kroeker