Commit Graph
411 Commits
Author SHA1 Message Date
zhir 46aa158dcc AArch64: NEON omatcopy CT/RT kernels 2026-06-20 22:54:04 +03:00
Martin Kroeker 0f9f6e4be5 Merge pull request #5710 from martin-frbg/issue5708
Work around miscompilation of the ARM64 non-SVE DDOT kernel
2026-03-27 22:09:08 +01:00
yuanjia e6eba9fa21 Add optimized FP16 shgemm for for NEOVERSEN2 target 2026-03-27 17:55:06 +08:00
Martin Kroeker e3ce4623c2 Use volatile attribute for SDOT only, to avoid creating new miscompilations 2026-03-24 23:08:02 +01:00
Martin Kroeker 3f6e928d34 Declare result as volatile to keep compilers from optimizing it out 2026-03-20 11:32:23 +01:00
Fadi Arafeh f30202b705 Accelerate SVE128 SBGEMM/BGEMM
This accelerates SBGEMM/BGEMM by extending the existing 8x4 kernel to 8x8 (unrolling N by 8)

Not sure if it's a good idea to delete the previous 8x4 kernel?

Here are the speedups on single core Neoverse-V2 (SVE128) compared to prev state:

Per-shape speedup
  M=N=K=64: SBGEMM 1.164x (16.42%), BGEMM 1.133x (13.30%)
  M=N=K=128: SBGEMM 1.220x (22.02%), BGEMM 1.186x (18.56%)
  M=N=K=256: SBGEMM 1.241x (24.08%), BGEMM 1.235x (23.54%)
  M=N=K=512: SBGEMM 1.240x (23.95%), BGEMM 1.227x (22.75%)
  M=N=K=1024: SBGEMM 1.251x (25.11%), BGEMM 1.232x (23.23%)
  M=N=K=2048: SBGEMM 1.235x (23.47%), BGEMM 1.246x (24.64%)

Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
2026-03-05 13:50:07 +00:00
Martin Kroeker 69d92490c1 move inclusion of sme_abi header into the conditional section 2026-01-29 22:24:00 +01:00
Martin Kroeker 861b3db733 Reuse ?SUM kernels from ThunderX2T99 2026-01-20 15:42:09 +01:00
Martin Kroeker 71261a7b3f Trivially derive optimized S/DSUM for existing SASUM/DASUM kernels 2026-01-20 15:38:50 +01:00
Jameson NashandClaude Opus 4.5 a18a4ee08a arm64: fix clang ICE on Windows for zdot_thunderx2t99.c
Guard .align directive to avoid internal compiler error on
AArch64 Windows with clang.

See: https://github.com/llvm/llvm-project/issues/149547
See: #5076

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 15:36:14 +00:00
Martin Kroeker 6de062cfc2 Merge branch 'OpenMathLib:develop' into issue5414 2026-01-11 17:45:11 +01:00
Martin Kroeker d1de282a4e Improve the precision of S/CNRM2 by summing in double precision 2026-01-11 13:04:00 +01:00
Martin Kroeker a9a6edaf17 Adapt for DYNAMIC_ARCH with multiple ...preprocess symbols 2026-01-09 15:29:36 +01:00
Martin Kroeker 2d46f1ec65 Merge branch 'develop' into issue5414 2026-01-09 15:04:06 +01:00
Martin Kroeker c040d5ed86 Merge pull request #5591 from quic/topic/ssyr2k_direct_sme1
Support for SME1 based ssyr2k_direct kernel for cblas_ssyr2k level 3 API
2026-01-08 15:47:38 +01:00
Zhiqing xie 6939a43c3b Support for SME1 based ssyr2k_direct kernel for cblas_ssyr2k level 3 API 2026-01-08 11:09:04 +08:00
Martin Kroeker d39b77748f Make .align conditional on not being on WoA and strip CRLF endings 2025-12-24 20:00:45 +01:00
Martin Kroeker ac2c66321d remove special handling of C/ZDOT for LLVM on WoA 2025-12-19 17:04:21 +01:00
Martin Kroeker cfa28bcf71 Support compilation with LLVM for Windows on Arm 2025-12-19 17:00:47 +01:00
Martin Kroeker e85efb8d86 remove za from clobber lists 2025-12-03 22:40:02 +01:00
Martin Kroeker a683287006 rework for dynamic_arch 2025-11-24 22:24:06 +01:00
Martin Kroeker b185c9a4ce small fixes for separating sme and dummy parts 2025-11-24 22:22:14 +01:00
Martin Kroeker 8c0b13c41c Merge branch 'OpenMathLib:develop' into issue5414 2025-11-23 23:12:49 +01:00
Martin Kroeker ea85b6696f Merge branch 'OpenMathLib:develop' into issue5414 2025-11-23 10:14:07 +01:00
Abhishek Kumar a14caf464f add tt for a64fx dot
Signed-off-by: Abhishek Kumar <abhishek.r.kumar@fujitsu.com>
2025-11-20 12:14:17 +05:30
Martin Kroeker 17f2e94260 Merge pull request #5539 from FRosner/arm64-dot-kernel-refactoring
Refactoring: ARM64 dot Kernel: don't call num_cpu_avail twice
2025-11-18 23:25:51 +01:00
Frank Rosner 762ed66c72 Refactoring: ARM64 dot Kernel: don't call num_cpu_avail twice 2025-11-18 15:34:51 +01:00
mayeut 39d5e44723 fix: dot_kernel_sve "n" usage & clobber list 2025-11-17 21:53:51 +01:00
Martin Kroeker f2d010de12 Merge pull request #5512 from quic/topic/ssyrk_direct_sme1
Support for SME1 based ssyrk_direct kernel for cblas_ssyrk level 3 API
2025-11-06 14:06:43 -08:00
Martin Kroeker 585e6d0680 Merge pull request #5515 from iha-taisei/feature/ger_unroll
Improve single-thread performance of [SD]GER on A64FX and Neoverse V1
2025-10-24 08:17:06 -07:00
Iha, Taisei cb66aca707 Improve single-thread performance of [SD]GER on A64FX and Neoverse V1 2025-10-22 19:56:14 +09:00
Yichao Yu 3d19d3b60a Make dummy function have the same linkage as the real one 2025-10-20 12:42:39 -04:00
changjua 43d38d336f Support for SME1 based ssyrk_direct kernel for cblas_ssyrk level 3 API 2025-10-20 11:35:20 +08:00
Martin Kroeker 9bfc3612f9 Merge branch 'OpenMathLib:develop' into issue5414 2025-10-12 09:18:06 -07:00
Yichao Yu b94e9b92ad Fix compilation on ARM
Define a dummy function if SME is not supported, following what sgemm does
2025-10-11 20:28:59 -04:00
Martin Kroeker e40714cabd Merge pull request #5450 from quic/topic/strmm_direct_sme1
Support for SME1 based strmm_direct kernel for cblas_strmm level 3 API
2025-10-11 15:20:19 -07:00
changjua 644ea07ef9 Support for SME1 based strmm_direct kernel for cblas_strmm level 3 API 2025-10-10 10:48:27 +08:00
Martin Kroeker 47a66aef0f Update limits based on benchmarking the SME code on Apple M4 2025-10-08 14:36:17 +02:00
Martin Kroeker 20f5ed1a94 Merge branch 'OpenMathLib:develop' into issue5414 2025-10-08 05:27:28 -07:00
Chris Sidebottom 578e7dae85 Fix bf16->f32 conversion for NEOVERSEV1 and NEOVERSEN2 targets
This fixes an issue originally introduced with the BGEMM kernel.

I've updated the tests to run with `beta=1.0` so as to test loading and
updating from C.

Alongside this, the tests now return sensible return values to reduce
the risk of them being ignored.

Also fixed a bug in `generic/gemv_t.c` resulting in weird outputs for
`bgemv`.
2025-10-06 18:05:58 +00:00
Martin Kroeker c889558317 Rework for DYNAMIC_ARCH use and use of SGEMM functions by SSYMM 2025-10-02 07:39:24 -07:00
Martin Kroeker fc516af155 Merge branch 'develop' into issue5414 2025-10-01 14:12:59 -07:00
Rajendra Prasad Matcha 19268471cc Support for SME1 based ssymm_direct kernel for cblas_ssymm level 3 API 2025-09-30 15:05:33 +05:30
Martin Kroeker 2b5d8c789d remove debugging printout 2025-08-24 13:50:08 -07:00
Martin Kroeker b4fc09e9e1 Add registers d8 to d15 to clobber lists as the code does not expressly save them 2025-08-23 14:39:27 -07:00
Martin Kroeker 8e50b8d525 Add d8 to d15 to clobber lists as the code does not expressly save them 2025-08-23 14:36:49 -07:00
h-motoki 855945befb Implementing SVE in [SD]AXPY Kernels for A64FX and Graviton3E 2025-08-21 20:56:58 +09:00
Martin Kroeker edaa73fd24 Hide the local 2VLx2VL symbol as static is insufficient for this with gcc 2025-08-20 06:33:28 -07:00
Martin Kroeker 501728a354 adjust register 20 accesses to 21 after moving x18 2025-08-20 06:24:38 -07:00
Martin Kroeker 05dbb54362 Delete misplaced file 2025-08-19 05:12:09 -07:00