Files
OpenBLAS/lapack-netlib/LAPACKE/utils
Yue ChangandClaude Fable 5 ede136ee4e Fix 32-bit integer overflow in LAPACKE nancheck index arithmetic
The optional NaN checks performed by LAPACKE front-ends computed array
offsets and lengths in 32-bit lapack_int arithmetic, which overflows for
large matrices (e.g. LAPACKE_dpotrf with n = lda >= 46341 overflows
j*lda in LAPACKE_dtr_nancheck, leading to out-of-bounds reads and
crashes). Several nancheck helpers (ge, gb, tp, tf) already cast to
size_t; this applies the same treatment to the ones that were missed:

- ?tr_nancheck: cast a[i+j*lda] index to size_t (also covers po/sy/he
  checks, i.e. the POTRF/Cholesky path)
- ?_nancheck (vector): iterate by element count instead of n*inc, with
  size_t indexing
- ?sp/?pp/?pf/?hp/?tp/?tf_nancheck: compute n*(n+1)/2 length in size_t
- ?tz_nancheck: widen the part offsets to int64_t (keeping the -1
  sentinel) and compute them with a widening cast

Verified with UBSan: LAPACKE_str_nancheck at n = lda = 47000 previously
reported signed integer overflow and crashed with SEGV at -O0; it now
passes and still detects the NaN. Old and new code produce identical
results across 1907 enumerated small-size cases (all layouts, uplo,
diag, shapes and NaN positions for tr/tz/vector/packed variants).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 23:42:28 -04:00
..