Ruifeng Zheng created SPARK-59618:
-------------------------------------
Summary: Concurrent F2J LAPACK initialization can hang
MultivariateGaussian
Key: SPARK-59618
URL: https://issues.apache.org/jira/browse/SPARK-59618
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 4.2.0
Reporter: Ruifeng Zheng
F2J LAPACK caches machine parameters in unsynchronized mutable static fields.
During concurrent first use, Dlamc2 publishes first=false before lrmin and
lrmax are initialized. A racing Dsyev call can consume zero machine limits and
enter a non-terminating Dlascl scaling loop.
This can occur when MultivariateGaussian instances evaluate their lazy
covariance constants concurrently. Repeated thread dumps show one task
remaining RUNNABLE in Dlascl -> Dsyev -> F2jLAPACK while its peer finishes
normally.
The proposed fix warms up the selected LAPACK backend once per executor JVM
before concurrent eigSym calls. It does not change library dependencies or
serialize real eigendecompositions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
