Akshay Thorat created SPARK-59516:
-------------------------------------
Summary: Infinity norm of an all-zero SparseVector raises
ValueError with NumPy before 2.3
Key: SPARK-59516
URL: https://issues.apache.org/jira/browse/SPARK-59516
Project: Spark
Issue Type: Bug
Components: MLlib, PySpark
Affects Versions: 5.0.0
Reporter: Akshay Thorat
h2. Problem
With NumPy versions before 2.3, SparseVector.norm passes an empty values array
to numpy.linalg.norm, which raises on the maximum reduction. Dense vectors and
sparse vectors storing an explicit zero return 0.0 for the same logical vector.
Both pyspark.ml.linalg and pyspark.mllib.linalg are affected.
h2. Reproduction
Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python 3.10.11
and NumPy 2.2.6. No SparkSession is required.
{code:python}
from pyspark.ml.linalg import SparseVector
SparseVector(3, [], []).norm(float("inf"))
# ValueError: zero-size array to reduction operation maximum which has no
identity
{code}
h2. Expected behavior
The infinity norm should be 0.0, as it is for DenseVector([0.0, 0.0, 0.0]).
Supported NumPy versions should give the same result for positive-dimensional
zero vectors.
h2. Proposed fix and verification
Return zero for infinity norm when a positive-dimensional sparse vector has no
stored entries, in both linalg APIs. Preserve NumPy behavior for
zero-dimensional vectors and invalid norm orders. Regression tests cover both
APIs and implicit/explicit zeros.
The regression fails with NumPy 2.2.6 before the fix and passes afterward.
NumPy 2.3 independently changed the empty-array norm to return zero; the Spark
fix also supports older NumPy releases. See
https://numpy.org/doc/2.3/release/2.3.0-notes.html#changes
Pull request: https://github.com/apache/spark/pull/58788
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
