HUANG GUAN LUN created SPARK-60151:
--------------------------------------
Summary: CrossValidatorModel.load() returns the last fold's
sub-models for every fold
Key: SPARK-60151
URL: https://issues.apache.org/jira/browse/SPARK-60151
Project: Spark
Issue Type: Bug
Components: ML, PySpark
Affects Versions: 4.0.4, 4.1.3, 4.2.0, 3.5.8, 5.0.0
Reporter: HUANG GUAN LUN
When a CrossValidatorModel is saved with sub-models and loaded back in PySpark,
every fold's entry of \{{subModels}} refers to the same list, which holds the
last fold's models.
The cause is in \{{CrossValidatorModelReader.load}}
(python/pyspark/ml/tuning.py):
{code:python}
subModels = [[None] * len(estimatorParamMaps)] * numFolds
{code}
{\{[x] * n}} repeats a reference to the same inner list, so each fold's
assignment overwrites the previous one. \{{CrossValidator._fit}},
\{{CrossValidatorModel.copy}} and \{{_from_java}} all build an independent list
per fold, as does the Scala reader (\{{Array.fill(numFolds)(...)}}).
The files on disk are correct. Only the PySpark reader is affected.
\{{bestModel}} and \{{avgMetrics}} are not affected. However, saving a loaded
model again writes the last fold's models into every fold directory, so the
original sub-models are lost.
*Reproduce*
{code:python}
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.linalg import Vectors
from pyspark.ml.tuning import CrossValidator, CrossValidatorModel,
ParamGridBuilder
df = spark.createDataFrame(
[(Vectors.dense([float(i)]), float(i % 2)) for i in range(30)],
["features", "label"])
lr = LogisticRegression(maxIter=2)
grid = ParamGridBuilder().addGrid(lr.regParam, [0.0, 0.1]).build()
cv = CrossValidator(estimator=lr, estimatorParamMaps=grid,
evaluator=BinaryClassificationEvaluator(),
numFolds=3, collectSubModels=True, seed=1)
model = cv.fit(df)
model.write().save("/tmp/cv")
loaded = CrossValidatorModel.load("/tmp/cv")
print([m.coefficients for m in model.subModels[0]]) # fold 0
print([m.coefficients for m in loaded.subModels[0]]) # same as fold 2
print(loaded.subModels[0] is loaded.subModels[1]) # True
{code}
First coefficient of each sub-model:
|| ||fold0||fold1||fold2||
|after fit|[0.0705, 0.0439]|[0.0067, 0.0047]|[-0.0155, -0.0108]|
|on disk after save|[0.0705, 0.0439]|[0.0067, 0.0047]|[-0.0155, -0.0108]|
|after load|[-0.0155, -0.0108]|[-0.0155, -0.0108]|[-0.0155, -0.0108]|
The existing test in python/pyspark/ml/tests/test_tuning.py does not catch
this. It compares \{{model.subModels[i][j].uid}} with itself, and after loading
it only checks the list lengths. All sub-models share the estimator's uid, so a
uid comparison would not detect it either.
The line was introduced by SPARK-33520.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
