[
https://issues.apache.org/jira/browse/SPARK-57860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mao Li updated SPARK-57860:
---------------------------
Description:
Introduce a `HasIntermediateStorageLevel` shared param (in both Scala
`sharedParams`
and PySpark `shared.py`) following the pattern ALS already uses, and apply it
to KMeans
as the reference implementation for SPARK-47103.
This lets users control the StorageLevel of the *intermediate* datasets that
MLlib
persists internally during training (e.g. blockified/standardized RDDs). These
are not
the user's input DataFrame, so users currently have no way to change their
storage level
from the hardcoded MEMORY_AND_DISK. Making it configurable (e.g. DISK_ONLY)
improves
resilience to executor loss, since the External Shuffle Service (SPARK-27677)
can serve
disk-persisted cached blocks.
Design follows the reviewer suggestion on the earlier PR #45182 (per-estimator
param via
a shared `HasIntermediateStorageLevel` trait, rather than a global SQL config).
Remaining estimators are tracked in the sibling sub-tasks.
was:
Add a HasIntermediateStorageLevel trait to sharedParams.scala, following the
same pattern
as other Has* param traits. Apply it to the following estimators that currently
hardcode
StorageLevel.MEMORY_AND_DISK for intermediate dataset persistence:
Classification: LogisticRegression, LinearSVC, FMClassifier, OneVsRest
Clustering: KMeans, GaussianMixture, LDA
Regression: LinearRegression, AFTSurvivalRegression, FMRegressor,
IsotonicRegression
Other: FPGrowth
Also update the corresponding PySpark classes.
Part of SPARK-47103.
> Add HasIntermediateStorageLevel shared param and apply to KMeans
> ----------------------------------------------------------------
>
> Key: SPARK-57860
> URL: https://issues.apache.org/jira/browse/SPARK-57860
> Project: Spark
> Issue Type: Sub-task
> Components: MLlib
> Affects Versions: 4.0.0
> Reporter: Mao Li
> Priority: Major
>
> Introduce a `HasIntermediateStorageLevel` shared param (in both Scala
> `sharedParams`
> and PySpark `shared.py`) following the pattern ALS already uses, and apply it
> to KMeans
> as the reference implementation for SPARK-47103.
> This lets users control the StorageLevel of the *intermediate* datasets that
> MLlib
> persists internally during training (e.g. blockified/standardized RDDs).
> These are not
> the user's input DataFrame, so users currently have no way to change their
> storage level
> from the hardcoded MEMORY_AND_DISK. Making it configurable (e.g. DISK_ONLY)
> improves
> resilience to executor loss, since the External Shuffle Service (SPARK-27677)
> can serve
> disk-persisted cached blocks.
> Design follows the reviewer suggestion on the earlier PR #45182
> (per-estimator param via
> a shared `HasIntermediateStorageLevel` trait, rather than a global SQL
> config).
> Remaining estimators are tracked in the sibling sub-tasks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
