[
https://issues.apache.org/jira/browse/SPARK-57860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57860:
-----------------------------------
Labels: pull-request-available (was: )
> Add HasIntermediateStorageLevel shared param and apply to KMeans
> ----------------------------------------------------------------
>
> Key: SPARK-57860
> URL: https://issues.apache.org/jira/browse/SPARK-57860
> Project: Spark
> Issue Type: Sub-task
> Components: MLlib
> Affects Versions: 4.0.0
> Reporter: Mao Li
> Priority: Major
> Labels: pull-request-available
>
> Introduce a `HasIntermediateStorageLevel` shared param (in both Scala
> `sharedParams`
> and PySpark `shared.py`) following the pattern ALS already uses, and apply it
> to KMeans
> as the reference implementation for SPARK-47103.
> This lets users control the StorageLevel of the *intermediate* datasets that
> MLlib
> persists internally during training (e.g. blockified/standardized RDDs).
> These are not
> the user's input DataFrame, so users currently have no way to change their
> storage level
> from the hardcoded MEMORY_AND_DISK. Making it configurable (e.g. DISK_ONLY)
> improves
> resilience to executor loss, since the External Shuffle Service (SPARK-27677)
> can serve
> disk-persisted cached blocks.
> Design follows the reviewer suggestion on the earlier PR #45182
> (per-estimator param via
> a shared `HasIntermediateStorageLevel` trait, rather than a global SQL
> config).
> Remaining estimators are tracked in the sibling sub-tasks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
