Ruifeng Zheng created SPARK-61017:
-------------------------------------
Summary: Support configurable intermediate storage levels in
iterative classifiers
Key: SPARK-61017
URL: https://issues.apache.org/jira/browse/SPARK-61017
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
LogisticRegression, LinearSVC, FMClassifier, and MultilayerPerceptronClassifier
currently persist intermediate training data at a hard-coded MEMORY_AND_DISK
storage level. Caching the input DataFrame does not configure these internally
created datasets.
Expose the existing intermediateStorageLevel shared parameter on these four
classifiers in Scala and PySpark. Preserve MEMORY_AND_DISK as the default, use
the existing validation that rejects NONE and invalid storage levels, and pass
the configured level to each algorithm's internal persistence call. For
MultilayerPerceptronClassifier, forward it to FeedForwardTrainer.
Cover parameter defaults, validation, copying and persistence, actual storage
used during training, and PySpark parameter transfer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
