[
https://issues.apache.org/jira/browse/SPARK-58181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58181:
-----------------------------------
Labels: pull-request-available (was: )
> Improve connect ML model size estimation
> ----------------------------------------
>
> Key: SPARK-58181
> URL: https://issues.apache.org/jira/browse/SPARK-58181
> Project: Spark
> Issue Type: Improvement
> Components: Connect, ML
> Affects Versions: 4.3.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> ### Background
> Spark Connect uses Model.estimatedSize to enforce per-model and per-session
> limits in its server-side ML model cache.
> Model size estimates currently follow two inconsistent paths:
> - Most models use SizeEstimator.estimate(self), which can overcount shared
> state reachable through Model.parent, including SparkSession state
> (SPARK-57521).
> - Some models override estimatedSize with a model-specific calculation.
> Several of these include parameter metadata through
> Params.estimateMatadataSize, while tree-based models historically counted
> only learned
> tree nodes.
> This makes cache accounting inconsistent across ML model implementations.
> ### Goal
> Provide a consistent, model-owned definition of estimated size:
> estimated model size =
> parameter metadata
> + learned model state
> + explicitly retained distributed state, where applicable
> The estimate must exclude shared or incidental runtime state, especially
> the parent estimator and any Spark session/context reachable through it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
