Hyukjin Kwon created SPARK-61158:
------------------------------------
Summary: Delete the cached server model of a native LinearSVC or
LogisticRegression fit when its last owner is dropped, as pyspark.ml does
Key: SPARK-61158
URL: https://issues.apache.org/jira/browse/SPARK-61158
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A fitted native LinearSVC or LogisticRegression model is cached on the Spark
Connect server, and the cache entry was never released: the client never sent
MlCommand.Delete. Ten fits followed by del and gc.collect() left ten models in
MlCommand.GetCacheInfo for the session, where pyspark.ml leaves none, because
its RemoteModelRef deletes the cached model when the last model, copy or
transformed DataFrame that refers to it goes.
The fix shares the cached model between a model, its clones and every DataFrame
it transforms, and sends MlCommand.Delete on the fitting session from the drop
of the last owner, without waiting for the server.
Repro: fit LinearSVC(maxIter=2) ten times in one session, del each model,
gc.collect(), then count the entries of MlCommand.GetCacheInfo: ours 10,
official pyspark.ml 0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
