[
https://issues.apache.org/jira/browse/SPARK-46046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-46046.
----------------------------------
Resolution: Fixed
Resolving as Fixed — the requested capability exists on current master. The
ticket title "Isolated classloader per spark session" is provably implemented
in shipped code. sql/core/.../artifact/ArtifactManager.scala builds a
per-session ChildFirstURLClassLoader/URLClassLoader over the session's
jar/class URLs keyed on session.sessionUUID with sharedState.jarClassLoader as
fallback (buildClassLoader lines ~337-364), caches it per manager (classloader
lines ~325-331), and closes it on session teardown (cleanUpResources lines
~452-477). SQLConf.scala:5790 defines ARTIFACTS_SESSION_ISOLATION_ENABLED
(version 4.0.0, internal, default true), so per-session isolation is enabled by
default in a released version. A concrete implementing commit exists:
26330355836f [SPARK-
> Isolated classloader per spark session
> --------------------------------------
>
> Key: SPARK-46046
> URL: https://issues.apache.org/jira/browse/SPARK-46046
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 3.5.0
> Reporter: Faiz Halde
> Priority: Major
>
> Hello,
> We use spark 3.5.0 and were wondering if the following is achievable using
> spark-core
> Our use case involves spinning up a spark cluster wherein the driver
> application loads user jars on-the-fly ( the user jar is not the spark
> driver/application ) but merely a catalog of transformations. A single spark
> application can load multiple jars in its lifetime with potential of
> classpath conflict if care is not taken by the framework
> The driver needs to load the jar, add the jar to the executor & call a
> predefined class.method to trigger the transformation
> Each transformation runs in its own spark session inside the same spark
> application
> AFAIK, on the executor side, isolated classloader per session is only
> possible when using the spark-connect facilities. Is it possible to do this
> without using spark connect?
> Spark connect is the only facility that adds the jar into a sessionUUID
> directory of executor and when an executor runs tasks of a job from that
> session, it sets a ChildFirstClassLoader pointing to the sessionUUID directory
>
> Thank you
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
