[
https://issues.apache.org/jira/browse/SPARK-51571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh updated SPARK-51571:
---------------------------------------------
Description:
When running Spark SQL, a new Hive session state creates a session resource
directory, but these directories are not cleaned up upon exit if the session
state isn't started.
{code:java}
private def closeHiveSessionStateIfStarted(state: SessionState): Unit = {
if (ReflectionUtils.getSuperField(state, "isStarted").asInstanceOf[Boolean]) {
state.close()
}
} {code}
This can lead to an accumulation of unused directories, consuming disk space
and potentially affecting performance over time. The expected behaviour is to
ensure these directories are removed when the Spark SQL session ends.
> Hive SessionState Resource directories persists if SessionState is not started
> ------------------------------------------------------------------------------
>
> Key: SPARK-51571
> URL: https://issues.apache.org/jira/browse/SPARK-51571
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.2
> Reporter: Indhumathi Muthumurugesh
> Priority: Major
>
> When running Spark SQL, a new Hive session state creates a session resource
> directory, but these directories are not cleaned up upon exit if the session
> state isn't started.
> {code:java}
> private def closeHiveSessionStateIfStarted(state: SessionState): Unit = {
> if (ReflectionUtils.getSuperField(state,
> "isStarted").asInstanceOf[Boolean]) {
> state.close()
> }
> } {code}
>
> This can lead to an accumulation of unused directories, consuming disk space
> and potentially affecting performance over time. The expected behaviour is to
> ensure these directories are removed when the Spark SQL session ends.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
