[jira] [Created] (HUDI-2960) create hudi table may cause memory leak in spark thrift server

[suheng.cloud (Jira)]

suheng.cloud created HUDI-2960:
----------------------------------

             Summary: create hudi table may cause memory leak in spark thrift 
server
                 Key: HUDI-2960
                 URL: https://issues.apache.org/jira/browse/HUDI-2960
             Project: Apache Hudi
          Issue Type: Bug
          Components: Spark Integration
    Affects Versions: 0.10.0
            Reporter: suheng.cloud


Hi, community

I currently try to use spark-hudi integration in spark-thrift-server, and after 
test create hudi table for a while, I found it would finally result in 
META-SPACE OOM(in my case, jvm option -XX:MaxMetaspaceSize=256m assigned).

After track the source, I found that every time a CreateHoodieTableCommand 
performed, `HiveClientUtils.newClientForMetadata` will be invoked, thus a 
IsolatedClientLoader will be created, in my scene, the OOM will occured after 
about 10 create statement executed.

Why not use 
`sessionState.catalog.externalCatalog.asInstanceOf[ExternalCatalogWithListener].unwrapped.asInstanceOf[HiveExternalCatalog].client
 ` instead ? Does it has anything side effect?

env: hudi master/spark-3.1.2/hive-2.3.6

Thanks.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)