yuqi1129 commented on PR #8252: URL: https://github.com/apache/gravitino/pull/8252#issuecomment-3227813810
> > > > > My biggest concern is the maintainability of the clear resource logic. It seems that we are doing some complex work that should belong to the JVM level's GC, and this workload will get out of control as the project develops. > > > > > I don't know if we have other alternative solutions to solve the OOM problem, but the current solution seems not to be the optimal one in the long run. WDYT? @jerryshao @FANNG1 > > > > > > > > > > > > I agree. My thought is that though we may have lots of catalogs, the number of providers is limited, which means the number of classloaders are limited, we don't have to create a classloader for each catalog. We can reuse the classloader for the same type of catalogs, and with this we can reduce the class-related OOM issues. > > > > Also, many engines like Spark/Trino that support multiple connectors will face the same problem, especially for Trino, which can also support dynamic connector register/unregister. How do they handle this problem? We'd better have a long-term solution. > > > > For now, I think we can have a short solution, but we should clearly separate this from the main logic, so we can remove it easily. Also, we should add the comments as possible as we can. > > > > > > > > > I see. > > > I will try to introduce the class loader reuse mechanism in the next version, and possibly there will be static variable sharing problems like `UGI` exists, so conducting full tests is necessary before this changes. > > > > > > In Trino, there will be only one classloader for each type of catalog. > > We can also have the same solution as I mentioned above. The problem we face is the same as Trino (they also have the same UGI issue). Please investigate it and have a long-term solution. I will. Should I move this PR or focus on a new solution? I'm afraid we won't be able to finish it within the time frame of 1.0.0, as large number of tests are needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
