wuchong commented on PR #20003: URL: https://github.com/apache/flink/pull/20003#issuecomment-1163056508
> Do you think this will likely stay the case? Are these jars small enough that users won't worry about that? Yes. Spark/Databricks also doesn't support remove jar[1]. If a user wants to drop a jar, he can simply restart a session. Usually, the jars are small because only contain UDFs and limited dependencies. > I assume the components that make use of the user CL all created when the session is started? Yes. They are created when the session is started and released when the session is closed. > Is there any state kept across these statement that has a hard requirement for having access to the user CL? For example, when I submit CREATE TEMPORARY FUNCTION ..., is the function eagerly loaded and put into some data-structure (a catalog I guess?)? Or do we store just some description and load it later when required? Yes. Currentlly, for example, `CREATE CATALOG` and `LOAD MODULE` will eagerly load the `Catalog` and `Module` instances and bookkeep them in `CatalogManager` and `ModuleManager`. I think you raised a good idea that lazily creating the classloader when needed (e.g. validation, execution). Technical-wise, this is possible by holding everything as descriptions. However, this may slow down the SQL compiling speed because the classloaders need to be reloaded every time, and catalogs/modules/... need to be re-discovered by SPI every time. Considering this needs a big refactor on the current architecture, the parsing mechanism, and there maybe some corner problems I don't notice so far. Therefore, I would prefer to go with `MutableURLClassloader` (mark as `@Internal`) approach first. We can revisit it sometime later whether it has any problems or can support all use cases, to see whether we need to refactor the architecture. What do you think about this? [1]: https://docs.gcp.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-aux-resource-mgmt-add-jar.html#related-statements -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
