zentol commented on PR #20003: URL: https://github.com/apache/flink/pull/20003#issuecomment-1161787438
> What's more, the same class will be loaded by different classloaders, there might be many class cast exceptions (X cannot be cast to X) even if the classes are in the same version. That depends on how the class-loading order is set up and how you actually use it. If you load everything parent-first within the added sub-tree this problem will not occur. I actually have to correct myself here; the addition of another jar should have no impact on the loading of a previously loaded class, because the URLClassLoader _should_ (because why not) access the jars in the order they were passed. Hence the most recently added jar is checked last. It should behave like parent-first within the sub-tree. If we start removing URLs however this very much changes. > Introducing a new classloader mechanism (each user jar in a new CL level) different from the existing one (all user jars are in the same CL level) will lead to inconsistent and unexpected behavior. > it is more friendly to fail fast before distributed execution. > it's important to keep the same classloader mechanism between local job compiling and distributed execution These are good points, but doesn't that mean that the _full_ user CL should be built eagerly in the CLI before any user-code is called? (and forbid us from breaking that pattern) Because that will be the actual behavior at runtime. I.E., we collect all the jars we need, then build the user CL, and only then start using the jars. Can you clarify on whether the jars are accessed in between `addUrl` calls? Would it be technically feasible to first determine all the required jars before creating the first user CL? Are there no plans to isolate the usage of a jar to a specific operator? > Besides, there are no differences in this case when creating a new classloader to wrap the existing classloader. It is indeed technically similar; but I already listed the benefits compared to the mutable CL in my previous comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
