linrrzqqq opened a new pull request, #67511: URL: https://github.com/apache/doris/pull/67511
### What problem does this PR solve? Fixes Python UDF module isolation and cache cleanup issues when different functions contain modules with the same name. Python UDF/UDAF/UDTF functions run in shared Python server processes and therefore share process-wide import state such as `sys.modules` and `sys.path`. When two function packages contain dependencies with the same module name, one function may reuse the module loaded from another package. This is especially visible with: - Imports executed lazily inside function bodies. - Imports executed from user-created threads. - UDAF state serialization and deserialization. - Concurrent module loading. - Function cache cleanup while a module is being imported. Cache cleanup failures could also be ignored by the BE, allowing module files to be deleted while Python processes still referenced them. ### What is changed? - Isolate imported modules by UDF package location using a per-location module context. - Restore the corresponding module context while executing module UDF/UDAF/UDTF code. - Intercept `__import__` and `importlib.import_module` so delayed imports use the correct function package. - Propagate module context to user-created Python threads. - Coordinate normal imports and temporary `sys.path`/`sys.modules` modifications with a reader-writer locking scheme. - Avoid the global write lock and full `sys.modules` scan for cached imports. - Resolve UDAF state classes directly from the corresponding module context during deserialization. - Use the new module import lock when clearing Python module caches. - Propagate Python cache cleanup failures to `UserFunctionCache`. - Preserve cache entries and module files when cleanup fails, allowing the cleanup operation to be retried safely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
