lriggs opened a new issue, #51663: URL: https://github.com/apache/arrow/issues/51663
### Describe the bug, including details regarding any error messages, version, and platform. ### Background I originally identified this issue in GH-601 and fixed it on the arrow-java side with a coarse grained lock on the Projector and Filter Make methods. That fixed the issue but adds some undesirable overhead in certain multi threaded workflows. This issue captures the deeper problem that is occurring in the C++ side of Gandiva and was not touched by my previous fix though it is prevented from happening. I have a proposed solution for this and believe it is a better overall solution. ### Problem Projector::Make() and Filter::Make() read the shared expression cache once to decide the is_cached status and then call SetLLVMObjectCache(). That performs its own second, unsynchronized read of the same key before pre-loading a cached object into the LLJIT. Those two reads could disagree. If another thread compiling the identical (schema, expressions, selection vector mode, configuration) tuple inserted between them, the first thread would: take the is_cached == false path, generating expr_0_0 into its IR module, and also see a hit on the second read and addObjectFile() a cached object that defines expr_0_0 as well. Both then call JITDylib::define for the same symbol in the same JITDylib, and ORC's duplicate-symbol detection fires: CodeGenError in Gandiva: Failed to add IR module to LLJIT: In gdv_module_..., duplicate definition of symbol 'expr_0_0' ### Component(s) C++, Gandiva -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
