timsaucer commented on PR #18672: URL: https://github.com/apache/datafusion/pull/18672#issuecomment-3568686771
> Right now we are relying on memory address of a static to determine if we are in a foreign/local context, and although that raw memory check makes it now, I am not sure if having some sort of global registry would help future proofing the whole improvement. The global registry could still hold the raw memory pointer but also some UUID generated when a new library gets registered. Do you think that would help? This is a good question. I did play around with doing some kind of registry, but I don't think it necessarily fits the model we currently have of how these structures are created and used. Take the `ExecutionPlan` for instance. Those typically only exist during their execution and then get dropped when the strea is complete. If we had registered those, we'd probably have an `Arc<dyn ExecutionPlan>` floating around in the registry. That would lead to library memory bloat. There are other things one could use such a registry for. I haven't really thought of any so I'm not sure how your suggestion would make the appoach more future proof. Do you have a use case in mind already? If there's value add in doing so, we can rework this PR but it would likely be more complex because I suspect each one of these structs would need a custom registry. I think the advantage of what we have here is that it is very simple. The address for the static variable is guaranteed to be unique from one library or application to another. One thing this PR doesn't do and it would be *awesome* if I could figure out a way is to figure out if we have libraries that can access each other's internal memory structures. That's near impossible because Rust doesn't have a stable ABI so the same exact code built with two different versions of `rustc` *could* end up with different struct layouts in memory. If there was some way that we could compile a unique ID of the compiler with all it's options and check against that, then we could open up a way for libraries to jump into each other without all of this FFI mess. There is no way that I know of to create such a thing, however. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
