kylebarron commented on issue #1008: URL: https://github.com/apache/datafusion-python/issues/1008#issuecomment-2622430293
I agree! This is why I created [obstore](https://github.com/developmentseed/obstore). It's a fast Python binding for `object_store`. > However, it's useful to be able to define just one ObjectStore (like from a configuration) and use it both for datafusion and for related object-store operations like artifact cleanup. This is the goal of [`pyo3_object_store`](https://github.com/developmentseed/obstore/tree/main/pyo3-object_store), so that we can define configuration and builders around `object_store` _once_ and then reuse it across many different Rust-Python libraries that internally use `object_store`. I.e. `obstore` is for Python end users who want to use `object_store` from Python, while `pyo3_object_store` is for other Rust developers creating their own Python packages who want to use `object_store` from Rust. So far, I've put more effort into publishing `obstore` than `pyo3_object_store`, but I'd like to polish up `pyo3_object_store` (especially after the next `object_store` release) and I'd be happy to explore using it inside `datafusion-python` if there's interest. Unfortunately, you can't currently use _one_ store class across multiple Python libraries because `object_store` is not FFI stable. So you need to use the class _exported from `obstore`_ whenever you use obstore methods, and you'd need to use the class _exported from `datafusion.python`_ whenever you use datafusion methods (even though each class would take the same builder params). Or, we could try to solve the object_store FFI problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
