Hi Jorge,
Le 24/09/2020 à 17:04, Jorge Cardoso Leitão a écrit : > Thank you both for the prompt response. > > Just to check I understand, Antoine, your recommendation is: > > 1. Rust implementation should expose the ABI > 2. Rust implementation should be able to consume (and use) the ABI (without > owning it, but still call the `release`) Indeed. Basically, when Rust is done with an exported array or schema, call its release callback. Note that exposing and consuming are entirely distinct. You can start implementing one without the other. > 1. in pyarrow, I was only able to find Array.from_buffers and from_pandas. > Is the ABI implemented but not documented? It is implemented in C++ and also exposed (but undocumented) in Python. The Python methods are called `Array._import_from_c`, `Array._export_to_c`, likewise for `Schema` and `RecordBatch`. You can find the source for Array methods here: https://github.com/apache/arrow/blob/master/python/pyarrow/array.pxi#L1201 There are ad-hoc tests for Python here: https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_cffi.py The core C++ implementation is exported and documented here: https://github.com/apache/arrow/blob/master/cpp/src/arrow/c/bridge.h > 3. do we have a place in the project where we test these things (maybe > integration?). IMO we need to compile both projects and have both > communicate in the same process. I have been doing this via Python (pypi > pyarrow and pyo3 for rust), but for this both need to be compiled from > master. It should work with PyArrow 1.0. > 4. Is there a "source of truth" that we can use to generate and consume > these in-memory structs, e.g. to perform round-trips. Nothing really as such, but the C++ tests are quite comprehensive. Especially, you can take a look at the schema and array import tests: https://github.com/apache/arrow/blob/master/cpp/src/arrow/c/bridge_test.cc#L1157 Feel free to ask any questions, though! Best regards Antoine.
