Per the other e-mail thread I think this is a good idea and will unblock a number of use cases (for example, a C FFI with no knowledge of pyarrow could consume the record batches produced by a Dataset). I'm not sure it's necessary to formalize it / freeze the ABI right away until we get some more people kicking the tires, but after a short period of evaluation we should try to formalize it with a vote to prevent possible ABI incompatibilities between different versions of libraries.
On the docs -- I see the data interface and the stream/iterator interface as going hand-in-hand, so it may make sense to merge the docs into a common "Arrow C interface" doc On Tue, Sep 1, 2020 at 10:38 AM Antoine Pitrou <anto...@python.org> wrote: > > > Also, for the record, I've temporarily uploaded a rendered version of > the doc here: > https://pitrou.net/arrowdevdoc/format/CStreamInterface.html > > Regards > > Antoine. > > > Le 01/09/2020 à 17:25, Antoine Pitrou a écrit : > > > > Hello, > > > > We are proposing an experimental C stream interface which builds on the > > existing C data interface and is defined by a single C struct: > > > > struct ArrowArrayStream { > > // Callbacks providing stream functionality > > int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out); > > int (*get_next)(struct ArrowArrayStream*, struct ArrowArray* out); > > const char* (*get_last_error)(struct ArrowArrayStream*); > > > > // Release callback > > void (*release)(struct ArrowArrayStream*); > > > > // Opaque producer-specific data > > void* private_data; > > }; > > > > A PR is up and ready for review (including docs, a C++ import/export > > implementation, and a simple Python binding): > > https://github.com/apache/arrow/pull/8052 > > > > Here is the current documentation as of this writing (otherwise you can > > also view the file from the PR diff view): > > https://github.com/apache/arrow/blob/04c25d8cb8eb4c78066e651575aa8bfeef3ea542/docs/source/format/CStreamInterface.rst > > > > Best regards > > > > Antoine. > >
