If I remember correctly the reason cuDF interacts with the Cython code for IPC stuff is that in the past the existing IPC machinery in Arrow didn't work correctly with GPU memory. If that is fixed I think there's a case to remove this code entirely from cuDF and instruct users to use the higher level PyArrow IPC APIs instead.
-Keith On Wed, Aug 25, 2021 at 11:12 AM Joris Van den Bossche < jorisvandenboss...@gmail.com> wrote: > One example of consumer of our Cython API is cudf ( > https://github.com/rapidsai/cudf). > I am not very familiar with the package itself, but browsing its code, I > see that they do for example cimport RecordBatchReader ( > > https://github.com/rapidsai/cudf/blob/f6d31fa95d9b8d8658301438d0f9ba22a1c131aa/python/cudf/cudf/_lib/gpuarrow.pyx#L20 > ), > a case that would be impacted by > https://github.com/apache/arrow/pull/10162 > > Another question: do we regard `from pyarrow.includes.* import xx` as > public? > > On Fri, 20 Aug 2021 at 12:25, Alessandro Molina < > alessan...@ursacomputing.com> wrote: > > > ... > > > > Personally, even at risk of breaking third parties code, I think it would > > be wise to aim for the minimum exposed surface. I'd consider Cython > mostly > > an implementation detail and promote usage of libarrow from C/C++ > directly > > if you need to work on high performance Python extensions. > > > > Personally I am not sure if such recommendation is necessarily needed or > useful (re-using our Cython code makes it easier to develop python > extensions > that interact with (py)arrow). >
