I did try the approach to not link against pyarrow but leave out the symbols,
just ensure pyarrow is imported before the vaex extension. This works
out-of-the-box on macOS but fails on Linux as symbols have a scope there.
Adding the following lines to load Arrow into the global scope made it work
though:
import ctypes
arrow_python = ctypes.CDLL('libarrow.so', ctypes.RTLD_GLOBAL)
libarrow_python = ctypes.CDLL('libarrow_python.so', ctypes.RTLD_GLOBAL)
On Thu, Jul 2, 2020, at 4:32 PM, Uwe L. Korn wrote:
> I had so much fun with the wheels in the past, I'm now a happy member
> of conda-forge core instead :D
>
> The good thing first:
>
> * The C++ ABI didn't change between the manylinux versions, it is the
> old one in all cases. So you mix & match manylinux versions.
>
> The sad things:
>
> * The manylinuxX standard are intented to provide a way to ship
> *self-contained* wheels that run on any recent Linux. The important
> part here is that they need to be self-contained. Having a binary
> dependency on another wheel is actually not allowed.
> * Thus the snowflake-python-connector ships the libarrow.so it was
> build with as part of its wheel. In this case auditwheel is happy with
> the wheel.
> * It is working with numpy as a dependency because NumPy linkage is
> similar to the import lib behaviour on Windows: You don't actually link
> against numpy but you statically link a set of functions that are
> resolved to NumPy's function when you import numpy. Quick googling
> leads to https://github.com/yugr/Implib.so which could provide
> something similar for Linux.
> * You could actually omit linking to libarrow and try to populate the
> symbols before you load the library. This is how the Python symbols are
> available to extensions without linking to libpython.
>
>
> On Thu, Jul 2, 2020, at 2:43 PM, Maarten Breddels wrote:
> > Ok, thanks!
> >
> > I'm setting up a repo with an example here, using pybind11:
> > https://github.com/vaexio/vaex-arrow-ext
> >
> > and I'll just try all possible combinations and report back.
> >
> > cheers,
> >
> > Maarten Breddels
> > Software engineer / consultant / data scientist
> > Python / C++ / Javascript / Jupyter
> > www.maartenbreddels.com / vaex.io
> > maartenbredd...@gmail.com +31 6 2464 0838 <+31+6+24640838>
> > [image: Twitter] <https://twitter.com/maartenbreddels>[image: Github]
> > <https://github.com/maartenbreddels>[image: LinkedIn]
> > <https://linkedin.com/in/maartenbreddels>[image: Skype]
> >
> >
> >
> >
> > Op do 2 jul. 2020 om 14:32 schreef Joris Van den Bossche <
> > jorisvandenboss...@gmail.com>:
> >
> > > Also no concrete answer, but one such example is turbodbc, I think.
> > > But it seems they only have conda binary packages, and don't
> > > distribute wheels ..
> > > (https://turbodbc.readthedocs.io/en/latest/pages/getting_started.html),
> > > so not that relevant as comparison (they also need to build against an
> > > odbc driver in addition to arrow).
> > > But maybe Uwe has some more experience in this regard (and with
> > > attempts building wheels for turbodbc, eg
> > > https://github.com/blue-yonder/turbodbc/pull/108).
> > >
> > > Joris
> > >
> > > On Thu, 2 Jul 2020 at 11:05, Antoine Pitrou <anto...@python.org> wrote:
> > > >
> > > >
> > > > Hi Maarten,
> > > >
> > > > Le 02/07/2020 à 10:53, Maarten Breddels a écrit :
> > > > >
> > > > > Also, I see pyarrow distributes manylinux1/2010/2014 wheels. Would a
> > > vaex
> > > > > extension distributed as a 2010 wheel, and build with the pyarrow 2010
> > > > > wheel, work in an environment where someone installed a pyarrow 2014
> > > > > wheel, or build from source, or installed from conda-forge?
> > > >
> > > > I have no idea about the concrete answer, but it probably depends
> > > > whether the libstdc++ ABI changed between those two versions. I'm
> > > > afraid you'll have to experiment yourself.
> > > >
> > > > (if you want to eschew C++ ABI issues, you may use the C Data Interface:
> > > > https://arrow.apache.org/docs/format/CDataInterface.html
> > > > though of course you won't have access to all the useful helpers in the
> > > > Arrow C++ library)
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > >
> >
>
