Hello Tim, thanks for the hint. I see that you build arrow by yourselves in the Dockerfile. Could it be that in the end you statically link the arrow libraries?
As there are no wheel on PyPI, I couldn't verify whether that assumption is true. Best Uwe On Thu, Jul 2, 2020, at 4:53 PM, Tim Paine wrote: > We spent a ton of time on this for perspective, the end result is a > mostly compatible set of wheels for most platforms, I believe we > skipped py2 but nobody cares about those anyway. We link against > libarrow and libarrow_python on Linux, on windows we vendor them all > into our library. Feel free to scrape the perspective repo's cmake > lists and setup.py for details. > > Tim Paine > tim.paine.nyc > > > On Jul 2, 2020, at 10:32, Uwe L. Korn <uw...@xhochy.com> wrote: > > > > I had so much fun with the wheels in the past, I'm now a happy member of > > conda-forge core instead :D > > > > The good thing first: > > > > * The C++ ABI didn't change between the manylinux versions, it is the old > > one in all cases. So you mix & match manylinux versions. > > > > The sad things: > > > > * The manylinuxX standard are intented to provide a way to ship > > *self-contained* wheels that run on any recent Linux. The important part > > here is that they need to be self-contained. Having a binary dependency on > > another wheel is actually not allowed. > > * Thus the snowflake-python-connector ships the libarrow.so it was build > > with as part of its wheel. In this case auditwheel is happy with the wheel. > > * It is working with numpy as a dependency because NumPy linkage is similar > > to the import lib behaviour on Windows: You don't actually link against > > numpy but you statically link a set of functions that are resolved to > > NumPy's function when you import numpy. Quick googling leads to > > https://github.com/yugr/Implib.so which could provide something similar for > > Linux. > > * You could actually omit linking to libarrow and try to populate the > > symbols before you load the library. This is how the Python symbols are > > available to extensions without linking to libpython. > > > > > >> On Thu, Jul 2, 2020, at 2:43 PM, Maarten Breddels wrote: > >> Ok, thanks! > >> > >> I'm setting up a repo with an example here, using pybind11: > >> https://github.com/vaexio/vaex-arrow-ext > >> > >> and I'll just try all possible combinations and report back. > >> > >> cheers, > >> > >> Maarten Breddels > >> Software engineer / consultant / data scientist > >> Python / C++ / Javascript / Jupyter > >> www.maartenbreddels.com / vaex.io > >> maartenbredd...@gmail.com +31 6 2464 0838 <+31+6+24640838> > >> [image: Twitter] <https://twitter.com/maartenbreddels>[image: Github] > >> <https://github.com/maartenbreddels>[image: LinkedIn] > >> <https://linkedin.com/in/maartenbreddels>[image: Skype] > >> > >> > >> > >> > >> Op do 2 jul. 2020 om 14:32 schreef Joris Van den Bossche < > >> jorisvandenboss...@gmail.com>: > >> > >>> Also no concrete answer, but one such example is turbodbc, I think. > >>> But it seems they only have conda binary packages, and don't > >>> distribute wheels .. > >>> (https://turbodbc.readthedocs.io/en/latest/pages/getting_started.html), > >>> so not that relevant as comparison (they also need to build against an > >>> odbc driver in addition to arrow). > >>> But maybe Uwe has some more experience in this regard (and with > >>> attempts building wheels for turbodbc, eg > >>> https://github.com/blue-yonder/turbodbc/pull/108). > >>> > >>> Joris > >>> > >>> On Thu, 2 Jul 2020 at 11:05, Antoine Pitrou <anto...@python.org> wrote: > >>>> > >>>> > >>>> Hi Maarten, > >>>> > >>>> Le 02/07/2020 à 10:53, Maarten Breddels a écrit : > >>>>> > >>>>> Also, I see pyarrow distributes manylinux1/2010/2014 wheels. Would a > >>> vaex > >>>>> extension distributed as a 2010 wheel, and build with the pyarrow 2010 > >>>>> wheel, work in an environment where someone installed a pyarrow 2014 > >>>>> wheel, or build from source, or installed from conda-forge? > >>>> > >>>> I have no idea about the concrete answer, but it probably depends > >>>> whether the libstdc++ ABI changed between those two versions. I'm > >>>> afraid you'll have to experiment yourself. > >>>> > >>>> (if you want to eschew C++ ABI issues, you may use the C Data Interface: > >>>> https://arrow.apache.org/docs/format/CDataInterface.html > >>>> though of course you won't have access to all the useful helpers in the > >>>> Arrow C++ library) > >>>> > >>>> Regards > >>>> > >>>> Antoine. > >>>> > >>>> > >>> > >> >
