We spent a ton of time on this for perspective, the end result is a mostly compatible set of wheels for most platforms, I believe we skipped py2 but nobody cares about those anyway. We link against libarrow and libarrow_python on Linux, on windows we vendor them all into our library. Feel free to scrape the perspective repo's cmake lists and setup.py for details.
Tim Paine tim.paine.nyc > On Jul 2, 2020, at 10:32, Uwe L. Korn <uw...@xhochy.com> wrote: > > I had so much fun with the wheels in the past, I'm now a happy member of > conda-forge core instead :D > > The good thing first: > > * The C++ ABI didn't change between the manylinux versions, it is the old one > in all cases. So you mix & match manylinux versions. > > The sad things: > > * The manylinuxX standard are intented to provide a way to ship > *self-contained* wheels that run on any recent Linux. The important part here > is that they need to be self-contained. Having a binary dependency on another > wheel is actually not allowed. > * Thus the snowflake-python-connector ships the libarrow.so it was build with > as part of its wheel. In this case auditwheel is happy with the wheel. > * It is working with numpy as a dependency because NumPy linkage is similar > to the import lib behaviour on Windows: You don't actually link against numpy > but you statically link a set of functions that are resolved to NumPy's > function when you import numpy. Quick googling leads to > https://github.com/yugr/Implib.so which could provide something similar for > Linux. > * You could actually omit linking to libarrow and try to populate the symbols > before you load the library. This is how the Python symbols are available to > extensions without linking to libpython. > > >> On Thu, Jul 2, 2020, at 2:43 PM, Maarten Breddels wrote: >> Ok, thanks! >> >> I'm setting up a repo with an example here, using pybind11: >> https://github.com/vaexio/vaex-arrow-ext >> >> and I'll just try all possible combinations and report back. >> >> cheers, >> >> Maarten Breddels >> Software engineer / consultant / data scientist >> Python / C++ / Javascript / Jupyter >> www.maartenbreddels.com / vaex.io >> maartenbredd...@gmail.com +31 6 2464 0838 <+31+6+24640838> >> [image: Twitter] <https://twitter.com/maartenbreddels>[image: Github] >> <https://github.com/maartenbreddels>[image: LinkedIn] >> <https://linkedin.com/in/maartenbreddels>[image: Skype] >> >> >> >> >> Op do 2 jul. 2020 om 14:32 schreef Joris Van den Bossche < >> jorisvandenboss...@gmail.com>: >> >>> Also no concrete answer, but one such example is turbodbc, I think. >>> But it seems they only have conda binary packages, and don't >>> distribute wheels .. >>> (https://turbodbc.readthedocs.io/en/latest/pages/getting_started.html), >>> so not that relevant as comparison (they also need to build against an >>> odbc driver in addition to arrow). >>> But maybe Uwe has some more experience in this regard (and with >>> attempts building wheels for turbodbc, eg >>> https://github.com/blue-yonder/turbodbc/pull/108). >>> >>> Joris >>> >>> On Thu, 2 Jul 2020 at 11:05, Antoine Pitrou <anto...@python.org> wrote: >>>> >>>> >>>> Hi Maarten, >>>> >>>> Le 02/07/2020 à 10:53, Maarten Breddels a écrit : >>>>> >>>>> Also, I see pyarrow distributes manylinux1/2010/2014 wheels. Would a >>> vaex >>>>> extension distributed as a 2010 wheel, and build with the pyarrow 2010 >>>>> wheel, work in an environment where someone installed a pyarrow 2014 >>>>> wheel, or build from source, or installed from conda-forge? >>>> >>>> I have no idea about the concrete answer, but it probably depends >>>> whether the libstdc++ ABI changed between those two versions. I'm >>>> afraid you'll have to experiment yourself. >>>> >>>> (if you want to eschew C++ ABI issues, you may use the C Data Interface: >>>> https://arrow.apache.org/docs/format/CDataInterface.html >>>> though of course you won't have access to all the useful helpers in the >>>> Arrow C++ library) >>>> >>>> Regards >>>> >>>> Antoine. >>>> >>>> >>> >>
