I can confirm what Uwe said, manylinux doesn't cause issues. Here I've build inside a manylinux2010 docker a C++ Python extension (using the C++ of Arrow): https://github.com/vaexio/vaex-arrow-ext/runs/831763024?check_suite_focus=true It's built with the manylinux1 and manylinux2010 pyarrow wheel (the manylinux2014 cannot be installed in the manylinux2010 docker obviously)
After that, I've installed the manylinux2010 Python extension in the host OS, and ran the test with all manylinux versions of pyarrow I've done the same but now building the extension in the 2014 docker image, where I also check with building against the manylinux2014 pyarrow wheel. All combinations works. Regarding linking, as Uwe said, setting the symbol resolution to global, and not linking to the (libarrow.so and libarrow_python.so) libraries works: https://github.com/vaexio/vaex-arrow-ext/blob/0def54cc056710db5305477c308a1e68c9e9aba2/vaex_arrow_ext/__init__.py Note that this solution does not ship libarrow/libarrow_python in the wheel, but it could result into issues because symbols exported by libarrow.so and libarrow_python.so might be visible by other libraries (reading Uwe's article on arrow+tensorflow I guess that's settled). I'm not sure what people think of this solution, but it seems to work well. PS: I tried calling 'restoring' the symbol resolutions to local, but it seems to have no effect. Op do 2 jul. 2020 om 17:10 schreef Tim Paine <t.paine...@gmail.com>: > We build pyarrow in the docker image because auditwheel complains about > pyarrow otherwise which causes our wheels to fail auditwheel and not allow > the manylinux tag. But assuming we build pyarrow in the docker image, our > manylinux wheels that result are then compatible with the pyarrow manylinux > wheels. > > It has taken a few months of on-and-off work, you may need to also consult > this PR > https://github.com/finos/perspective/pull/1105/files < > https://github.com/finos/perspective/pull/1105/files> > > > > On Jul 2, 2020, at 11:04, Uwe L. Korn <uw...@xhochy.com> wrote: > > > > Hello Tim, > > > > thanks for the hint. I see that you build arrow by yourselves in the > Dockerfile. Could it be that in the end you statically link the arrow > libraries? > > > > As there are no wheel on PyPI, I couldn't verify whether that assumption > is true. > > > > Best > > Uwe > > > > On Thu, Jul 2, 2020, at 4:53 PM, Tim Paine wrote: > >> We spent a ton of time on this for perspective, the end result is a > >> mostly compatible set of wheels for most platforms, I believe we > >> skipped py2 but nobody cares about those anyway. We link against > >> libarrow and libarrow_python on Linux, on windows we vendor them all > >> into our library. Feel free to scrape the perspective repo's cmake > >> lists and setup.py for details. > >> > >> Tim Paine > >> tim.paine.nyc > >> > >>> On Jul 2, 2020, at 10:32, Uwe L. Korn <uw...@xhochy.com> wrote: > >>> > >>> I had so much fun with the wheels in the past, I'm now a happy member > of conda-forge core instead :D > >>> > >>> The good thing first: > >>> > >>> * The C++ ABI didn't change between the manylinux versions, it is the > old one in all cases. So you mix & match manylinux versions. > >>> > >>> The sad things: > >>> > >>> * The manylinuxX standard are intented to provide a way to ship > *self-contained* wheels that run on any recent Linux. The important part > here is that they need to be self-contained. Having a binary dependency on > another wheel is actually not allowed. > >>> * Thus the snowflake-python-connector ships the libarrow.so it was > build with as part of its wheel. In this case auditwheel is happy with the > wheel. > >>> * It is working with numpy as a dependency because NumPy linkage is > similar to the import lib behaviour on Windows: You don't actually link > against numpy but you statically link a set of functions that are resolved > to NumPy's function when you import numpy. Quick googling leads to > https://github.com/yugr/Implib.so which could provide something similar > for Linux. > >>> * You could actually omit linking to libarrow and try to populate the > symbols before you load the library. This is how the Python symbols are > available to extensions without linking to libpython. > >>> > >>> > >>>> On Thu, Jul 2, 2020, at 2:43 PM, Maarten Breddels wrote: > >>>> Ok, thanks! > >>>> > >>>> I'm setting up a repo with an example here, using pybind11: > >>>> https://github.com/vaexio/vaex-arrow-ext > >>>> > >>>> and I'll just try all possible combinations and report back. > >>>> > >>>> cheers, > >>>> > >>>> Maarten Breddels > >>>> Software engineer / consultant / data scientist > >>>> Python / C++ / Javascript / Jupyter > >>>> www.maartenbreddels.com / vaex.io > >>>> maartenbredd...@gmail.com +31 6 2464 0838 <+31+6+24640838> > >>>> [image: Twitter] <https://twitter.com/maartenbreddels>[image: Github] > >>>> <https://github.com/maartenbreddels>[image: LinkedIn] > >>>> <https://linkedin.com/in/maartenbreddels>[image: Skype] > >>>> > >>>> > >>>> > >>>> > >>>> Op do 2 jul. 2020 om 14:32 schreef Joris Van den Bossche < > >>>> jorisvandenboss...@gmail.com>: > >>>> > >>>>> Also no concrete answer, but one such example is turbodbc, I think. > >>>>> But it seems they only have conda binary packages, and don't > >>>>> distribute wheels .. > >>>>> ( > https://turbodbc.readthedocs.io/en/latest/pages/getting_started.html), > >>>>> so not that relevant as comparison (they also need to build against > an > >>>>> odbc driver in addition to arrow). > >>>>> But maybe Uwe has some more experience in this regard (and with > >>>>> attempts building wheels for turbodbc, eg > >>>>> https://github.com/blue-yonder/turbodbc/pull/108). > >>>>> > >>>>> Joris > >>>>> > >>>>> On Thu, 2 Jul 2020 at 11:05, Antoine Pitrou <anto...@python.org> > wrote: > >>>>>> > >>>>>> > >>>>>> Hi Maarten, > >>>>>> > >>>>>> Le 02/07/2020 à 10:53, Maarten Breddels a écrit : > >>>>>>> > >>>>>>> Also, I see pyarrow distributes manylinux1/2010/2014 wheels. Would > a > >>>>> vaex > >>>>>>> extension distributed as a 2010 wheel, and build with the pyarrow > 2010 > >>>>>>> wheel, work in an environment where someone installed a pyarrow > 2014 > >>>>>>> wheel, or build from source, or installed from conda-forge? > >>>>>> > >>>>>> I have no idea about the concrete answer, but it probably depends > >>>>>> whether the libstdc++ ABI changed between those two versions. I'm > >>>>>> afraid you'll have to experiment yourself. > >>>>>> > >>>>>> (if you want to eschew C++ ABI issues, you may use the C Data > Interface: > >>>>>> https://arrow.apache.org/docs/format/CDataInterface.html > >>>>>> though of course you won't have access to all the useful helpers in > the > >>>>>> Arrow C++ library) > >>>>>> > >>>>>> Regards > >>>>>> > >>>>>> Antoine. > >>>>>> > >>>>>> > >>>>> > >>>> > >> > >
