Dear all, I agree with Wes and Antoine, the way things are currently handled is not sustainable. If we are using wheels, it can only work if everybody is using the same toolchain. In the past the Arrow contributors have tried to "fix" TensorFlows non-compliance with the manylinux1 standards from the Arrow side, with little or no success (the most recent attempt which also doesn't solve the problem is [1], and there are many before). See also the long thread in [2]. Hopefully we can find a solution that works in the longer term:
- switch everybody to conda or - make TensorFlow compatible with the current manylinux standard or - define a new manylinux standard that everybody uses (see also [3]) More help with this from the TensorFlow side would be greatly appreciated. Best wishes, Philipp. [1] https://github.com/apache/arrow/pull/4232 [2] https://groups.google.com/a/tensorflow.org/forum/m/#!topic/build/WgtWKA4t_bs [3] https://github.com/apache/arrow/pull/4391 On Fri, Jun 21, 2019 at 10:29 AM Antoine Pitrou <[email protected]> wrote: > > I'm not only thinking about Google though :-) > > More generally, our woes with the wheel compliance process (especially > on Linux, but even on Windows and macOS we must be careful to bundle > absolutely everything) make it a very costly workaround for our shyness > to tell users to "just use conda". > > That's more of a rant at this point, though. > > Regards > > Antoine. > > > > Le 21/06/2019 à 18:54, Wes McKinney a écrit : > > I agree that wheels are a nuisance for package maintainers and Google > > would be doing everyone a favor if they would either stop using them > > or conform to the standard (or an evolved version thereof). > > > > On Fri, Jun 21, 2019 at 11:35 AM Antoine Pitrou <[email protected]> > wrote: > >> > >> > >> Side note: it's not only STL containers, it's also any non-trivial > >> stdlib type that appears in headers. Such as std::shared_ptr<>. > >> > >> So I'm not sure the endeavour makes sense at all. You'll have to > >> try and follow the libstdc++ ABI spec: > >> https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html > >> > >> Regards > >> > >> Antoine. > >> > >> > >> On Fri, 21 Jun 2019 18:12:02 +0200 > >> Antoine Pitrou <[email protected]> wrote: > >>> On Thu, 20 Jun 2019 15:47:49 -0700 > >>> Zhuo Peng <[email protected]> wrote: > >>>> > >>>> One might argue that everyone releasing manylinux1 packages should use > >>>> exactly the same compiler, as provided by the pypa docker image, > however > >>>> the standard only specifies the maximum versions of corresponding > >>>> fundamental libraries [5]. Newer GCC versions could be backported to > work > >>>> with older libraries [6]. > >>>> > >>>> A recent change in Arrow [7] has removed most (but not all [8]) of > the STL > >>>> members in publicly accessible class declarations and will resolve our > >>>> immediate problem, but I wonder if there is, or there should be an > explicit > >>>> policy on the ABI compatibility, especially regarding the usage of > template > >>>> functions / classes in public interfaces? > >>> > >>> IMHO, the only reasonable policy for now is that there is no ABI > >>> compatibility. If you'd like to benefit from the PyArrow binary > >>> packages, including the C++ API, then you need to use the same > toolchain > >>> (or an ABI-compatible toolchain, but I'm afraid there's no clear > >>> specification of ABI compatibility in g++ / libstdc++ land). > >>> > >>>> * Our wheel cannot pass “auditwheel repair” > >>>> > >>>> I don’t think it’s correct to pull libarrow.so and libarrow_python.so > into > >>>> our wheel and have user’s Python load both our libarrow.so and > pyarrow’s, > >>>> but that’s what “auditwheel repair” attempts to do. But if we don’t > allow > >>>> auditwheel to do so, it refuses to stamp on our wheel because it has > >>>> “external” dependencies. > >>> > >>> You know, I wish the scientific communities would stop producing wheels > >>> and instead encourage users to switch to conda. The wheel paradigm is > >>> conceptually antiquated and is really a nuisance to package developers > >>> and maintainers. > >>> > >>> Regards > >>> > >>> Antoine. > >>> > >>> > >>> > >> > >> > >> >
