Re: TensorFlow, PyTorch, and manylinux1

Hey all, Just a quick reminder that we're gonna have the follow-up call tomorrow (Tuesday) 5pm UTC, 9am PST, noon EST, 1am Wednesday Singapore. (About 23hrs from this email) so the folks in europe can make the call too. It'll be a hangouts call same as before and we'll put the link and dial-in numb

Re: TensorFlow, PyTorch, and manylinux1

Would building our manylinux2010 wheels against https://github.com/pypa/manylinux/pull/252 solve the C++11 problems? In that case we should just do that. Otherwise let's propose a minimally modified manylinux2011 that fixes C++11 support so we can move on and don't have to wait 9 more months till m

Re: TensorFlow, PyTorch, and manylinux1

The problems arose if some functionality of C++11 were used. It led to certain symbols being statically linked into the shared library which clashed with other shared libraries that had the same symbols in the same address space, linked against a different version of libstdc++ (specifically, tenso

Re: TensorFlow, PyTorch, and manylinux1

Le 06/02/2019 à 14:27, Manuel Klimek a écrit : > On Wed, Feb 6, 2019 at 12:38 PM Antoine Pitrou > wrote: > > > Le 06/02/2019 à 01:06, Philipp Moritz a écrit : > > Thanks for the meeting! One question concerning a point that is still > > not super clear to

Re: TensorFlow, PyTorch, and manylinux1

Le 06/02/2019 à 01:06, Philipp Moritz a écrit : > Thanks for the meeting! One question concerning a point that is still > not super clear to me: > > Say we define a new manylinux standard based on gcc >=5 (with stable > c++11 support). There will still be a lot of wheels form the manylinux1 > da

Re: TensorFlow, PyTorch, and manylinux1

Thanks everyone for attending the meeting! It was great to have people from so many different groups so we can figure out how to solve this best for everyone. :) A lot was discussed, I split the notes from the wheel part of the discussion out into a separate doc: https://docs.google.com/document/d

Re: TensorFlow, PyTorch, and manylinux1

Thanks for the meeting! One question concerning a point that is still not super clear to me: Say we define a new manylinux standard based on gcc >=5 (with stable c++11 support). There will still be a lot of wheels form the manylinux1 days that are built against gcc 4.8 that might use the c++11 fea

Re: TensorFlow, PyTorch, and manylinux1

Le 05/02/2019 à 16:29, Manuel Klimek a écrit : > > manylinux2010 hasn't stalled, it's been progressing slowly.  Apparently > pip 19.0 is out which supports downloading and installing manylinux2010 > packages.  See status page here: > https://github.com/pypa/manylinux/issues/179#i

Re: TensorFlow, PyTorch, and manylinux1

Le 05/02/2019 à 16:22, Manuel Klimek a écrit : > On Tue, Feb 5, 2019 at 2:01 PM Uwe L. Korn > wrote: > > Also to reiterate a point raised earlier: C++11 with manylinux1 > works smoothly. With gcc 4.8.5, everything we need in Arrow > supported. C++14 and mor

Re: TensorFlow, PyTorch, and manylinux1

> > From the requirements side (Martin will correct me if I'm getting these > wrong): > - it seems like from the TF point of view, our users are on pip, so we > need to deliver there > - LLVM is going to require C++14 ~in March as far as I can tell > - from trying to find info about manylinux2010 /

Re: TensorFlow, PyTorch, and manylinux1

Replying to the thread because the last two messages got dropped. On Mon, Feb 4, 2019 at 10:00 AM soumith wrote: > > I think trying to package CUDA is the wrong way to think about it. > Instead, perhaps you should try to make the package compatible with > system CUDA installs. > > I agree in pri

Re: TensorFlow, PyTorch, and manylinux1

> I think trying to package CUDA is the wrong way to think about it. Instead, perhaps you should try to make the package compatible with system CUDA installs. I agree in principle. The problem fundamentally stems from user expectation. In my ~6+ years of supporting Torch and PyTorch, installing C

Re: TensorFlow, PyTorch, and manylinux1

On Tue, 5 Feb 2019 01:45:34 +0800 Jason Zaman wrote: > On Tue, 5 Feb 2019 at 01:30, soumith wrote: > > > > Unfortunately I'll be on a long flight, and cannot make it to the SIGBuild > > meeting. > > I'm definitely interested in the meeting notes and any follow-up meeting. > > > > > I think we

Re: TensorFlow, PyTorch, and manylinux1

On Tue, 5 Feb 2019 at 01:30, soumith wrote: > > Unfortunately I'll be on a long flight, and cannot make it to the SIGBuild > meeting. > I'm definitely interested in the meeting notes and any follow-up meeting. > > > I think we should leave CUDA out of the > discussion initially and see if we can

Re: TensorFlow, PyTorch, and manylinux1

Unfortunately I'll be on a long flight, and cannot make it to the SIGBuild meeting. I'm definitely interested in the meeting notes and any follow-up meeting. > I think we should leave CUDA out of the discussion initially and see if we can get the cpu-only wheel working correctly. Hopefully cpu-onl

Re: TensorFlow, PyTorch, and manylinux1

Le 04/02/2019 à 17:36, Uwe L. Korn a écrit : > I think that problem is whether this would get merged. Conda was created > after a meeting with Guido van Rossum and other folks at a PyCon quite > some years ago where the final call was that this is not a problem of > the core Python ecosystem and

Re: TensorFlow, PyTorch, and manylinux1

Hm, lets have this SIG-Build meeting as scheduled and then have another follow-up later probably around 9am PST, 6pm Europe, 1am Singapore. Does that time work for everyone? (Date TBD). My take on this whole thing is that it sounds a lot like re-implementing an entire distro complete with package

Re: TensorFlow, PyTorch, and manylinux1

yeah that's expected. The timing is complicated with people spread all over. We will post notes after the meeting on the SIG-Build mailing list and I'd also be up for organizing a separate call with europe folks if that would be of interest. On Mon, 4 Feb 2019 at 19:29, 'Manuel Klimek' via SIG Bui

Re: TensorFlow, PyTorch, and manylinux1

Hey all, We're having the TensorFlow SIG-Build meeting on 5th Feb 3pm PST (https://www.timeanddate.com/worldclock/fixedtime.html?iso=20190205T15&p1=224). Agenda is linked from: https://groups.google.com/a/tensorflow.org/forum/#!topic/build/akyPcGoBIy4 I'd like to invite everyone from this thread

Re: TensorFlow, PyTorch, and manylinux1

Le 30/01/2019 à 16:09, Manuel Klimek a écrit : > > On Wed, Jan 30, 2019 at 3:51 PM Antoine Pitrou > wrote: > > > Le 30/01/2019 à 15:35, Manuel Klimek a écrit : > > > >     Am I reading you wrong, or are you actually proposing to > package another >

Re: TensorFlow, PyTorch, and manylinux1

Le 30/01/2019 à 15:35, Manuel Klimek a écrit : > > Am I reading you wrong, or are you actually proposing to package another > libstdc++ version as a Python wheel? > > > That would be the idea. >   > > If so, are you going to claim that the given wheel is > manylinux-compatible

Re: TensorFlow, PyTorch, and manylinux1

Le 30/01/2019 à 14:30, Manuel Klimek a écrit : > > > > What would the requirements for such a toolchain wheel be for it > to have a chance to be widely used? (note that I come from a C++ > background and don't have a lot of experience with Python outside of > modules using C++

Re: TensorFlow, PyTorch, and manylinux1

rFlow Developers" >> To: soumith >> Cc: Jean-Marc Ludwig , bu...@tensorflow.org, Wes McKinney >> , d...@arrow.apache.org, Philipp Moritz >> , TensorFlow Developers , >> ray...@googlegroups.com, yi...@yifeifeng.com, Edd Wilder-James >> >> Sub

Re: TensorFlow, PyTorch, and manylinux1

Thanks Soumith and Martin for the detailed thoughts. Jean-Marc would you be able to chime in or perhaps cc the relevant people? It'd be really great to hear from someone at NVIDIA, since NVIDIA seems best positioned to make manlinux2010 work out and will probably need to be part of a plan for many

Re: TensorFlow, PyTorch, and manylinux1

> The group on this thread is a good start, maybe we can get together and make a proposal that meets the need of the scientific computing community? I think that would probably involve updating the minimum requirements (possibly to CentOS 7, I heard there was talk of a manylinux2014), carving out N

Re: TensorFlow, PyTorch, and manylinux1

Hey Travis, PyTorch and anaconda are actually smooth. There are no issues with Anaconda, and we officially maintain conda packages (it's also our recommended and default package manager). Conda-forge recipes are currently not possible because conda-forge hasn't finalized their CUDA packaging mech

Re: TensorFlow, PyTorch, and manylinux1

> Somehow we need to arrange that the same compiler toolchain (with consistent minimum glibc, libstdc++ version) is used to build all of the binaries we are discussing here. Short of that some system configurations will continue to have problems. This was exactly the purpose of Anaconda's crosstoo

Re: TensorFlow, PyTorch, and manylinux1

hi Soumith, On Mon, Dec 17, 2018 at 12:32 AM soumith wrote: > > I'm reposting my original reply below the current reply (below a dotted > line). It was filtered out because I wasn't subscribed to the relevant > mailing lists. > > tl;dr: manylinux2010 looks pretty promising, because CUDA suppor

Re: TensorFlow, PyTorch, and manylinux1

Can PyTorch provide and maintain a conda-forge recipe? This would allow the large and growing conda forge ecosystem to easily install PyTorch in a community-supported way. Are there problems with using conda or another general package manager? I agree that the machine learning packages are tryin

Re: TensorFlow, PyTorch, and manylinux1

I'm reposting my original reply below the current reply (below a dotted line). It was filtered out because I wasn't subscribed to the relevant mailing lists. tl;dr: manylinux2010 looks pretty promising, because CUDA supports CentOS6 (for now). In the meanwhile, I dug into what pyarrow does, and

Re: TensorFlow, PyTorch, and manylinux1

Reposting since I wasn't subscribed to develop...@tensorflow.org. I also didn't see Soumith's response since it didn't come through to dev@arrow.apache.org In response to the non-conforming ABI in the TF and PyTorch wheels, we have attempted to hack around the issue with some elaborate workarounds

Re: TensorFlow, PyTorch, and manylinux1

In response to the non-conforming ABI in the TF and PyTorch wheels, we have attempted to hack around the issue with some elaborate workarounds [1] [2] that have ultimately proved to not work universally. The bottom line is that this is burdening other projects in the Python ecosystem and causing co

Re: TensorFlow, PyTorch, and manylinux1

On Sat, Dec 15, 2018 at 8:43 PM Philipp Moritz wrote: > Dear all, > > As some of you know, there is a standard in Python called manylinux ( > https://www.python.org/dev/peps/pep-0513/) to package binary executables > and libraries into a “wheel” in a way that allows the code to be run on a > wide

TensorFlow, PyTorch, and manylinux1

Dear all, As some of you know, there is a standard in Python called manylinux ( https://www.python.org/dev/peps/pep-0513/) to package binary executables and libraries into a “wheel” in a way that allows the code to be run on a wide variety of Linux distributions. This is very convenient for Python