Re: Confronting Arrow packaging problems

2018-03-29 Thread Wes McKinney
Some responses inline. Would be good to get some more feedback from folks in the Google doc. I think some of the particular details may remain hazy until we begin to lay hands to code to make things more concrete. On Mon, Mar 26, 2018 at 3:49 PM, Phillip Cloud wrote: > On Mon, Mar 26, 2018 at 1:3

Re: Confronting Arrow packaging problems

2018-03-26 Thread Phillip Cloud
On Mon, Mar 26, 2018 at 1:37 PM Wes McKinney wrote: > > Huge +1 on moving some of the packaging outside the scope of > responsibility of arrow dev, specifically I don't think we should be > responsible for anything except wheels and conda packages. > This is my ideal scenario, however unrealisti

Re: Confronting Arrow packaging problems

2018-03-26 Thread Wes McKinney
> Huge +1 on moving some of the packaging outside the scope of responsibility > of arrow dev, specifically I don't think we should be responsible for > anything except wheels and conda packages. In theory I agree, but until Apache Arrow grows popular enough and important enough for other communi

Re: Confronting Arrow packaging problems

2018-03-26 Thread Phillip Cloud
Responses inline. This kind of information is extremely helpful and informative. On Mon, Mar 26, 2018 at 11:26 AM Antoine Pitrou wrote: > > Hi, > > As someone who started contributing recently, I'd like to raise a few > points. I hope this post doesn't come accross as rambling or clueless, > ot

Re: Confronting Arrow packaging problems

2018-03-26 Thread Phillip Cloud
I've started a document on this issue here: https://docs.google.com/document/d/1IyhbQpiElxTsI8HbMZ-g9EGPOtcFdtMBzEyDJv48BKc/edit?usp=sharing On Sun, Mar 25, 2018 at 10:40 AM Wes McKinney wrote: > @Kou, automated commits / PRs to trigger Appveyor/Travis CI are most > likely part of the solution,

Re: Confronting Arrow packaging problems

2018-03-26 Thread Antoine Pitrou
Hi, As someone who started contributing recently, I'd like to raise a few points. I hope this post doesn't come accross as rambling or clueless, otherwise feel free to ignore / point it out :-) What does a release require? I didn't find any official documentation

Re: Confronting Arrow packaging problems

2018-03-25 Thread Wes McKinney
@Kou, automated commits / PRs to trigger Appveyor/Travis CI are most likely part of the solution, but there are other issues. We should start a Google document or something to enumerate all the things we want to implement and identify the technical issues with each thing. For example, at present,

Re: Confronting Arrow packaging problems

2018-03-25 Thread Kouhei Sutou
Hi, How about creating a pull request to apache/arrow-dist with changes to use the latest apache/arrow? (A sample script exists at the end.) If the pull request is created, we can test packaging with the latest apache/arrow on Travis CI. If we add the following configuration to .travis.yml in apa

Re: Confronting Arrow packaging problems

2018-03-25 Thread Uwe L. Korn
Hello all, I strongly support Wes' points about having better automated feedback in our build chain is essential, independent from the tool we use to make the builds. As it sadly seems that our newly uploaded Arrow wheels for OSX are broken, I'm going to start there and add them to the build ma

Re: Confronting Arrow packaging problems

2018-03-24 Thread Krisztián Szűcs
Just want to mention two other build systems: Pants: https://github.com/pantsbuild/pants (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c...@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fpantsbuild%2Fpants&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D) Meson:

Re: Confronting Arrow packaging problems

2018-03-24 Thread Wes McKinney
hi Phillip, Thank you for these thoughts. I am all for improving our build systems to be more readable/maintainable and easier to debug and develop. I see our problems are being different than improving the build systems for the Python and C++ libraries: we need to set up a build and packaging au

Re: Confronting Arrow packaging problems

2018-03-24 Thread Phillip Cloud
I think we need to use a tool that can perform every single step of the deployment process, end-to-end. Right now, cmake isn't cutting it IMO because it lends itself quite heavily to copy pasting and oodles of bash scripts that are indecipherable by anyone except the original author. With that in

Re: Confronting Arrow packaging problems

2018-03-23 Thread Holden Karau
I know in Spark we’ve benefited by having some of the different language devs act as RMs and each time that language dev has ended up improving a bunch of how their components packaging has been done. Not to suggest we should just do what other projects do, but maybe an idea to consider? On Fri, M