Well, a mid-week release candidate isn't looking too likely. I've spent most of today working on the Gandiva Windows build and haven't been able to do much 0.12 backlog items yesterday or today so far.
Krisztian -- is your GPG key in KEYS yet? If not, you cannot cut a release. Out of the remaining issues in the backlog https://issues.apache.org/jira/projects/ARROW/versions/12343858 Python issues * ARROW-2038: small comment to be addressed, then merged. Someone please create follow up JIRA about CI for S3 * ARROW-2298: I just nixed this for 0.12. There is a patch up but it needs more work * ARROW-2659 and ARROW-2860 are the same issue, I think. We should fix as it has impacted many users and been reported many times * ARROW-3344: Recurring Plasma test failure on Ubuntu 14.04. I will take a look and see if the fix is difficult or not * ARROW-3428: Close to merge ready, I will review and make sure all is good * ARROW-4138: Windows rough edge, I will look * ARROW-3916: There is a patch here; can someone look? Given the overlap between Arrow and Parquet users it would be good to fix this * ARROW-4181: Failing large_memory test -- can someone look (and also see if any other large_memory tests are failing)? Rust items: can be merged but need not block release * ARROW-4040 * ARROW-4193 The other issues * ARROW-3578: The release manager will need to be careful to see what happens with RAT in the build * ARROW-4199: patch available * ARROW-854: To be merged with experimental designation on green build * ARROW-4197: Emscripten issues with C++. Would be good to merge fix in time for release I would say we should close the backlog by end-of-week at latest and move forward with the release Thanks Wes On Mon, Jan 7, 2019 at 10:08 AM Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > > Cool. I'm going to help with the python issues right after fixing the spark > integration tests. > > On Mon, Jan 7, 2019 at 4:43 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > Great, thank you! > > > > There are 20 0.12 issues Open or In-Progress, I'm going to tackle a > > couple more Python things today and tomorrow, but let's see where we > > stand by Wednesday or so and decide when to cut the release > > > > On Mon, Jan 7, 2019 at 4:38 AM Krisztián Szűcs > > <szucs.kriszt...@gmail.com> wrote: > > > > > > Hey! > > > > > > On Fri, Jan 4, 2019 at 10:31 PM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > hi all, > > > > > > > > We should try to cut a release candidate for 0.12 as soon as > > > > practical. Since we're just coming off the holidays, it would be good > > > > to work for a few more business days to close out as many outstanding > > > > patches as possible, and be in position to start a vote sometime next > > > > week. > > > > > > > > There's a bunch of Python bugs in the backlog still -- if anyone can > > > > pick up one or two of these it would be a help > > > > > > > > Would someone (Krisztian or Antoine maybe?) like to be the release > > manager? > > > > > > > Yes, I volunteer :) > > > > > > > > > > > Thanks > > > > > > > > On Wed, Dec 12, 2018 at 8:15 AM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > > > > I agree that we should aim for time-based releases. Let's discuss a > > > > > time-based release schedule (my preference would be ~every 2 months) > > > > > for 2019 after we get 0.12 out. > > > > > On Wed, Dec 12, 2018 at 3:15 AM Antoine Pitrou <anto...@python.org> > > > > wrote: > > > > > > > > > > > > > > > > > > I think we should aim for time-based releases in general (rather > > than a > > > > > > specific set of features), but delaying this one sounds good to me. > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > Le 12/12/2018 à 01:34, Wes McKinney a écrit : > > > > > > > hi all, > > > > > > > > > > > > > > I'm looking at the 0.12 backlog and I am not too comfortable > > with the > > > > > > > things that would have to be cut to get a release out next week. > > > > > > > Additionally, not a lot of developers are going to be working the > > > > week > > > > > > > of December 24 because of the Christmas and New Year's holidays, > > so > > > > > > > even if we did release, it might not get seen by a lot of people > > > > until > > > > > > > after the New Year. > > > > > > > > > > > > > > Based on this, I would suggest we push to complete as much work > > as > > > > > > > possible (from the 0.12 backlog and beyond) by the end of the > > year, > > > > > > > and release as soon as possible in 2019. Of course, anyone is > > welcome > > > > > > > to contribute work that is not found in the 0.12 milestone =) > > > > > > > > > > > > > > Any objections? > > > > > > > > > > > > > > Thanks > > > > > > > Wes > > > > > > > On Mon, Dec 10, 2018 at 8:04 AM Andy Grove < > > andygrov...@gmail.com> > > > > wrote: > > > > > > >> > > > > > > >> Cool. I will continue to add primitive operations but I am now > > > > adding this > > > > > > >> in a separate source file to keep it separate from the core > > array > > > > code. > > > > > > >> > > > > > > >> I'm not sure how important it will be to support Rust data > > sources > > > > with > > > > > > >> Gandiva. I can see that each language should be able to > > construct > > > > the > > > > > > >> logical query plan to submit to Gandiva and let Gandiva handle > > > > execution. I > > > > > > >> think the more interesting part is how do we support > > > > language-specific > > > > > > >> lambda functions as part of that logical query plan. Maybe it is > > > > possible > > > > > > >> to compile the lambda down to LLVM (I haven't started learning > > > > about LLVM > > > > > > >> in detail yet so this is wild speculation on my part). Another > > > > option is > > > > > > >> for Gandiva to support calling into shared libraries and that > > maybe > > > > is > > > > > > >> simpler for languages that support building C-native shared > > > > libraries (Rust > > > > > > >> supports this with zero overhead). > > > > > > >> > > > > > > >> Andy. > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Sun, Dec 9, 2018 at 11:42 AM Wes McKinney < > > wesmck...@gmail.com> > > > > wrote: > > > > > > >> > > > > > > >>> hi Andy, > > > > > > >>> > > > > > > >>> I can see an argument for having some basic native function > > kernel > > > > > > >>> support in Rust. One of the things that Gandiva has begun is a > > > > > > >>> Protobuf-based serialized representation representation of > > > > projection > > > > > > >>> and filter expressions. In the long run I would like to see a > > more > > > > > > >>> complete relational algebra / logical query plan that can be > > > > submitted > > > > > > >>> for execution. There's complexities, though, such as bridging > > > > > > >>> iteration of data sources written in Rust, say, with a query > > engine > > > > > > >>> written in C++. You would need to provide some kind of a > > callback > > > > > > >>> mechanism for the query engine to request the next chunk of a > > > > dataset > > > > > > >>> to be materialized. > > > > > > >>> > > > > > > >>> It will be interested to see what contributors will be > > motivated > > > > > > >>> enough to build over the next few years. At the end of the day, > > > > Apache > > > > > > >>> projects are do-ocracies. > > > > > > >>> > > > > > > >>> - Wes > > > > > > >>> On Fri, Dec 7, 2018 at 6:22 AM Andy Grove < > > andygrov...@gmail.com> > > > > wrote: > > > > > > >>>> > > > > > > >>>> I've added one PR to the list ( > > > > https://github.com/apache/arrow/pull/3119 > > > > > > >>> ) > > > > > > >>>> to update the project to use Rust 2018 Edition. > > > > > > >>>> > > > > > > >>>> I'm also considering removing one PR from the list and would > > like > > > > to get > > > > > > >>>> opinions here. > > > > > > >>>> > > > > > > >>>> I have a PR (https://github.com/apache/arrow/pull/3033) to > > add > > > > some > > > > > > >>> basic > > > > > > >>>> math and comparison operators to primitive arrays. These are > > baby > > > > steps > > > > > > >>>> towards implementing more query execution capabilities such as > > > > > > >>> projection, > > > > > > >>>> selection, etc but Chao made a good point that other Rust > > > > implementations > > > > > > >>>> don't have these kind of capabilities and I am now wondering > > if > > > > this is a > > > > > > >>>> distraction. We already have Gandiva and the new efforts in > > Ursa > > > > labs and > > > > > > >>>> it would probably make more sense to look at having Rust > > bindings > > > > for the > > > > > > >>>> query execution capabilities there rather than having a > > competing > > > > (and > > > > > > >>> less > > > > > > >>>> capable) implementation in Rust. > > > > > > >>>> > > > > > > >>>> Thoughts? > > > > > > >>>> > > > > > > >>>> Andy. > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> On Thu, Dec 6, 2018 at 8:42 PM paddy horan < > > > > paddyho...@hotmail.com> > > > > > > >>> wrote: > > > > > > >>>> > > > > > > >>>>> Other than Andy’s PR below I’m going to try and find time to > > > > work on > > > > > > >>>>> ARROW-3827, I’ll bump it 0.13 if I can’t find the time early > > > > next week. > > > > > > >>>>> There is nothing else in the 0.12 backlog for Rust. It > > would be > > > > nice > > > > > > >>> to > > > > > > >>>>> get the parquet merge in though. > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> Paddy > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> > > > > > > >>>>> ________________________________ > > > > > > >>>>> From: Andy Grove <andygrov...@gmail.com> > > > > > > >>>>> Sent: Thursday, December 6, 2018 10:20:48 AM > > > > > > >>>>> To: dev@arrow.apache.org > > > > > > >>>>> Subject: Re: Timeline for Arrow 0.12.0 release > > > > > > >>>>> > > > > > > >>>>> I have PRs pending for all the Rust issues that I want to get > > > > into > > > > > > >>> 0.12.0 > > > > > > >>>>> and would appreciate some reviews so I can go ahead and > > merge: > > > > > > >>>>> > > > > > > >>>>> https://github.com/apache/arrow/pull/3033 (covers > > ARROW-3880 and > > > > > > >>>>> ARROW-3881 > > > > > > >>>>> - add math and comparison operations to primitive arrays) > > > > > > >>>>> https://github.com/apache/arrow/pull/3096 (ARROW-3885 - Rust > > > > release > > > > > > >>>>> process) > > > > > > >>>>> https://github.com/apache/arrow/pull/3111 (ARROW-3838 - CSV > > > > Writer) > > > > > > >>>>> > > > > > > >>>>> With these in place I plan on writing a tutorial for reading > > a > > > > CSV > > > > > > >>> file, > > > > > > >>>>> performing some operations on primitive arrays and writing > > the > > > > output > > > > > > >>> to a > > > > > > >>>>> new CSV file. > > > > > > >>>>> > > > > > > >>>>> I am deferring ARROW-3882 (casting for primitive arrays) to > > > > 0.13.0 > > > > > > >>>>> > > > > > > >>>>> Thanks, > > > > > > >>>>> > > > > > > >>>>> Andy. > > > > > > >>>>> > > > > > > >>>>> On Tue, Dec 4, 2018 at 7:57 PM Andy Grove < > > andygrov...@gmail.com > > > > > > > > > > > >>> wrote: > > > > > > >>>>> > > > > > > >>>>>> I'd love to tackle the three related issues for supporting > > > > simple > > > > > > >>>>>> math/comparison operations on primitive arrays and casting > > > > primitive > > > > > > >>>>> arrays > > > > > > >>>>>> but since the change to use Rust specialization feature I'm > > a > > > > bit > > > > > > >>> stuck > > > > > > >>>>> and > > > > > > >>>>>> need some assistance applying the math operations to the > > numeric > > > > > > >>> types > > > > > > >>>>> and > > > > > > >>>>>> not the boolean primitives. I have added a comment to > > > > > > >>>>>> https://github.com/apache/arrow/pull/3033 ... if I can get > > help > > > > > > >>> solving > > > > > > >>>>>> for this PR then I should be able to handle the others. I'll > > > > also do > > > > > > >>> some > > > > > > >>>>>> research and try and figure this out myself. > > > > > > >>>>>> > > > > > > >>>>>> Andy. > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> On Tue, Dec 4, 2018 at 7:03 PM Wes McKinney < > > > > wesmck...@gmail.com> > > > > > > >>> wrote: > > > > > > >>>>>> > > > > > > >>>>>>> Andy, Paddy, or other Rust developers -- could you review > > the 6 > > > > > > >>> issues > > > > > > >>>>>>> in TODO in the 0.12 backlog and either assign them or move > > > > them to > > > > > > >>> the > > > > > > >>>>>>> next release if they aren't going to be completed this > > week or > > > > next? > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>> On Fri, Nov 30, 2018 at 4:34 PM Wes McKinney < > > > > wesmck...@gmail.com> > > > > > > >>>>> wrote: > > > > > > >>>>>>>> > > > > > > >>>>>>>> hi folks, > > > > > > >>>>>>>> > > > > > > >>>>>>>> Tomorrow is December 1. The last major Arrow release > > (0.11.0) > > > > took > > > > > > >>>>>>>> place on October 8. Given how much work has happened in > > the > > > > > > >>> project in > > > > > > >>>>>>>> the last ~2 months, I think it would be great to complete > > the > > > > next > > > > > > >>>>>>>> major release before the end-of-year holidays set in. > > > > > > >>>>>>>> > > > > > > >>>>>>>> I've been curating the JIRA backlog the last couple of > > weeks, > > > > and > > > > > > >>> have > > > > > > >>>>>>>> just created a 0.12.0 release wiki page to help us stay > > > > organized > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>> > > > > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.12.0+Release > > > > > > >>>>>>>> > > > > > > >>>>>>>> Given that there are only 3 full working weeks between > > now and > > > > > > >>>>>>>> Christmas, I think we should be in position to cut a > > release > > > > by > > > > > > >>> the > > > > > > >>>>>>>> end of the week of December 10, i.e. by Friday December > > 14. > > > > Not > > > > > > >>> all of > > > > > > >>>>>>>> the TODO issues have to be completed to make the release, > > but > > > > it > > > > > > >>> would > > > > > > >>>>>>>> be good to push to complete as much as possible. Please > > help > > > > by > > > > > > >>>>>>>> reviewing the backlog, and if possible, assigning issues > > to > > > > > > >>> yourself > > > > > > >>>>>>>> that you'd like to pursue in the next 2 weeks. > > > > > > >>>>>>>> > > > > > > >>>>>>>> Let me know if this sounds reasonable, or any concerns. > > > > > > >>>>>>>> > > > > > > >>>>>>>> Thanks > > > > > > >>>>>>>> Wes > > > > > > >>>>>>> > > > > > > >>>>>> > > > > > > >>>>> > > > > > > >>> > > > > > >
