Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-07-07 Thread David Li
I just filed https://issues.apache.org/jira/browse/ARROW-9362 and plan to work on it tomorrow. David On 7/7/20, Wes McKinney wrote: > I don't recall a ticket for the Java work but you're certainly a good > candidate to take the lead on it. > > On Tue, Jul 7, 2020 at 3:16 PM David Li wrote: >> >

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-07-07 Thread Wes McKinney
I don't recall a ticket for the Java work but you're certainly a good candidate to take the lead on it. On Tue, Jul 7, 2020 at 3:16 PM David Li wrote: > > I see there's ARROW-9258 to do the backwards compatibility work for > C++ and ARROW-9333 to expose this for Python; is there any ticket or > a

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-07-07 Thread David Li
I see there's ARROW-9258 to do the backwards compatibility work for C++ and ARROW-9333 to expose this for Python; is there any ticket or anyone planning on doing this for Java? Otherwise I'm willing to look at it so that we can do some testing with Flight. Best, David On 6/29/20, Wes McKinney wr

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-29 Thread Wes McKinney
Thanks David. Indeed it seems that exposing IpcWriteOptions is going to be critical here. I'd like to avoid an "environment variable" workaround at the C++ level instead only providing such things in e.g. Python like we did for the alignment patch On Mon, Jun 29, 2020 at 9:30 AM David Li wrote: >

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-29 Thread David Li
This would cause compatibility issues for Flight servers/clients between versions as well. The situation is a little worse since IpcWriteOptions isn't exposed and so you can't control what version you write. But just exposing them in lieu of a full negotiation (which we should start thinking about)

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-28 Thread Wes McKinney
I opened a PR https://github.com/apache/arrow/pull/7566 We should prioritize getting through the other format changes, but we can vote on this in the meantime if there is consensus On Fri, Jun 26, 2020 at 2:58 PM Micah Kornfield wrote: > > I agree I think we have to do this given the number of c

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-26 Thread Micah Kornfield
I agree I think we have to do this given the number of changes in flight (especially union types). On Fri, Jun 26, 2020 at 7:29 AM Wes McKinney wrote: > I created a JIRA about this > > https://issues.apache.org/jira/browse/ARROW-9231 > > This issue is quite important so please take a look. > > O

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-26 Thread Wes McKinney
I created a JIRA about this https://issues.apache.org/jira/browse/ARROW-9231 This issue is quite important so please take a look. On Thu, Jun 25, 2020 at 8:53 AM Wes McKinney wrote: > > On Thu, Jun 25, 2020 at 5:31 AM Antoine Pitrou wrote: > > > > > > Le 25/06/2020 à 12:18, Antoine Pitrou a éc

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-25 Thread Wes McKinney
On Thu, Jun 25, 2020 at 5:31 AM Antoine Pitrou wrote: > > > Le 25/06/2020 à 12:18, Antoine Pitrou a écrit : > > > > Le 25/06/2020 à 00:40, Wes McKinney a écrit : > >> hi folks, > >> > >> This has come up in some other contexts, but I believe it would be a > >> good idea to increment the version nu

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-25 Thread Antoine Pitrou
Le 25/06/2020 à 12:18, Antoine Pitrou a écrit : > > Le 25/06/2020 à 00:40, Wes McKinney a écrit : >> hi folks, >> >> This has come up in some other contexts, but I believe it would be a >> good idea to increment the version number in Schema.fbs starting with >> 1.0.0 to separate the pre-1.0 and

Re: [DISCUSS] Incrementing Arrow MetadataVersion from V4 to V5 for 1.0.0 release

2020-06-25 Thread Antoine Pitrou
Le 25/06/2020 à 00:40, Wes McKinney a écrit : > hi folks, > > This has come up in some other contexts, but I believe it would be a > good idea to increment the version number in Schema.fbs starting with > 1.0.0 to separate the pre-1.0 and post-1.0 worlds > > https://github.com/apache/arrow/blob