Re: [DISCUSS] Format changes: process and requirements

2019-04-02 Thread Wes McKinney
I created https://cwiki.apache.org/confluence/display/ARROW/Columnar+Format+1.0+Milestone some time ago to try to track the status of different implementations and various in-flight discussions about columnar format evolution. Can some others take a look at that and perhaps update some sections?

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Paul Taylor
Hi Jacques, I think we should have two complete implementations. I don't think having one feature in C# and Go and another in JavaScript and Rust does justice to the project goals. Agree 100%. We may already be in this situation with the DictionaryBatch "isDelta" flag. I haven't checked the C

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Micah Kornfield
I think this makes sense, and it is a good clarification on process. It might be a good idea to also give a preliminary vote on an existing backlog of JIRAs so people don't waste time starting PRs if they won't be supported (or supported via a custom metadata type). I've included a list below doi

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Jacques Nadeau
> > How about "at least two native implementations" instead of > "Java and C++"? Now, we have multiple native > implementations: > I think we should have two complete implementations. I don't think having one feature in C# and Go and another in JavaScript and Rust does justice to the project goals

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Kouhei Sutou
t So, we can choose at least two of them to run integration tests. Thanks, -- kou In "[DISCUSS] Format changes: process and requirements" on Sun, 17 Mar 2019 14:18:35 -0700, Jacques Nadeau wrote: > I want to bring up a concern I have with the recent changes to the format.

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Wes McKinney
hi Jacques, I agree with your reasoning. I think it's important to go through the implementation exercise to make sure we have the details right and everyone is on board with adding something new (or changing or deprecating something existing). I also agree with formalizing the voting process with

[DISCUSS] Format changes: process and requirements

2019-03-17 Thread Jacques Nadeau
I want to bring up a concern I have with the recent changes to the format. To me, part of the strength of the project is that you have multiple bindings for Arrow the are cross-compatible and consistent. As such, I'd like to propose the following process when dealing with format changes: - Form