Re: [DISCUSS] Format changes: process and requirements

2019-04-02 Thread Wes McKinney
I created https://cwiki.apache.org/confluence/display/ARROW/Columnar+Format+1.0+Milestone some time ago to try to track the status of different implementations and various in-flight discussions about columnar format evolution. Can some others take a look at that and perhaps update some sections?

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Paul Taylor
Hi Jacques, I think we should have two complete implementations. I don't think having one feature in C# and Go and another in JavaScript and Rust does justice to the project goals. Agree 100%. We may already be in this situation with the DictionaryBatch "isDelta" flag. I haven't checked the C

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Micah Kornfield
I think this makes sense, and it is a good clarification on process. It might be a good idea to also give a preliminary vote on an existing backlog of JIRAs so people don't waste time starting PRs if they won't be supported (or supported via a custom metadata type). I've included a list below doi

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Jacques Nadeau
> > How about "at least two native implementations" instead of > "Java and C++"? Now, we have multiple native > implementations: > I think we should have two complete implementations. I don't think having one feature in C# and Go and another in JavaScript and Rust does justice to the project goals

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Kouhei Sutou
Hi, Basically, I agree with the proposal. > reference implementation/support in Java and C++ How about "at least two native implementations" instead of "Java and C++"? Now, we have multiple native implementations: * C++ * C# * Go * Java * JavaScript * Rust So, we can choose at leas

Re: [DISCUSS] Format changes: process and requirements

2019-03-17 Thread Wes McKinney
hi Jacques, I agree with your reasoning. I think it's important to go through the implementation exercise to make sure we have the details right and everyone is on board with adding something new (or changing or deprecating something existing). I also agree with formalizing the voting process with