Re: [Result] [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-12-01 Thread Ji Liu
ation changes. > > Thanks, > Micah > > On Tue, Nov 26, 2019 at 12:50 AM Sutou Kouhei wrote: > >> +1 (binding) >> >> In >> "[VOTE] Clarifications and forward compatibility changes for Dictionary >> Encoding (second iteration)" on Wed, 20

Re: [Result] [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-12-01 Thread Micah Kornfield
s, > Micah > > On Tue, Nov 26, 2019 at 12:50 AM Sutou Kouhei wrote: > >> +1 (binding) >> >> In >> "[VOTE] Clarifications and forward compatibility changes for Dictionary >> Encoding (second iteration)" on Wed, 20 Nov 2019 20:41:57 -08

[Result] [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-26 Thread Micah Kornfield
12:50 AM Sutou Kouhei wrote: > +1 (binding) > > In > "[VOTE] Clarifications and forward compatibility changes for Dictionary > Encoding (second iteration)" on Wed, 20 Nov 2019 20:41:57 -0800, > Micah Kornfield wrote: > > > Hello, > > As discussed o

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-26 Thread Sutou Kouhei
+1 (binding) In "[VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)" on Wed, 20 Nov 2019 20:41:57 -0800, Micah Kornfield wrote: > Hello, > As discussed on [1], I've proposed clarifications in a PR [2] that > clari

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-25 Thread Ji Liu
github.com/apache/arrow/pull/4960 -- From:Ji Liu Send Time:2019年11月26日(星期二) 14:04 To:dev ; Micah Kornfield Cc:Wes McKinney Subject:Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-25 Thread Ji Liu
+1 (non-binding) Thanks Ji Liu -- From:Fan Liya Send Time:2019年11月26日(星期二) 14:01 To:dev ; Micah Kornfield Cc:Wes McKinney Subject:Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-25 Thread Fan Liya
I am sorry I did not follow the thread closely (will follow up later). However, the proposal above looks good to me. So I am +0.5 for this. Best, Liya Fan On Tue, Nov 26, 2019 at 1:12 PM Micah Kornfield wrote: > Could other members of the community chime in on this? In particular > getting vie

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-25 Thread Micah Kornfield
Could other members of the community chime in on this? In particular getting views from other language maintainers would be good. Thanks, Micah On Thu, Nov 21, 2019 at 12:23 PM Micah Kornfield wrote: > Forgot to say, My vote is +1 (binding). > > On Thu, Nov 21, 2019 at 12:09 PM Wes McKinney

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-21 Thread Micah Kornfield
Forgot to say, My vote is +1 (binding). On Thu, Nov 21, 2019 at 12:09 PM Wes McKinney wrote: > +1 (binding). Thanks Micah > > On Wed, Nov 20, 2019 at 10:42 PM Micah Kornfield > wrote: > > > > Hello, > > As discussed on [1], I've proposed clarifications in a PR [2] that > > clarifies: > > > > 1

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-21 Thread Wes McKinney
+1 (binding). Thanks Micah On Wed, Nov 20, 2019 at 10:42 PM Micah Kornfield wrote: > > Hello, > As discussed on [1], I've proposed clarifications in a PR [2] that > clarifies: > > 1. It is not required that all dictionary batches occur at the beginning > of the IPC stream format (if a the first

[VOTE] Clarifications and forward compatibility changes for Dictionary Encoding (second iteration)

2019-11-20 Thread Micah Kornfield
Hello, As discussed on [1], I've proposed clarifications in a PR [2] that clarifies: 1. It is not required that all dictionary batches occur at the beginning of the IPC stream format (if a the first record batch has an all null dictionary encoded column, the null column's dictionary might not be

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding

2019-10-30 Thread Wes McKinney
I wrote in on the original DISCUSS thread. I believe Antoine is unavailable this week, but hopefully we can drive the discussion to a consensus point next week so we can vote On Sat, Oct 26, 2019 at 12:01 AM Micah Kornfield wrote: > > I think at least the wording was confusing because you raised

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding

2019-10-25 Thread Micah Kornfield
I think at least the wording was confusing because you raised questions on the PR and Antoine commented here. I agree with your analysis that it probably would not be hard to support. But don't feel too strongly either way on this particular point, aside from coming to a resolution. If I had to

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding

2019-10-25 Thread Wes McKinney
Can we discuss the delta dictionary issue a bit more? I admit I don't share that same concerns. >From the perspective of a file and stream producer, the code paths should be nearly identical. The differences with the file format are: * Magic numbers to detect that it is the "file format" * Accumu

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding

2019-10-24 Thread Micah Kornfield
Hi Antoine, There is a defined order for dictionaries in metadata. What isn't well defined is relative ordering between record batches and Delta dictionaries. However, this point seems confusing. I can't think of a real-world use case we're it would be valuable enough to include, so I will remov

Re: [VOTE] Clarifications and forward compatibility changes for Dictionary Encoding

2019-10-24 Thread Antoine Pitrou
Le 24/10/2019 à 04:39, Micah Kornfield a écrit : > > 3. Clarifies that the file format, can only contain 1 "NON-delta" > dictionary batch and multiple "delta" dictionary batches. This is a bit weird. If the file format can carry delta dictionaries, it means order is significant, so it may as

[VOTE] Clarifications and forward compatibility changes for Dictionary Encoding

2019-10-23 Thread Micah Kornfield
Hello, As discussed on [1], I've proposed clarifications in a PR [2] that clarifies: 1. It is not required that all dictionary batches occur at the beginning of the IPC stream format (if a the first record batch has an all null dictionary encoded column, the null column's dictionary might not be