Re: Question about schema evolution in iceberg table

2019-02-21 Thread Ryan Blue
The PR to fix this is https://github.com/apache/incubator-iceberg/pull/108 I need to look into a couple of task failures, but could you validate that it works as you expect? Thanks! On Wed, Feb 20, 2019 at 1:04 PM suds wrote: > Thank you for looking into this issue. I was planning to debug iss

Re: Question about schema evolution in iceberg table

2019-02-20 Thread suds
Thank you for looking into this issue. I was planning to debug issue this week but looks like you already figured it out :) I will follow issue on github to know more about fix. -- Thanks On Wed, Feb 20, 2019 at 11:13 AM Ryan Blue wrote: > Sudsport, > > Good catch here, and thank you for the gi

Re: Question about schema evolution in iceberg table

2019-02-20 Thread Ryan Blue
Sudsport, Good catch here, and thank you for the gist that reproduces the issue. The problem happens when pushing predicates down to manifest files. Manifests keep track of the schema and partition spec that was used to write the manifest. The reader currently uses that schema when converting and

Re: Question about schema evolution in iceberg table

2019-02-15 Thread suds
Thanks for reply Ryan. I created gist with code example https://gist.github.com/sudssf/e5f2de7463487f98c0a269221bbe0f1a Please let me know if I am not using API correctly. On Thu, Feb 14, 2019 at 5:38 PM Ryan Blue wrote: > Sudsport, > > I'm wondering if you had the table cached somewhere? Th

Re: Question about schema evolution in iceberg table

2019-02-14 Thread Ryan Blue
Sudsport, I'm wondering if you had the table cached somewhere? Those renames should work. My guess is that the query used a table version that was out of date. Can you put together a minimal script that reproduces the error and open an issue? That way I can fix it. rb On Thu, Feb 14, 2019 at 3:

Re: Question about schema evolution in iceberg table

2019-02-14 Thread sudsport s
Adding dev@iceberg.apache.org On Thu, Feb 14, 2019 at 3:00 PM sudsport s wrote: > HI I am doing some testing with schema evolution. I looked at > testSchemaUpdate method and SchemaUpdate class for reference. > > > Here are steps I doing to test schema evolution validation > > initially data is