Thank you for looking into this issue. I was planning to debug issue this week but looks like you already figured it out :) I will follow issue on github to know more about fix.
-- Thanks On Wed, Feb 20, 2019 at 11:13 AM Ryan Blue <rb...@netflix.com> wrote: > Sudsport, > > Good catch here, and thank you for the gist that reproduces the issue. > > The problem happens when pushing predicates down to manifest files. > Manifests keep track of the schema and partition spec that was used to > write the manifest. The reader currently uses that schema when converting > and binding predicates to evaluate on the partition data in the manifest. > So this is a bug where we haven't passed the current table schema down to > the manifest reader. > > I'll open an issue for it and fix this. Thanks! > > rb > > On Fri, Feb 15, 2019 at 11:34 AM suds <sudssf2...@gmail.com> wrote: > >> Thanks for reply Ryan. >> >> I created gist with code example >> >> https://gist.github.com/sudssf/e5f2de7463487f98c0a269221bbe0f1a >> >> Please let me know if I am not using API correctly. >> >> >> On Thu, Feb 14, 2019 at 5:38 PM Ryan Blue <rb...@netflix.com> wrote: >> >>> Sudsport, >>> >>> I'm wondering if you had the table cached somewhere? Those renames >>> should work. My guess is that the query used a table version that was out >>> of date. >>> >>> Can you put together a minimal script that reproduces the error and open >>> an issue? That way I can fix it. >>> >>> rb >>> >>> On Thu, Feb 14, 2019 at 3:01 PM sudsport s <sudssf2...@gmail.com> wrote: >>> >>>> Adding dev@iceberg.apache.org >>>> >>>> >>>> On Thu, Feb 14, 2019 at 3:00 PM sudsport s <sudssf2...@gmail.com> >>>> wrote: >>>> >>>>> HI I am doing some testing with schema evolution. I looked at >>>>> testSchemaUpdate method and SchemaUpdate class for reference. >>>>> >>>>> >>>>> Here are steps I doing to test schema evolution validation >>>>> >>>>> initially data is created with following schema using "key" as >>>>> partition key >>>>> >>>>> root >>>>> |-- id: string (nullable = true) >>>>> |-- value: string (nullable = true) >>>>> |-- key: integer (nullable = false) >>>>> |-- value1: string (nullable = true) >>>>> |-- value2: string (nullable = true) >>>>> >>>>> schema update to rename value1 -> v1 >>>>> >>>>> root >>>>> |-- id: string (nullable = true) >>>>> |-- value: string (nullable = true) >>>>> |-- key: integer (nullable = false) >>>>> |-- v1: string (nullable = true) >>>>> |-- value2: string (nullable = true) >>>>> >>>>> schema update to rename key -> newKey ( I know changing partition key >>>>> is not good idea but this is a test :) ) >>>>> >>>>> root >>>>> |-- id: string (nullable = true) >>>>> |-- value: string (nullable = true) >>>>> |-- newKey: integer (nullable = false) >>>>> |-- v1: string (nullable = true) >>>>> |-- value2: string (nullable = true) >>>>> >>>>> >>>>> when I read data frame using spark I get following schema >>>>> >>>>> root >>>>> |-- id: string (nullable = true) >>>>> |-- value: string (nullable = true) >>>>> |-- newKey: integer (nullable = false) >>>>> |-- v1: string (nullable = true) >>>>> |-- value2: string (nullable = true) >>>>> >>>>> but when I try to run query or scan using changed column in where >>>>> clause I get following exception >>>>> >>>>> >>>>> INFO TableScan: Scanning table /tmp/schema-evolution snapshot >>>>> 1550184572006 created at 2019-02-14 14:49:32.189 with filter >>>>> not_null(ref(name="v1")) >>>>> Exception in thread "main" >>>>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, >>>>> tree: >>>>> Exchange SinglePartition >>>>> +- *(1) HashAggregate(keys=[], functions=[partial_count(1)], >>>>> output=[count#77L]) >>>>> +- *(1) Project >>>>> +- *(1) Filter (isnotnull(v1#60) && (cast(v1#60 as int) = 0)) >>>>> +- *(1) DataSourceV2Scan [v1#60], >>>>> IcebergScan(table=/tmp/schema-evolution, type=struct<4: v1: optional >>>>> string>, filters=[not_null(ref(name="v1"))]) >>>>> >>>>> at >>>>> org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56) >>>>> at >>>>> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.doExecute(ShuffleExchangeExec.scala:119) >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) >>>>> >>>>> Caused by: com.netflix.iceberg.exceptions.ValidationException: Cannot >>>>> find field 'v1' in struct: struct<1: id: optional string, 2: value: >>>>> optional string, 3: key: required int, 4: value1: optional string, 5: >>>>> value2: optional string> >>>>> at >>>>> com.netflix.iceberg.exceptions.ValidationException.check(ValidationException.java:39) >>>>> at >>>>> com.netflix.iceberg.expressions.UnboundPredicate.bind(UnboundPredicate.java:46) >>>>> >>>>> >>>>> I ran same query using where various combinations "v1 = 0" , "value1 = >>>>> 0" , "key = 0" and "newKey = 0" >>>>> >>>>> What is best way to query data in iceberg table when schema is changed? >>>>> >>>>> >>>>> following output from metadata json >>>>> >>>>> >>>>> < "name" : "key", >>>>> --- >>>>> > "name" : "newKey", >>>>> 25c25 >>>>> < "name" : "value1", >>>>> --- >>>>> > "name" : "v1", >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Iceberg Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to iceberg-devel+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to iceberg-de...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/iceberg-devel/3efe985e-2302-412b-a899-8efe1fbf13c8%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/iceberg-devel/3efe985e-2302-412b-a899-8efe1fbf13c8%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Iceberg Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to iceberg-devel+unsubscr...@googlegroups.com. >>>> To post to this group, send email to iceberg-de...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/iceberg-devel/CAO32DPxrri4Oz%2BuX6vwgdh3NhW5FgxEmTumRrba5N6M6Rkuy5Q%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/iceberg-devel/CAO32DPxrri4Oz%2BuX6vwgdh3NhW5FgxEmTumRrba5N6M6Rkuy5Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Iceberg Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to iceberg-devel+unsubscr...@googlegroups.com. >> To post to this group, send email to iceberg-de...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/iceberg-devel/CAO32DPy3pnDY8qohaVjRFyLsEnT-bdkcHYX0X9dgW5dKpuoW8w%40mail.gmail.com >> <https://groups.google.com/d/msgid/iceberg-devel/CAO32DPy3pnDY8qohaVjRFyLsEnT-bdkcHYX0X9dgW5dKpuoW8w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Ryan Blue > Software Engineer > Netflix >
