Hi Shone, Iceberg should be able to handle out of order data columns in nested structures. We probably just need to relax that compatibility check to allow it. Can you post the error message that you're getting?
On Sun, Sep 22, 2019 at 4:49 AM Shone Sadler <ssad...@adobe.com.invalid> wrote: > Hello everyone, > > This question is related to schema evolution support in Iceberg. > > We have data coming in with fields out-of-order wrt to the schema in > Iceberg (e.g. inbound struct(a,b,c) vs. iceberg struct(c,b,a)) > > As a result we are hitting the following error in Iceberg when saving the > data -> "Cannot write incompatible dataset to table with schema", > generated within the IcebergeSource -> > https://github.com/apache/incubator-iceberg/blob/d1f0b540f5f14f002be86133ef9f66445f7e0926/spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java#L157 > > I also noted in the documentation that re-ordering was allowed -> > https://iceberg.apache.org/evolution/ , which led me to believe that we > could update the schema prior to writing the data, However, I see no means > of re-ordering fields on the current UpdateSchema API. > > How are people handling out-of-order fields today? > > Our data is deeply nested, as a result I am hoping not to have to > transform/prep on ingest and looking for alternatives. > > Any thoughts appreciated! > > Regards, > Shone Sadler > > > > -- Ryan Blue Software Engineer Netflix
