Re: Use a Spark schema util method to figure out new delta nested fields

2019-06-18 Thread Ryan Blue
I think it makes sense that you might want to do this. I'd be happy to review a PR with the new methods. Thanks! On Fri, Jun 14, 2019 at 8:24 AM Filip wrote: > Hi Iceberg devs, > > I was thinking about a general use-case of having data arriving with a > partial schema than that of a dataset. By

Use a Spark schema util method to figure out new delta nested fields

2019-06-14 Thread Filip
Hi Iceberg devs, I was thinking about a general use-case of having data arriving with a partial schema than that of a dataset. By having that dataset backed by an Iceberg table we need to make sure that before each file is committed we at least add the new columns to the Iceberg schema first. I w