Mostly avro today, but ORC also has union types. I'll create an issue for this later today.
Thanks, R. On Wed, May 15, 2019 at 11:16 AM Ryan Blue <rb...@netflix.com.invalid> wrote: > Are you talking about Avro data? I think Parquet data would work fine > because unions are represented as a struct of optionals. > > I think this makes sense. Maybe we could also allow projecting the > contents of unions by representing them as structs of optionals and > materializing them that way. I'd be up for reviewing this. > > On Mon, May 13, 2019 at 1:48 PM RD <rdsr...@gmail.com> wrote: > >> To add to this, I'm not suggesting to change Iceberg writers to support >> writing non-optional unions. The motivation for this is to support legacy >> datasets [not written by Iceberg]. >> >> On Mon, May 13, 2019 at 10:05 AM RD <rdsr...@gmail.com> wrote: >> >>> Iceberg today does not support non optional unions and that is the right >>> behaviour, but we do have a lot of datasets which have non-optional union >>> fields. I'm wondering whether Iceberg should allow reading these datasets >>> as long as the user does not project the union field. >>> >>> I tried it out and today is throws an exception during column pruning. >>> >>> If we think this should be support, I'll create an issue for this. >>> >>> -Best, >>> R. >>> >> > > -- > Ryan Blue > Software Engineer > Netflix >
