Re: Iceberg's type system

2020-09-11 Thread Chen Song
Thanks for the reply. We have this requirement as we may want to include some very specific types that are not generic at all. Say if we use Iceberg's data apis to read and write data files directly, all I think is needed is to be able to specify the new types in metadata. Annotate an existing typ

Re: Iceberg's type system

2020-09-11 Thread Ryan Blue
Sorry I missed this email! Right now, we don't support extending Iceberg's type system. We are currently targeting a small set of types so that engines can easily implement them. And we've already hit some issues with the types that Iceberg supports: Spark, for example, doesn't support a time type

Re: Iceberg's type system

2020-09-11 Thread Chen Song
Any thoughts, or suggestions on this? On Tue, Sep 8, 2020 at 3:01 PM Chen Song wrote: > Hi > > I have a general question on Iceberg's data type system. Iceberg has a > well defined type spec > which can be > mapped to types in Avro, Parqu