Thanks for the reply. We have this requirement as we may want to include some very specific types that are not generic at all. Say if we use Iceberg's data apis to read and write data files directly, all I think is needed is to be able to specify the new types in metadata. Annotate an existing type with a higher custom type (in form of annotation), as long as there is a base type (something like byte array maybe?) we can rely on.
Chen On Fri, Sep 11, 2020 at 1:35 PM Ryan Blue <[email protected]> wrote: > Sorry I missed this email! > > Right now, we don't support extending Iceberg's type system. We are > currently targeting a small set of types so that engines can easily > implement them. And we've already hit some issues with the types that > Iceberg supports: Spark, for example, doesn't support a time type, and very > few engines support the UUID type. > > That said, I think we may be able to support some form of logical types if > all you need to do is to annotate that a certain value has a higher-level > type. Like annotating that an Integer is actually an IP address. We've not > thought about this very much, but it would be great to hear proposals. > > rb > > On Fri, Sep 11, 2020 at 10:30 AM Chen Song <[email protected]> wrote: > >> Any thoughts, or suggestions on this? >> >> On Tue, Sep 8, 2020 at 3:01 PM Chen Song <[email protected]> wrote: >> >>> Hi >>> >>> I have a general question on Iceberg's data type system. Iceberg has a >>> well defined type spec >>> <https://iceberg.apache.org/spec/#schemas-and-data-types> which can be >>> mapped to types in Avro, Parquet, ORC. If users want to use Iceberg and >>> extend the universe of data types (e.g., adding custom types on top of >>> existing types), is it possible to do this w/o changing Iceberg core >>> modules? >>> >>> Chen >>> >> >> >> -- >> Chen Song >> >> > > -- > Ryan Blue > Software Engineer > Netflix > -- Chen Song
