Hi Team,

With Shardul we had a longer discussion yesterday about the schema 
synchronization between Iceberg and Hive, and we thought that it would be good 
to ask the opinion of the greater community too.

We can have 2 sources for the schemas. 
Hive table definition / schema
Iceberg schema.

If we want Iceberg and Hive to work together we have to find a way to 
synchronize them. Either by defining a master schema, or by defining a 
compatibility matrix and conversion for them.
In previous Hive integrations we can see examples for both:
With Avro there is a possibility to read the schema from the data file 
directly, and the master schema is the one which is in Avro.
With HBase you can provide a mapping between HBase columns by providing the 
hbase.columns.mapping table property

Maybe the differences are caused by how the storage format is perceived Avro 
being a simple storage format, HBase being an independent query engine - but 
his is just a questionable opinion :)

I would like us to decide how Iceberg - Hive integration should be handled.

There are at least 2 questions:
How flexible we should be with the type mapping between Hive and Iceberg types?
Shall we have a strict mapping - This way if we have an Iceberg schema we can 
immediately derive the Hive schema from it.
Shall we be more relaxed on this - Automatic casting / conversions can be built 
into the integration, allowing the users to skip view and/or UDF creation for 
typical conversions
How flexible we should be with column mappings?
Shall we have strict 1-on-1 mapping - This way if we have an Iceberg schema we 
can immediately derive the Hive schema from it. We still have to omit Iceberg 
columns which does not have a representation available in Hive.
Shall we allow flexibility on Hive table creation to chose specific Iceberg 
columns instead of immediately creating a Hive table with all of the columns 
from the Iceberg table

Currently I would chose:
Strict type mapping because of the following reasons:
Faster execution (we want as few checks and conversions as possible, since it 
will be executed for every record)
Complexity exponentially increases with every conversion
Flexible column mapping:
I think it will be a typical situation when we have a huge Iceberg table 
storing the facts with big number of columns and we would like to create 
multiple Hive tables above that. The problem could be solved by creating the 
table and adding a view above that table, but I think it would be more 
user-friendly if we could avoid this extra step.
The added complexity is at table creation / query planning which has far 
smaller impact on the overall performance

I would love to hear your thoughts as well since the choice should really 
depend on the user base, and what are the expected use-cases.

Thanks,
Peter


Appendix 1 - Type mapping proposal:
Iceberg type    Hive2 type      Hive3 type      Status
boolean BOOLEAN BOOLEAN OK
int     INTEGER INTEGER OK
long    BIGINT  BIGINT  OK
float   FLOAT   FLOAT   OK
double  DOUBLE  DOUBLE  OK
decimal(P,S)    DECIMAL(P,S)    DECIMAL(P,S)    OK
binary  BINARY  BINARY  OK
date    DATE    DATE    OK
timestamp       TIMESTAMP       TIMESTAMP       OK
timestamptz     TIMESTAMP       TIMESTAMP WITH LOCAL TIMEZONE   TODO
string  STRING  STRING  OK
uuid    STRING or BINARY        STRING or BINARY        TODO
time    -       -       -
fixed(L)        -       -       -
-       TINYINT TINYINT -
-       SMALLINT        SMALLINT        -
-       INTERVAL        INTERVAL        -
-       VARCHAR VARCHAR -
-       CHAR    CHAR    -

Reply via email to