A few replies inline. On Thu, Nov 26, 2020 at 3:49 AM Peter Vary <pv...@cloudera.com> wrote:
> I think the column mapping should also be 1-to-1. Hive would have trouble > writing to a table if it didn't include all required columns. I think that > the right thing is for all engines to provide uniform access to all columns. > > > I got your point. I think that Hive would be able to write to the table > without problem if the given columns are optional (but that is only > focusing on a small part of the problem) > > Would this mean that we have to create a mapping for every iceberg type > currently not supported by Hive (uuid/time/fixed), and implement something > like a conversion to and from a string or binary form of the data, so every > Iceberg column is accessible? > Related question: How often do we introduce new types to Iceberg? > In Spark, we convert UUID to and from String, and we validate the length of binary data written to a fixed column. I think that time currently throws an exception (probably not correct). I think it is either reasonable to ignore a column or to do some minimal work to use a compatible representation like for UUID and fixed. Introducing a new type would require an update to the spec, so I expect that to be very infrequent. > Last, Iceberg must be the source of truth for schema information. Schema > in the Hive metastore should always come from Iceberg, or else columns > would be inaccessible and Hive would require extra DDL to access new > columns. Hive should have the same view of an Iceberg table that any other > engine would. > > > This will mean that every access for a given table will generate an extra > fetch from HSM + S3 and then we have to parse the snapshot files. > We had similar problem with a partitioned Avro table where the tables > schema was stored in the Avro file itself. A simple getPartitions call > floored the system because of the multiple parallel IOs. > I suspect that this was a problem with Avro because Hive allows partition-level customization that Iceberg does not. The current schema for an Iceberg table applies to all data, and would require only reading the current metadata file. There would be no operation at partition granularity to cause trouble. > Adding to this that Hive creates SerDes often multiple times for the > queries, we have to find a way to cache the schema to speed things up. > Yes, we may need to add a caching layer. I think our recent discussion about how to interact with catalogs would help quite a bit here. We already have a caching catalog that avoids loading a table multiple times. Using that would avoid multiple independent reads of the metadata file. -- Ryan Blue Software Engineer Netflix
