I think it’s difficult for Iceberg to support the partition-related commands because it ends up being a scan over metadata files rather than a metastore operation. We have been trying to move away from our own getPartitions API because it is expensive to satisfy those queries compared to Hive because you have to find all matching files and then aggregate the data.
I think a separate sensor for Airflow sounds like the right option because that could use Iceberg metadata to find just the partitions that changed. That’s a much more efficient operation because you’re basically consuming snapshot metadata incrementally after some high watermark. You may also want to think about different processing patterns in Airflow. Part of what we want to do in Iceberg is to hide partitioning so that it is configuration that can be changed. Using partitions for implicit deletes (like INSERT OVERWRITE in Hive) or for units of work ties infrastructure to partitioning and makes it so you can’t evolve it without breaking something. I would like to avoid that. That’s why we built the new v2 write API in Spark that overwrites using an explicit filter rather than implicitly replacing partitions. Here, I would recommend an alternative sensor that identifies when a portion of a table is complete and then produces a filter to select that portion. That way you don’t depend directly on partitioning. On Tue, Mar 2, 2021 at 10:21 AM Gustavo Torres Torres <gustavo.tor...@airbnb.com.invalid> wrote: > Thanks Peter! > > So in that case we do let users create iceberg tables with Hive DDL > `CREATE EXTERNAL TABLE ice_table PARTITIONED BY ...` but my guess is that > `SHOW PARTITIONS ice_table` would not work. > > Has there been any discussion about whether Iceberg tables in Hive should > support these partition-related commands? I know Spark and Trino (ex. > PrestoSQL) have metadata tables that would let you find out the partitions > of your table. > > On Tue, Mar 2, 2021 at 3:35 AM Peter Vary <pv...@cloudera.com.invalid> > wrote: > >> Hi Gustavo, >> >> Not too familiar with the Airflow user base/use cases, but we had to >> consider similar things when decided what to do with `CREATE EXTERNAL TABLE >> ice_table PARTITIONED BY ...` Hive queries. >> See: https://github.com/apache/iceberg/pull/1917 >> >> The decision there was, that even thought the user issued a command to >> create a partitioned Hive table, we created an unpartitioned Hive table, >> where the backing Iceberg table was using identity partitions for the >> originally requested columns. >> >> Hope this helps a bit. >> >> Thanks, >> Peter >> >> On Mar 2, 2021, at 03:38, Gustavo Torres Torres < >> gustavo.tor...@airbnb.com.INVALID> wrote: >> >> Hey folks, >> >> Lately I've been thinking about integration between Airflow & Iceberg for >> a smooth transition from Hive-based tables to Iceberg ones and would like >> to hear about your experience. Specifically about Iceberg partition sensors >> in Airflow. >> >> From the way I see it, there are two ways to go about this (at least for >> Hive-based catalogs): >> >> >> 1. Modify our Hive Metastore API so that partitions-APIs are handled >> directly by the Iceberg API. This has the advantage of being mostly >> transparent to users but has the downside of being confusing since Iceberg >> creates tables with the Hive catalog as external non-partitioned tables. >> 2. Create a separate sensor that makes it clear that we are sensing >> over an Iceberg table. This is probably the most straightforward approach, >> but if we do this we would probably need to do the same for any tool that >> used the metastore to get partition information. >> >> >> Would love to hear what your experiences have been. >> Thanks >> >> >> -- Ryan Blue Software Engineer Netflix
