Are we planning to make sure that the tables with relative paths will always contain every data/metadata file in single root folder?
This depends on the use case. For table mirroring, there would be a back-end service making copies of all metadata, but not for other use cases. For example, the rename use case would have a service that is moving files, possibly deleting the original as it goes. Many decisions depend on external services and the components that are configured for a table. Whether to fall back to reading a different path depends on the use case and what you’re trying to do. If you’re mirroring tables across regions or clouds, then it’s unlikely you’d want an automatic fallback. But if you’re renaming a table and moving the files then you probably do. A location selector can select the initial path and FileIO should be able to handle the fallback strategy. So this is customizable. There are a lot of choices and what I think we are aiming for is to provide flexibility to get these tasks done. The current single location system is not very flexible. And I don’t think that using a single table location from the catalog level is flexible enough for these use cases, either. But the trade-off is making it well-defined enough that you can make some assumptions about file access. I think the default should be that we assume that all files exist in the location provided by a table location selector. That supports the DR use case. Anything more complicated could be plugged in through catalog customizations, with the understanding that if you change the assumption about whether a file exists, you’d need to update your FileIO to have a fallback strategy. On Sun, Aug 22, 2021 at 11:22 PM Peter Vary pv...@cloudera.com.invalid <http://mailto:pv...@cloudera.com.invalid> wrote: Here is a few scenarios which I am thinking about: > 1. Table T1 is created with relative path in HadoopCatalog/HiveCatalog > where the root location is L1, and this location is generated from the > TableIdentifier (at least in creation time) > 2. Data inserted to the table, so data files are created under L1, and the > metadata files contain R1 relative path to the current L1. > 3a. Table location is updated to L2 (Hive: ALTER TABLE T1 SET LOCATION L2) > 3b. Table renamed to T2 (Hive: ALTER TABLE T1 RENAME TO T2) - Table > location is updated for Hive tables if the old location was the default > location. > 4. When we try to read this table we should read the old data/metadata > files as well. > > So in both cases we have to move the old data/metadata files around like > Hive does for the native tables, and for the tables with relative paths we > do not have to change the metadata other than the root path? Will we do the > same thing with other engines as well? > >> I think this depends on how the engine handles it. If the engine doesn’t move files, then moving the location breaks the table. But the engine might move the files before returning from 3a. It may also choose to add a second location to the table and move the files in the background, using a custom FileIO that will fall back across table locations. This is a good example of how Iceberg is always dependent on what engines or services choose to do with it. There are other cases where engines or services can do bad things. For example, deleting a data file and adding it back later will cause Iceberg’s expireSnapshots operation to delete the data file because it doesn’t check whether the file is currently referenced (that’s really expensive, and what the action does). Ryan -- Ryan Blue Tabular
