Thanks for your help + suggestions, Peter. Thanks for the pointer to Nessie, Jacques.
To wrap up the thread: I took an afternoon to put together a Postgres-backed metastore to make sure I understood all the moving pieces and have some confidence that I know what would go into the migration no matter what we choose (HMS vs. something custom vs. something else), just an internal question as to what we prefer now. Building a custom metastore was pleasantly pain-free — thanks to everyone involved in documenting the process + building things to have points of extension. Marko On Mon, Nov 23, 2020 at 12:43 AM Peter Vary <pv...@cloudera.com.invalid> wrote: > Hi Marco, > > See my comments below: > > On Nov 20, 2020, at 19:58, Marko Babic <ma...@narrative.io.INVALID> wrote: > > Hi Peter. Thanks for responding. > > > The command you mention below: `CREATE EXTERNAL TABLE` above an existing > Iceberg table will not transfer the "responsibility" of tracking the > snapshot to HMS. It only creates a HMS external table ... > > So my understanding is that the HiveCatalog is basically just using HMS as > an atomically updateable pointer to a metadata file (excepting recent work > to make Iceberg tables queryable _from_ Hive which we won't be doing) > > > If you really do not want to read the tables from Hive and do not have an > already working HMS then Jacque's suggestion would make some sense to use > Nessie or another more lightweight custom catalog implementation. Maturity > should be considered as a plus for HMS, but probably you can get away > without introducing another component to your architecture. > > so what I'm doing with that command is mimicking the DDL for a > HiveCatalog-created table, which sets up Iceberg tables as external > tables in HMS > <https://github.com/apache/iceberg/blob/f8c68ebcb4e35db5d7f5ccb8e20d53df3abdf8b1/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L230-L292>, > and > manually updating the `metadata_location` table property to point to latest > metadata file for the existing table that I want to integrate with HMS. > Updating the metadata pointer, along with obviously updating all > readers/writers to load the table via the HiveCatalog, seems to be all I > need to do to make that work but I'm just naively dipping my toes in here > and could absolutely be missing something. E.g. I figured out I'd have to > rename the latest metadata file from the existing table I want to integrate > with HMS so that BaseMetastoreTableOperations could parse the version > number, but only realized later that I'd have to rename _all_ the old > metadata files + twiddle the metadata log entries to use the updated names. > > > I would try to avoid renaming files, because then you get yourself into > the mess when you are updating all the path information in the metadata > files, which with the snapshot/manifest hierarchy might cause headaches. > AFAIK there is no requirements for the naming of the snapshot files for > the HiveCatalog as it is using UUID to make sure the name of the snapshot > files are unique. If I know correctly the v1/v2 etc requirement is only for > HadoopCatalog and HadoopTables. > So you can try just manually updating the metadata location. This could > work, but I do not know of any official tool for this. > > > > What I would do is this: ... > > Makes sense, the external table creation + metadata pointer mangling is my > attempt to do basically this but I'm not confident I know everything that > needs to go into making step 2 happen. :) > > The following is what I'm thinking: > > - Given an existing Hadoop table on top of S3 at s3://old_table/, create > a new table with the same schema + partition spec via HiveCatalog. > - Parse metadata files from the old table and update them to be > HadoopCatalog-compatible: all I'd be updating is metadata file names + the > metadata log as described above. > - Write updated metadata files to s3://old_table/metadata/. Update new > table in HMS to point to latest, updated metadata file and update the table > location to point to s3://old_table/. > > I could alternatively `aws s3 sync` data files from the old table to the > new one, rewrite all the old metadata + snapshot manifest lists + manifest > files to point to the new data directory, and leave s3://old_table/ > untouched, but I guess that's a decision I'd make once I'm into things and > have a better sense of what'd be less error-prone. > > Thanks again! > > Marko > > > On Fri, Nov 20, 2020 at 12:39 AM Peter Vary <pv...@cloudera.com.invalid> > wrote: > >> Hi Marko, >> >> The command you mention below: `CREATE EXTERNAL TABLE` above an existing >> Iceberg table will not transfer the "responsibility" of tracking the >> snapshot to HMS. It only creates a HMS external table which will allow Hive >> queries to read the given table. If you want to track the snapshot in the >> HMS then you have to originally create a table in HMS using HiveCatalog. >> >> What I would do is this: >> >> 1. Create a new Iceberg table in a catalog which supports concurrent >> writes (Hive/Hadoop/Custom) >> 2. Migrate the tables to the new catalog. Maybe there are some >> already existing tools there, or with some java/spark code the snapshot >> files can be read and rewritten. By my understanding you definitely do not >> have to rewrite the data files, just the snapshot files (and maybe the >> manifest files) >> >> Hope this helps, >> Peter >> >> >> On Nov 19, 2020, at 21:29, John Clara <john.anthony.cl...@gmail.com> >> wrote: >> >> Hi, >> >> My team has been using the custom catalog along with atomic metadata >> updates but we never migrated existing iceberg tables onto it. We also >> haven't turned on integration with the hive catalog, so I'm not sure how >> >> easy it is to plug in there (I think there was some recent work on >> that?). Dynamo provides a local mock which you could combine with s3mock >> (check iceberg tests) to test it out: >> https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html >> >> >> Only weird things we've run into with dynamo is: >> 1. it seems like we get rate limited by dynamo pretty hard when first >> writing to a new table until rate limits are adjusted (potentially by aws >> dynamically dynamo's internal partitions?) >> 2. make sure to page scans if you have a lot of values when doing lists >> (we haven't enabled catalog listing yet, but we've ran into this before) >> >> We chose dynamo because we were using it for other usecases. I'm not sure >> if it's the best aws provided option for atomic changes. >> >> John >> >> On 11/19/20 10:07 AM, Marko Babic wrote: >> >> Hi everyone, >> >> At my org we’ve spun up a few Iceberg tables on top of S3 without a >> metastore (conscious of the consequences) and we’ve arrived at the point >> that we need to support concurrent writes. :) I was hoping to get some >> advice as to what the best way to integrate an existing Iceberg table into >> a Hive Metastore or an alternative might be. We’re still relatively early >> in our adoption of Iceberg and have no real prior experience with Hive so I >> don’t know what I don’t know. >> >> Some options we’re weighing: >> >> - Existing tables aren’t so big that the moral equivalent of "CREATE >> TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the >> question, but we’d prefer to not have to read + rewrite everything. We also >> have stateful readers (tracking which snapshots they have previously read) >> and preserving table history would make life easier. >> >> - Doing something along the lines of the following and importing the >> tables into Hive as external tables looks like it should work given my >> understanding of how Iceberg is using HMS, but I don’t know if it’s >> encouraged and I haven’t done diligence to understand potential >> consequences: >> >> ``` >> hive> CREATE EXTERNAL TABLE `existing_table` (...) >> LOCATION >> 's3://existing-table/' >> -- serde, input/output formats omitted >> TBLPROPERTIES ( >> -- Assuming latest metadata file for Hadoop table is >> v99.metadata.json, rename it to 00099-uuid.metadata.json >> -- so that BaseMetastoreTableOperations can correctly parse the >> version number. >> 'metadata_location'=' >> s3://existing-table/metadata/00099-uuid.metadata.json', >> 'table_type'='ICEBERG' >> ) >> ``` >> >> - Others seem to have had success implementing + maintaining a custom >> catalog (https://iceberg.apache.org/custom-catalog/ < >> https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB for >> atomic metadata updates, which could appeal to us. Seems like migration in >> this case consists of implementing the catalog and plopping the latest >> metadata into the backing store. Are custom catalogs more of an escape >> hatch when HMS can’t be used, or would that maybe be a reasonable way >> forward if we find we don’t want to maintain + operate on top of HMS? >> >> Apologies if this was discussed or documented somewhere else and I’ve >> missed it. >> >> Thanks! >> >> Marko >> >> >> >
