Hi Eduard, Thank you for getting back to me! I filed this issue https://github.com/apache/iceberg/issues/9164. Please let me know if you need any other information. This is all I can think of for now. I am also interested in fixing this issue, so let me know if you have any thoughts on how we should fix it. Thank you! Best, Ha
From: Eduard Tudenhoefner <edu...@tabular.io> Sent: Thursday, November 23, 2023 3:12 AM To: dev@iceberg.apache.org Subject: Re: Question about CatalogUtil:dropTableData Hello Ha, This might have been an unintended oversight that we can improve and fix. Could you please open an issue with any further details that you have? Eduard On Wed, Nov 22, 2023 at 7:37 PM Ha Cao <ha....@twosigma.com<mailto:ha....@twosigma.com>> wrote: Hello, I am using the CatalogUtil::dropTableData<https://secure-web.cisco.com/1_4kdC7xYHsfDLgVB1jNDY6OWb2KIINgV_hzNVIiMTyZnZJW61Bds70UtGN8DTkcCpG5ueCQLl8B4Q5M9-d5bNHia2o-Mk64BTw9vjZRDLM1PoQk_drlRh_m7Ribh5JPx-9CSzl-Rxx0qClTKBXSOHwVTFuqzxdQQG427WVZZ54L9jqtChe-0j7-Ftxmchbu-3O-OgvvMgfUJrJPCxKleNhDKjePcMtee57Z2YNpE8XysN_Y3sB1XOAQmpsaUuYufd64GHZnFsUuUcfFjOocLWg/https%3A%2F%2Fsourcegraph.com%2Fgithub.com%2Fapache%2Ficeberg%40e8cf33db7d3fc637504a51a801c055dce54474b7%2F-%2Fblob%2Fcore%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Ficeberg%2FCatalogUtil.java%3FL86> to delete data and metadata files for my tables after I drop them from the catalog. I notice that if the program terminates unexpectedly (e.g. VM shutdown, connection exception, etc.) midway through dropTableData, if it successfully deletes the manifest files (.avro files), but fails somewhere after that (it hasn’t got to delete the metadata.json file), on a retry, CatalogUtil::dropTableData will fail at line 94 – Iterables.addAll(manifestsToDelete, snapshot.allManifests(io))<https://secure-web.cisco.com/13otzNrQ1-b65unaj_skra3DRTCrxSkGjqphQULSmageFknCV354-Xw2Ed-4QO4U_a6tZgw07CyTRCx1u16k-pOcM8773UGQT9ivPviRkXBMjlnjLLcETccDV9a3oyMUvCCr9b7nPNkgDl2t58pIg3IxXrcAKxyNQ8oMvnIYosP4q7IJAEFQzLDlan-KeTSXPZLBeGt0bpj63r9ATwzgoa1xKTNdpsuXrKU41iZk76gbjSueBjjhisx1wCqZdSxd4Cw-pba28YbiqBkIV8uSHfg/https%3A%2F%2Fsourcegraph.com%2Fgithub.com%2Fapache%2Ficeberg%40e8cf33db7d3fc637504a51a801c055dce54474b7%2F-%2Fblob%2Fcore%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Ficeberg%2FCatalogUtil.java%3FL94> (specifically because of snapshot.allManifests) with awssdk NoSuchKeyException because the manifest file (.avro file) is no longer there. I wonder why in the current implementation, we prefer to let it throw an exception at this line, instead of catching it, adding a WARN log or something like that, and letting it continue to delete other files afterwards. Thank you very much! Happy Thanksgiving, Ha
