Thanks Russell, Truncate suggested by you 
https://spark.apache.org/docs/3.3.0/sql-ref-syntax-ddl-truncate-table.html#content
 
<https://spark.apache.org/docs/3.3.0/sql-ref-syntax-ddl-truncate-table.html#content>
 help generate a new empty snapshot which enable "previously last" snapshot to 
be expired. I think it works for me. Appreciate your pointers

Thanks,
Steve Zhang



> On Jun 29, 2022, at 4:32 PM, Russell Spitzer <russell.spit...@gmail.com> 
> wrote:
> 
> Is "truncate" not an option? This would do a table wide delete which would 
> create a new snapshot which you can keep. No data files would be valid after 
> this?
> 
> On Wed, Jun 29, 2022 at 6:29 PM Steve Zhang <hongyue_zh...@apple.com.invalid> 
> wrote:
> Hey Iceberg Community:
> 
> I am wondering if there’s any best practice to handle residual of data files 
> deleted from last snapshot in the iceberg table. 
> 
> Let me explain the use case here, considering the data retention policy in 
> place where some of the sensitive data can only be stored on disk for a 
> month. In iceberg way to keep the data off the disk, we need to generally 
> complete it in 3 steps
> 1. delete data from the table, or drop partition (logical deletion)
> 2. expire old snapshots (physical deletion to get data off the disk)
> 3. remove orphaned files (not needed, but at scale this might be needed to 
> account for any failure in 2nd steps)
> 
> However, from what I can tell, the iceberg expire-snapshot stored procedure 
> will not delete the last snapshot of the given table as stated in 
> https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L141-L146
>  
> <https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L141-L146>.
>  
> 
> So if the last snapshot happen to be the delete in step 1, and if there’s no 
> more transaction happen to the table, then the snapshot will not be expired 
> properly and leave the data files behind. I am not sure what’s the right way 
> to clean up the data files from the disk to comply with our retention policy. 
> Can anyone share some ideas? 
> 
> I guess drop table is one workaround but I am looking for less intrusive way 
> to leave the table as is, like its original state right after table creation, 
> before any data is written.
> 
> Thanks,
> Steve Zhang
> 
> 
> 

Reply via email to