Hi Jean, One use case is Hive to Iceberg migration, where DROP PARTITION does not need to change to DELETE queries prior to the migration.
That said, I am not in favor of adding this to Iceberg directly (or Iceberg-Spark) due to the reasons Jean mentioned. It might be possible to do it in a custom extension or custom connector outside Iceberg that is specific for the use case (e.g., the migration use case I mentioned above). Further, as Szhehon said, it would not make sense without ADD PARTITION. However, ADD PARTITION requires a spec change (since Iceberg does not support empty partitions but ADD PARTITION does). So overall I am -1 to DROP PARTITION in Iceberg default implementation, and I think it is better to consider implementing in a use case specific implementation. Thanks, Walaa. On Wed, Jul 17, 2024 at 12:34 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Gabor > > Do you have user requests for that ? As Iceberg produces partitions by > taking column values (optionally with a transform function). So the > hidden partitioning doesn't require user actions. I wonder the use > cases for dynamic partitioning (using ADD/DROP). Is it more for > partition maintenance ? > > Thanks ! > Regards > JB > > On Wed, Jul 17, 2024 at 11:11 AM Gabor Kaszab <gaborkas...@apache.org> > wrote: > > > > Hey Community, > > > > I learned recently that Spark doesn't support DROP PARTITION for Iceberg > tables. I understand this is because the DROP PARTITION is something being > used for Hive tables and Iceberg's model for hidden partitioning makes it > unnatural to have commands like this. > > > > However, I think that DROP PARTITION would still have some value for > users. In fact in Impala we implemented this even for Iceberg tables. > Benefits could be: > > - Users having workloads on Hive tables could use their workloads after > they migrated their tables to Iceberg. > > - Opposed to DELETE FROM, DROP PARTITION has a guarantee that this is > going to be a metadata only operation and no delete files are going to be > written. > > > > I'm curious what the community thinks of this. > > Gabor > > >
