Hi Walaa It makes sense, thanks for pointing the use case.
I agree that it's better to consider a use-case specific impl. Regards JB On Wed, Jul 17, 2024 at 11:36 PM Walaa Eldin Moustafa <wa.moust...@gmail.com> wrote: > > Hi Jean, One use case is Hive to Iceberg migration, where DROP PARTITION does > not need to change to DELETE queries prior to the migration. > > That said, I am not in favor of adding this to Iceberg directly (or > Iceberg-Spark) due to the reasons Jean mentioned. It might be possible to do > it in a custom extension or custom connector outside Iceberg that is specific > for the use case (e.g., the migration use case I mentioned above). > > Further, as Szhehon said, it would not make sense without ADD PARTITION. > However, ADD PARTITION requires a spec change (since Iceberg does not support > empty partitions but ADD PARTITION does). > > So overall I am -1 to DROP PARTITION in Iceberg default implementation, and I > think it is better to consider implementing in a use case specific > implementation. > > Thanks, > Walaa. > > > On Wed, Jul 17, 2024 at 12:34 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: >> >> Hi Gabor >> >> Do you have user requests for that ? As Iceberg produces partitions by >> taking column values (optionally with a transform function). So the >> hidden partitioning doesn't require user actions. I wonder the use >> cases for dynamic partitioning (using ADD/DROP). Is it more for >> partition maintenance ? >> >> Thanks ! >> Regards >> JB >> >> On Wed, Jul 17, 2024 at 11:11 AM Gabor Kaszab <gaborkas...@apache.org> wrote: >> > >> > Hey Community, >> > >> > I learned recently that Spark doesn't support DROP PARTITION for Iceberg >> > tables. I understand this is because the DROP PARTITION is something being >> > used for Hive tables and Iceberg's model for hidden partitioning makes it >> > unnatural to have commands like this. >> > >> > However, I think that DROP PARTITION would still have some value for >> > users. In fact in Impala we implemented this even for Iceberg tables. >> > Benefits could be: >> > - Users having workloads on Hive tables could use their workloads after >> > they migrated their tables to Iceberg. >> > - Opposed to DELETE FROM, DROP PARTITION has a guarantee that this is >> > going to be a metadata only operation and no delete files are going to be >> > written. >> > >> > I'm curious what the community thinks of this. >> > Gabor >> >
