Hi Szehon, Thanks for the update. Can you please point me to the work on supporting DELETE/UPDATE/MERGE in the DataFrame API? Thanks, Wing Yew
On Tue, Jul 9, 2024 at 10:05 PM Szehon Ho <szehon.apa...@gmail.com> wrote: > Hi, > > Just FYI, good news, this change is merged on the Spark side : > https://github.com/apache/spark/pull/46707 (its the third effort!). In > next version of Spark, we will be able to pass read properties via SQL to a > particular Iceberg table such as > > SELECT * FROM iceberg.db.table1 WITH (`locality` = `true`) > > I will look at write options after this. > > There's also progress in supporting DELETE/UPDATE/MERGE from Dataframes as > well, it should also be coming soon in Spark. > > Thanks, > Szehon > > > > On Wed, Jul 26, 2023 at 12:46 PM Wing Yew Poon <wyp...@cloudera.com.invalid> > wrote: > >> We are talking about DELETE/UPDATE/MERGE operations. There is only SQL >> support for these operations. There is no DataFrame API support for them.* >> Therefore write options are not applicable. Thus SQLConf is the only >> available mechanism I can use to override the table property. >> For reference, we currently support setting distribution mode using write >> option, SQLConf and table property. It seems to me that >> https://github.com/apache/iceberg/pull/6838/ is a precedent for what I'd >> like to do. >> >> * It would be of interest to support performing DELETE/UPDATE/MERGE from >> DataFrames, but that is a whole other topic. >> >> >> On Wed, Jul 26, 2023 at 12:04 PM Ryan Blue <b...@tabular.io> wrote: >> >>> I think we should aim to have the same behavior across properties that >>> are set in SQL conf, table config, and write options. Having SQL conf >>> override table config for this doesn't make sense to me. If the need is to >>> override table configuration, then write options are the right way to do it. >>> >>> On Wed, Jul 26, 2023 at 10:10 AM Wing Yew Poon >>> <wyp...@cloudera.com.invalid> wrote: >>> >>>> I was on vacation. >>>> Currently, write modes (copy-on-write/merge-on-read) can only be set as >>>> table properties, and default to copy-on-write. We have a customer who >>>> wants to use copy-on-write for certain Spark jobs that write to some >>>> Iceberg table and merge-on-read for other Spark jobs writing to the same >>>> table, because of the write characteristics of those jobs. This seems like >>>> a use case that should be supported. The only way they can do this >>>> currently is to toggle the table property as needed before doing the >>>> writes. This is not a sustainable workaround. >>>> Hence, I think it would be useful to be able to configure the write >>>> mode as a SQLConf. I also disagree that the table property should always >>>> win. If this is the case, there is no way to override it. The existing >>>> behavior in SparkConfParser is to use the option if set, else use the >>>> session conf if set, else use the table property. This applies across the >>>> board. >>>> - Wing Yew >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Jul 16, 2023 at 4:48 PM Ryan Blue <b...@tabular.io> wrote: >>>> >>>>> Yes, I agree that there is value for administrators from having some >>>>> things exposed as Spark SQL configuration. That gets much harder when you >>>>> want to use the SQLConf for table-level settings, though. For example, the >>>>> target split size is something that was an engine setting in the Hadoop >>>>> world, even though it makes no sense to use the same setting across vastly >>>>> different tables --- think about joining a fact table with a dimension >>>>> table. >>>>> >>>>> Settings like write mode are table-level settings. It matters what is >>>>> downstream of the table. You may want to set a *default* write mode, but >>>>> the table-level setting should always win. Currently, there are limits to >>>>> overriding the write mode in SQL. That's why we should add hints. For >>>>> anything beyond that, I think we need to discuss what you're trying to do. >>>>> If it's to override a table-level setting with a SQL global, then we >>>>> should >>>>> understand the use case better. >>>>> >>>>> On Fri, Jul 14, 2023 at 6:09 PM Wing Yew Poon >>>>> <wyp...@cloudera.com.invalid> wrote: >>>>> >>>>>> Also, in the case of write mode (I mean write.delete.mode, >>>>>> write.update.mode, write.merge.mode), these cannot be set as options >>>>>> currently; they are only settable as table properties. >>>>>> >>>>>> On Fri, Jul 14, 2023 at 5:58 PM Wing Yew Poon <wyp...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> I think that different use cases benefit from or even require >>>>>>> different solutions. I think enabling options in Spark SQL is helpful, >>>>>>> but >>>>>>> allowing some configurations to be done in SQLConf is also helpful. >>>>>>> For Cheng Pan's use case (to disable locality), I think providing a >>>>>>> conf (which can be added to spark-defaults.conf by a cluster admin) is >>>>>>> useful. >>>>>>> For my customer's use case ( >>>>>>> https://github.com/apache/iceberg/pull/7790), being able to set the >>>>>>> write mode per Spark job (where right now it can only be set as a table >>>>>>> property) is useful. Allowing this to be done in the SQL with an >>>>>>> option/hint could also work, but as I understand it, Szehon's PR ( >>>>>>> https://github.com/apache/spark/pull/416830) is only applicable to >>>>>>> reads, not writes. >>>>>>> >>>>>>> - Wing Yew >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 13, 2023 at 1:04 AM Cheng Pan <pan3...@gmail.com> wrote: >>>>>>> >>>>>>>> Ryan, I understand that option should be job-specific, and >>>>>>>> introducing an OPTIONS HINT can make Spark SQL achieves similar >>>>>>>> capabilities as DataFrame API does. >>>>>>>> >>>>>>>> My point is, some of the Iceberg options should not be job-specific. >>>>>>>> >>>>>>>> For example, Iceberg has an option “locality” which only allows >>>>>>>> setting at the job level, but Spark has a configuration >>>>>>>> “spark.shuffle.reduceLocality.enabled” which allows setting at the >>>>>>>> cluster >>>>>>>> level, this is a gap block Spark administers migrate to Iceberg because >>>>>>>> they can not disable it at the cluster level. >>>>>>>> >>>>>>>> So, what’s the principle in the Iceberg of classifying a >>>>>>>> configuration into SQLConf or OPTION? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Cheng Pan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> > On Jul 5, 2023, at 16:26, Cheng Pan <pan3...@gmail.com> wrote: >>>>>>>> > >>>>>>>> > I would argue that the SQLConf way is more in line with Spark >>>>>>>> user/administrator habits. >>>>>>>> > >>>>>>>> > It’s a common practice that Spark administrators set >>>>>>>> configurations in spark-defaults.conf at the cluster level , and when >>>>>>>> the >>>>>>>> user has issues with their Spark SQL/Jobs, the first question they >>>>>>>> asked >>>>>>>> mostly is: can it be fixed by adding a spark configuration? >>>>>>>> > >>>>>>>> > The OPTIONS way brings additional learning efforts to Spark users >>>>>>>> and how can Spark administrators set them at cluster level? >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Cheng Pan >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> >> On Jun 17, 2023, at 04:01, Wing Yew Poon >>>>>>>> <wyp...@cloudera.com.INVALID> wrote: >>>>>>>> >> >>>>>>>> >> Hi, >>>>>>>> >> I recently put up a PR, >>>>>>>> https://github.com/apache/iceberg/pull/7790, to allow the write >>>>>>>> mode (copy-on-write/merge-on-read) to be specified in SQLConf. The use >>>>>>>> case >>>>>>>> is explained in the PR. >>>>>>>> >> Cheng Pan has an open PR, >>>>>>>> https://github.com/apache/iceberg/pull/7733, to allow locality to >>>>>>>> be specified in SQLConf. >>>>>>>> >> In the recent past, https://github.com/apache/iceberg/pull/6838/ >>>>>>>> was a PR to allow the write distribution mode to be specified in >>>>>>>> SQLConf. >>>>>>>> This was merged. >>>>>>>> >> Cheng Pan asks if there is any guidance on when we should allow >>>>>>>> configs to be specified in SQLConf. >>>>>>>> >> Thanks, >>>>>>>> >> Wing Yew >>>>>>>> >> >>>>>>>> >> ps. The above open PRs could use reviews by committers. >>>>>>>> >> >>>>>>>> > >>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> Ryan Blue >>>>> Tabular >>>>> >>>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >>
