Also +1 for OPTIONS hints. It is useful to allow some options to be specified in SQLConf.
On Thu, Jul 6, 2023 at 1:05 AM Ryan Blue <b...@tabular.io> wrote: > Cheng, that's true of certain options that are targeted at administrators. > But the DataFrameReader or DataFrameWriter options are job-specific, which > is why a hint makes the most sense. > > On Wed, Jul 5, 2023 at 1:26 AM Cheng Pan <pan3...@gmail.com> wrote: > >> I would argue that the SQLConf way is more in line with Spark >> user/administrator habits. >> >> It’s a common practice that Spark administrators set configurations in >> spark-defaults.conf at the cluster level , and when the user has issues >> with their Spark SQL/Jobs, the first question they asked mostly is: can it >> be fixed by adding a spark configuration? >> >> The OPTIONS way brings additional learning efforts to Spark users and how >> can Spark administrators set them at cluster level? >> >> Thanks, >> Cheng Pan >> >> >> >> >> > On Jun 17, 2023, at 04:01, Wing Yew Poon <wyp...@cloudera.com.INVALID> >> wrote: >> > >> > Hi, >> > I recently put up a PR, https://github.com/apache/iceberg/pull/7790, >> to allow the write mode (copy-on-write/merge-on-read) to be specified in >> SQLConf. The use case is explained in the PR. >> > Cheng Pan has an open PR, https://github.com/apache/iceberg/pull/7733, >> to allow locality to be specified in SQLConf. >> > In the recent past, https://github.com/apache/iceberg/pull/6838/ was a >> PR to allow the write distribution mode to be specified in SQLConf. This >> was merged. >> > Cheng Pan asks if there is any guidance on when we should allow configs >> to be specified in SQLConf. >> > Thanks, >> > Wing Yew >> > >> > ps. The above open PRs could use reviews by committers. >> > >> >> > > -- > Ryan Blue > Tabular >
