FYI, PR was open at https://github.com/apache/spark/pull/27459. Thanks Nicholas. Hope guys find some time to take a look.
2020년 1월 28일 (화) 오전 8:15, Nicholas Chammas <nicholas.cham...@gmail.com>님이 작성: > I am! Thanks for the reference. > > On Thu, Jan 16, 2020 at 9:53 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: > >> Nicholas, are you interested in taking a stab at this? You could refer >> https://github.com/apache/spark/commit/60472dbfd97acfd6c4420a13f9b32bc9d84219f3 >> >> 2020년 1월 17일 (금) 오전 8:48, Takeshi Yamamuro <linguin....@gmail.com>님이 작성: >> >>> The idea looks nice. I think web documents always help end users. >>> >>> Bests, >>> Takeshi >>> >>> On Fri, Jan 17, 2020 at 4:04 AM Shixiong(Ryan) Zhu < >>> shixi...@databricks.com> wrote: >>> >>>> "spark.sql("set -v")" returns a Dataset that has all non-internal SQL >>>> configurations. Should be pretty easy to automatically generate a SQL >>>> configuration page. >>>> >>>> Best Regards, >>>> Ryan >>>> >>>> >>>> On Wed, Jan 15, 2020 at 5:47 AM Hyukjin Kwon <gurwls...@gmail.com> >>>> wrote: >>>> >>>>> I think automatically creating a configuration page isn't a bad idea >>>>> because I think we deprecate and remove configurations which are not >>>>> created via .internal() in SQLConf anyway. >>>>> >>>>> I already tried this automatic generation from the codes at SQL >>>>> built-in functions and I'm pretty sure we can do the similar thing for >>>>> configurations as well. >>>>> >>>>> We could perhaps mimic what hadoop does >>>>> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/core-default.xml >>>>> >>>>> On Wed, 15 Jan 2020, 10:46 Sean Owen, <sro...@gmail.com> wrote: >>>>> >>>>>> Some of it is intentionally undocumented, as far as I know, as an >>>>>> experimental option that may change, or legacy, or safety valve flag. >>>>>> Certainly anything that's marked an internal conf. (That does raise >>>>>> the question of who it's for, if you have to read source to find it.) >>>>>> >>>>>> I don't know if we need to overhaul the conf system, but there may >>>>>> indeed be some confs that could legitimately be documented. I don't >>>>>> know which. >>>>>> >>>>>> On Tue, Jan 14, 2020 at 7:32 PM Nicholas Chammas >>>>>> <nicholas.cham...@gmail.com> wrote: >>>>>> > >>>>>> > I filed SPARK-30510 thinking that we had forgotten to document an >>>>>> option, but it turns out that there's a whole bunch of stuff under >>>>>> SQLConf.scala that has no public documentation under >>>>>> http://spark.apache.org/docs. >>>>>> > >>>>>> > Would it be appropriate to somehow automatically generate a >>>>>> documentation page from SQLConf.scala, as Hyukjin suggested on that >>>>>> ticket? >>>>>> > >>>>>> > Another thought that comes to mind is moving the config definitions >>>>>> out of Scala and into a data format like YAML or JSON, and then sourcing >>>>>> that both for SQLConf as well as for whatever documentation page we want >>>>>> to >>>>>> generate. What do you think of that idea? >>>>>> > >>>>>> > Nick >>>>>> > >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>> >>>>>> >>> >>> -- >>> --- >>> Takeshi Yamamuro >>> >>
