I think automatically creating a configuration page isn't a bad idea because I think we deprecate and remove configurations which are not created via .internal() in SQLConf anyway.
I already tried this automatic generation from the codes at SQL built-in functions and I'm pretty sure we can do the similar thing for configurations as well. We could perhaps mimic what hadoop does https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/core-default.xml On Wed, 15 Jan 2020, 10:46 Sean Owen, <sro...@gmail.com> wrote: > Some of it is intentionally undocumented, as far as I know, as an > experimental option that may change, or legacy, or safety valve flag. > Certainly anything that's marked an internal conf. (That does raise > the question of who it's for, if you have to read source to find it.) > > I don't know if we need to overhaul the conf system, but there may > indeed be some confs that could legitimately be documented. I don't > know which. > > On Tue, Jan 14, 2020 at 7:32 PM Nicholas Chammas > <nicholas.cham...@gmail.com> wrote: > > > > I filed SPARK-30510 thinking that we had forgotten to document an > option, but it turns out that there's a whole bunch of stuff under > SQLConf.scala that has no public documentation under > http://spark.apache.org/docs. > > > > Would it be appropriate to somehow automatically generate a > documentation page from SQLConf.scala, as Hyukjin suggested on that ticket? > > > > Another thought that comes to mind is moving the config definitions out > of Scala and into a data format like YAML or JSON, and then sourcing that > both for SQLConf as well as for whatever documentation page we want to > generate. What do you think of that idea? > > > > Nick > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
