I think it’s a good idea ________________________________ From: Hyukjin Kwon <gurwls...@gmail.com> Sent: Wednesday, January 15, 2020 5:49:12 AM To: dev <dev@spark.apache.org> Cc: Sean Owen <sro...@gmail.com>; Nicholas Chammas <nicholas.cham...@gmail.com> Subject: Re: More publicly documenting the options under spark.sql.*
Resending to the dev list for archive purpose: I think automatically creating a configuration page isn't a bad idea because I think we deprecate and remove configurations which are not created via .internal() in SQLConf anyway. I already tried this automatic generation from the codes at SQL built-in functions and I'm pretty sure we can do the similar thing for configurations as well. We could perhaps mimic what hadoop does https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/core-default.xml On Wed, 15 Jan 2020, 22:46 Hyukjin Kwon, <gurwls...@gmail.com<mailto:gurwls...@gmail.com>> wrote: I think automatically creating a configuration page isn't a bad idea because I think we deprecate and remove configurations which are not created via .internal() in SQLConf anyway. I already tried this automatic generation from the codes at SQL built-in functions and I'm pretty sure we can do the similar thing for configurations as well. We could perhaps mimic what hadoop does https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/core-default.xml On Wed, 15 Jan 2020, 10:46 Sean Owen, <sro...@gmail.com<mailto:sro...@gmail.com>> wrote: Some of it is intentionally undocumented, as far as I know, as an experimental option that may change, or legacy, or safety valve flag. Certainly anything that's marked an internal conf. (That does raise the question of who it's for, if you have to read source to find it.) I don't know if we need to overhaul the conf system, but there may indeed be some confs that could legitimately be documented. I don't know which. On Tue, Jan 14, 2020 at 7:32 PM Nicholas Chammas <nicholas.cham...@gmail.com<mailto:nicholas.cham...@gmail.com>> wrote: > > I filed SPARK-30510 thinking that we had forgotten to document an option, but > it turns out that there's a whole bunch of stuff under SQLConf.scala that has > no public documentation under http://spark.apache.org/docs. > > Would it be appropriate to somehow automatically generate a documentation > page from SQLConf.scala, as Hyukjin suggested on that ticket? > > Another thought that comes to mind is moving the config definitions out of > Scala and into a data format like YAML or JSON, and then sourcing that both > for SQLConf as well as for whatever documentation page we want to generate. > What do you think of that idea? > > Nick > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org>
