Re: [Spark Docs]: Question about building docs including docs of internal packages
Hello, by default, some packages (that are treated as internal) are excluded from documentation generation task. To generate Javadoc/Scaladoc for classes from them, you would need to comment relevant line in build definition file. For example. package `org/apache/spark/sql/execution` is mentioned here - https://github.com/apache/spark/blob/master/project/SparkBuild.scala#L857 Once the package is commented, `unidoc` task would generate documentation from it. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Scala 3 support approach
Hi all! I'd like to ask for an opinion and discuss the next thing: at this moment in general Spark could be built with Scala 2.11 and 2.12 (mostly), and close to the point to have support for Scala 2.13. On the other hand, Scala 3 is going into the pre-release phase (with 3.0.0-M1 released at the beginning of October). Previously, support of the current Scala version by Spark was a bit behind of desired state, dictated by all circumstances. To move things differently with Scala 3 I'd like to contribute my efforts (and help others if there would be any) to support it starting as soon as possible (ie to have Spark build compiled with Scala 3 and to have release artifacts when it would be possible). I suggest that it would require to add an experimental profile to the build file so further changes to compile, test and run other tasks could be done in incremental manner (with respect to compatibility with current code for versions 2.12 and 2.13 and backporting where possible). I'd like to do it that way since I do not represent any company, contribute in my own time and thus cannot guarantee consistent time spent on this (so just in case of anything such contribution would not be left in the fork repo). In fact, with recent changes to move Spark build to use the latest SBT, such starting changes are pretty small on the SBT side (about 10 LOC) and I was already able to see how build fails with Scala 3 compiler :) To summarize: 1. Is this approach suitable for the project at this moment, so it would be accepted and accounted for in the release schedule (in 2021 I assume)? 2. how should it be filed, as an umbrella Jira ticket with minor tasks or as a SPIP at first with more thorough analysis?
Re: Scala 3 support approach
Sorry for the noise. Please reply to this one. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Scala 3 support approach
Thanks for the input, Sean. > Spark depends on a number of Scala libraries, so needs them all to support > version X before Spark can. This only happened for 2.13 about 4-5 months > ago. I wonder if even a fraction of the necessary libraries have 3.0 > support yet? As far as I understand, here shows off one of the differences, we could use Scala 2.13 binary dependencies in Scala 3 project: it was expected that all the 3d-party libraries would lag behind compiler and standard library for significant amount of time, so it is "retro-compatible" in their terms [1][2]. And that's why I was able to see actual compilation errors using all dependency set from Spark with Scala 2.13 [1] https://contributors.scala-lang.org/t/scala-2-to-3-transition-some-updates-from-the-scala-center/4013 [2] https://scalacenter.github.io/scala-3-migration-guide/docs/compatibility.html#a-scala-3-module-depending-on-a-scala-2-artifact I believe this changes the perspective a bit: that only code changes are left to consider on how much to divert from current state of the sources and is it possible to maintain it in the same source tree. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
