I think an interesting exercise would be to consider what changes we are putting off for a major version and if they make enough of a change to warrent the work involved or keep pushing it off.
Personally the first thing that comes to mind is I'd like to revisit the accumulator APIs again and see if we can do something with then. What's top of everyone else's mind? On Jan 20, 2018 6:32 AM, "Sean Owen" <so...@cloudera.com> wrote: > Forking this thread to muse about Spark 3. Like Spark 2, I assume it would > be more about making all those accumulated breaking changes and updating > lots of dependencies. Hadoop 3 looms large in that list as well as Scala > 2.12. > > Spark 1 was release in May 2014, and Spark 2 in July 2016. If Spark 2.3 is > out in Feb 2018 and it takes the now-usual 6 months until a next release, > Spark 3 could reasonably be next. > > However the release cycles are naturally slowing down, and it could also > be said that 2019 would be more on schedule for Spark 3. > > Nothing particularly urgent about deciding, but I'm curious if anyone had > an opinion on whether to move on to Spark 3 next or just continue with 2.4 > later this year. > > On Fri, Jan 19, 2018 at 11:13 AM Sean Owen <so...@cloudera.com> wrote: > >> Yeah, if users are using Kryo directly, they should be insulated from a >> Spark-side change because of shading. >> However this also entails updating (unshaded) Chill from 0.8.x to 0.9.x. >> I am not sure if that causes problems for apps. >> >> Normally I'd avoid any major-version change in a minor release. This one >> looked potentially entirely internal. >> I think if there are any doubts, we can leave it for Spark 3. There was a >> bug report that needed a fix from Kryo 4, but it might be minor after all. >> >>> >>>
