I think that the DataFrame write API needs improvement and I'm looking forward to getting DataSourceV2 to be complete and reliable, but neither of those require breaking changes in the short term.
For both, I think we should develop in parallel until they are mature enough to replace what we use currently, and only then make a breaking change to remove the old code. For me, that means aiming for a breaking release some time in 2019. rb On Fri, Jan 19, 2018 at 9:43 AM, Holden Karau <holden.ka...@gmail.com> wrote: > I think an interesting exercise would be to consider what changes we are > putting off for a major version and if they make enough of a change to > warrent the work involved or keep pushing it off. > > Personally the first thing that comes to mind is I'd like to revisit the > accumulator APIs again and see if we can do something with then. What's top > of everyone else's mind? > > On Jan 20, 2018 6:32 AM, "Sean Owen" <so...@cloudera.com> wrote: > >> Forking this thread to muse about Spark 3. Like Spark 2, I assume it >> would be more about making all those accumulated breaking changes and >> updating lots of dependencies. Hadoop 3 looms large in that list as well as >> Scala 2.12. >> >> Spark 1 was release in May 2014, and Spark 2 in July 2016. If Spark 2.3 >> is out in Feb 2018 and it takes the now-usual 6 months until a next >> release, Spark 3 could reasonably be next. >> >> However the release cycles are naturally slowing down, and it could also >> be said that 2019 would be more on schedule for Spark 3. >> >> Nothing particularly urgent about deciding, but I'm curious if anyone had >> an opinion on whether to move on to Spark 3 next or just continue with 2.4 >> later this year. >> >> On Fri, Jan 19, 2018 at 11:13 AM Sean Owen <so...@cloudera.com> wrote: >> >>> Yeah, if users are using Kryo directly, they should be insulated from a >>> Spark-side change because of shading. >>> However this also entails updating (unshaded) Chill from 0.8.x to 0.9.x. >>> I am not sure if that causes problems for apps. >>> >>> Normally I'd avoid any major-version change in a minor release. This one >>> looked potentially entirely internal. >>> I think if there are any doubts, we can leave it for Spark 3. There was >>> a bug report that needed a fix from Kryo 4, but it might be minor after all. >>> >>>> >>>> -- Ryan Blue Software Engineer Netflix
