Re: Dataset and Aggregator API pain points

2016-07-02 Thread Reynold Xin
Thanks, Koert, for the great email. They are all great points. We should probably create an umbrella JIRA for easier tracking. On Saturday, July 2, 2016, Koert Kuipers wrote: > after working with the Dataset and Aggregator apis for a few weeks porting > some fairly complex RDD algos (an overall

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Reynold Xin
Because in that case you cannot merge anything meant for 2.1 until 2.0 is released. On Saturday, July 2, 2016, Jacek Laskowski wrote: > Hi, > > Always release from master. What could be the gotchas? > > Pozdrawiam, > Jacek Laskowski > > https://medium.com/@jaceklaskowski/ > Mastering Apache

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Jacek Laskowski
Hi, Always release from master. What could be the gotchas? Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sat, Jul 2, 2016 at 11:36 PM, Sean Owen wrote: > I am n

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Sean Owen
I am not sure any other process makes sense. What are you suggesting should happen? On Sat, Jul 2, 2016, 22:27 Jacek Laskowski wrote: > Hi, > > Thanks Sean! It makes sense. > > I'm not fully convinced that's how it should be, so I apologize if I > ever ask about the version management in Spark a

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Jacek Laskowski
Hi, Thanks Sean! It makes sense. I'm not fully convinced that's how it should be, so I apologize if I ever ask about the version management in Spark again :) Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me

Dataset and Aggregator API pain points

2016-07-02 Thread Koert Kuipers
after working with the Dataset and Aggregator apis for a few weeks porting some fairly complex RDD algos (an overall pleasant experience) i wanted to summarize the pain points and some suggestions for improvement given my experience. all of these are already mentioned on mailing list or jira, but i

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Holden Karau
2.0.1 just means that the fix will be included in 2.0.1 (eg its not in the current 2.0.0 RC). On Saturday, July 2, 2016, Jacek Laskowski wrote: > Hi Sean, devs, > > How is this possible that Fix Version/s is 2.0.1 given 2.0.0 was not > released yet? Why is that that master is not what's going to

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Sean Owen
Because a 2.0.0 release candidate is out. If for some reason the release candidate becomes the 2.0.0 release, then anything merged to branch-2.0 after it is necessarily fixed in 2.0.1 at best. At this stage we know the RC1 will not be 2.0.0, so really that vote should be formally cancelled. Then we

Re: [jira] [Resolved] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them

2016-07-02 Thread Jacek Laskowski
Hi Sean, devs, How is this possible that Fix Version/s is 2.0.1 given 2.0.0 was not released yet? Why is that that master is not what's going to be released so eventually becomes 2.0.0? I don't get it. Appreciate any guidance. Thanks. Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklask

Re: branch-2.0 is now 2.0.1-SNAPSHOT?

2016-07-02 Thread Koert Kuipers
that helps, now i know i simply need to look at master On Sat, Jul 2, 2016 at 1:37 PM, Sean Owen wrote: > So, on the one hand I think branch-2.0 should really still be on > 2.0.0-SNAPSHOT but is on 2.0.1-SNAPSHOT, and while master should > technically be on 2.1.0-SNAPSHOT but we can't quite beca

Can't build scala unidoc since Kafka 0.10 support was added

2016-07-02 Thread Michael Allman
Hello, I'm no longer able to successfully run `sbt unidoc` in branch-2.0, and the problem seems to stem from the addition of Kafka 0.10 support. If I remove either the Kafka 0.8 or 0.10 projects from the build then unidoc works. If I keep both in I get two dozen inexplicable compilation errors

Re: branch-2.0 is now 2.0.1-SNAPSHOT?

2016-07-02 Thread Sean Owen
So, on the one hand I think branch-2.0 should really still be on 2.0.0-SNAPSHOT but is on 2.0.1-SNAPSHOT, and while master should technically be on 2.1.0-SNAPSHOT but we can't quite because of MiMa right now, I do see that both snapshots are being produced still: https://repository.apache.org/cont

Re: branch-2.0 is now 2.0.1-SNAPSHOT?

2016-07-02 Thread Koert Kuipers
You do, snapshots for spark 2.0.0-SNAPSHOT are updated daily on the apache snapshot repo. I use them in our own unit tests to find regressions etc. in spark and report them back On Jul 2, 2016 3:35 AM, "Sean Owen" wrote: > Yeah, interesting question about whether it should be 2.0.1-SNAPSHOT > at

Re: branch-2.0 is now 2.0.1-SNAPSHOT?

2016-07-02 Thread Sean Owen
Yeah, interesting question about whether it should be 2.0.1-SNAPSHOT at this stage because 2.0.0 is not yet released. But I'm not sure we publish snapshots anyway? On Sat, Jul 2, 2016 at 5:41 AM, Koert Kuipers wrote: > is that correct? > where do i get the latest 2.0.0-SNAPSHOT? > thanks, > koert