On Mon, Aug 4, 2014 at 1:01 PM, Anand Avati <av...@gluster.org> wrote:
> > > > On Sun, Aug 3, 2014 at 9:09 PM, Patrick Wendell <pwend...@gmail.com> > wrote: > >> Hey Anand, >> >> Thanks for looking into this - it's great to see momentum towards Scala >> 2.11 and I'd love if this land in Spark 1.2. >> >> For the external dependencies, it would be good to create a sub-task of >> SPARK-1812 to track our efforts encouraging other projects to upgrade. In >> certain cases (e.g. Kafka) there is fairly late-stage work on this already, >> so we can e.g. link to those JIRA's as well. A good starting point is to >> just go to their dev list and ask what the status is, most Scala projects >> have put at least some thought into this already. Another thing we can do >> is submit patches ourselves to those projects to help get them upgraded. >> The twitter libraries, e.g., tend to be pretty small and also open to >> external contributions. >> >> One other thing in the mix here - Prashant Sharma has also spent some >> time looking at this, so it might be good for you two to connect (probably >> off list) and sync up. Prashant has contributed to many Scala projects, so >> he might have cycles to go and help some of our dependencies get upgraded - >> but I won't commit to that on his behalf :). >> >> Regarding Akka - I shaded and published akka as a one-off thing: >> https://github.com/pwendell/akka/tree/2.2.3-shaded-proto >> >> Over time we've had to publish our own versions of a small number of >> dependencies. It's somewhat high overhead, but it actually works quite well >> in terms of avoiding some of the nastier dependency conflicts. At least >> better than other alternatives I've seen such as using a shader build >> plug-in. >> >> Going forward, I'd actually like to track these in the Spark repo itself. >> For instance, we have a bash script in the spark repo that can e.g. check >> out akka, apply a few patches or regular expressions, and then you have a >> fully shaded dependency that can be published to maven. If you wanted to >> take a crack at something like that for akka 2.3.4, be my guest. I can help >> with the actual publishing. >> > > Will give it a try, thanks! > > Patrick, I have a set of scripts at https://github.com/avati/spark-shaded which generate shaded artifacts for akka 2.3.4 and transitive dependencies under org.spark-project group for both Scala 2.10 and 2.11 in your local maven repo ($HOME/.m2/...) Publishing these artifacts is necessary for https://github.com/apache/spark/pull/1685. Once merged, bumping up Scala version to 2.11 will be smooth w.r.t akka (verified it on my system) If you think it will be useful, I can submit a PR of those scripts into spark.git/extras or somewhere. Thanks
