On Sun, Aug 3, 2014 at 9:09 PM, Patrick Wendell <pwend...@gmail.com> wrote:
> Hey Anand, > > Thanks for looking into this - it's great to see momentum towards Scala > 2.11 and I'd love if this land in Spark 1.2. > > For the external dependencies, it would be good to create a sub-task of > SPARK-1812 to track our efforts encouraging other projects to upgrade. In > certain cases (e.g. Kafka) there is fairly late-stage work on this already, > so we can e.g. link to those JIRA's as well. A good starting point is to > just go to their dev list and ask what the status is, most Scala projects > have put at least some thought into this already. Another thing we can do > is submit patches ourselves to those projects to help get them upgraded. > The twitter libraries, e.g., tend to be pretty small and also open to > external contributions. > > One other thing in the mix here - Prashant Sharma has also spent some time > looking at this, so it might be good for you two to connect (probably off > list) and sync up. Prashant has contributed to many Scala projects, so he > might have cycles to go and help some of our dependencies get upgraded - > but I won't commit to that on his behalf :). > > Regarding Akka - I shaded and published akka as a one-off thing: > https://github.com/pwendell/akka/tree/2.2.3-shaded-proto > > Over time we've had to publish our own versions of a small number of > dependencies. It's somewhat high overhead, but it actually works quite well > in terms of avoiding some of the nastier dependency conflicts. At least > better than other alternatives I've seen such as using a shader build > plug-in. > > Going forward, I'd actually like to track these in the Spark repo itself. > For instance, we have a bash script in the spark repo that can e.g. check > out akka, apply a few patches or regular expressions, and then you have a > fully shaded dependency that can be published to maven. If you wanted to > take a crack at something like that for akka 2.3.4, be my guest. I can help > with the actual publishing. > Will give it a try, thanks!
