Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-19 Thread Stephan Ewen
Per offline discussion with Robert: He or I will get onto this, probably next week. On Thu, Feb 19, 2015 at 10:07 AM, Till Rohrmann wrote: > Dependency conflicts were also the reason why we have to use a different > Akka version for the Hadoop 2.0.0-alpha build profile. > > Thus, +1. > > On Wed,

Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-19 Thread Till Rohrmann
Dependency conflicts were also the reason why we have to use a different Akka version for the Hadoop 2.0.0-alpha build profile. Thus, +1. On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger wrote: > I'm also in favor of shading commonly used libraries to resolve this issue > for our upstream users.

Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-18 Thread Robert Metzger
I'm also in favor of shading commonly used libraries to resolve this issue for our upstream users. I recently wrote this distributed TPC-H datagenerator, which had a hard dependency on a newer guava version. So I needed to shade guava in my project to make it work. Another candidate to shade is th

Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-17 Thread Stephan Ewen
I think the way we shaded Guava is a problem for the way IntelliJ uses maven (compile dependency projects, not package them). Since we do not apply relocation to our code for this, it should have no effect on the IDE usability. On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi wrote: > > On 17 Feb 2

Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-17 Thread Ufuk Celebi
On 17 Feb 2015, at 09:40, Stephan Ewen wrote: > Hi everyone! > > We have been time and time again struck by the problem that Hadoop bundles > many dependencies in certain versions, that conflict either with versions > of the dependencies we use, or with versions that users use. > > The most pr

[DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts

2015-02-17 Thread Stephan Ewen
Hi everyone! We have been time and time again struck by the problem that Hadoop bundles many dependencies in certain versions, that conflict either with versions of the dependencies we use, or with versions that users use. The most prominent examples are Guava and Protobuf. One way to solve this