does maven support cross building for different scala versions? we do this inhouse all the time with sbt. i know spark does not cross build at this point, but is it guaranteed to stay that way?
On Sat, Mar 1, 2014 at 12:02 PM, Koert Kuipers <ko...@tresata.com> wrote: > i am still unsure what is wrong with sbt assembly. i would like a > real-world example of where it does not work, that i can run. > > this is what i know: > > 1) sbt assembly works fine for version conflicts for an artifact. no > exclusion rules are needed. > > 2) if artifacts have the same classes inside yet are not recognized as > different versions of the same artifact (due to renaming of artifacts > typically, or due to the inclusion of classes from another jar) then a > manual exclusion rule will be needed, or else sbt will apply a simple but > programmable rule to pick one class and drop the rest. i do not see how > maven could do this better or without manual exclusion rules. > > > > On Sat, Mar 1, 2014 at 1:00 AM, Mridul Muralidharan <mri...@gmail.com>wrote: > >> On Sat, Mar 1, 2014 at 2:05 AM, Patrick Wendell <pwend...@gmail.com> >> wrote: >> > Hey, >> > >> > Thanks everyone for chiming in on this. I wanted to summarize these >> > issues a bit particularly wrt the constituents involved - does this >> > seem accurate? >> > >> > = Spark Users = >> > In general those linking against Spark should be totally unaffected by >> > the build choice. Spark will continue to publish well-formed poms and >> > jars to maven central. This is a no-op wrt this decision. >> > >> > = Spark Developers = >> > There are two concerns. (a) General day-to-day development and >> > packaging and (b) Spark binaries and packages for distribution. >> > >> > For (a) - sbt seems better because it's just nicer for doing scala >> > development (incremental complication is simple, we have some >> > home-baked tools for compiling Spark vs. the spark deps etc). The >> > arguments that maven has more "general know how", at least so far, >> > haven't affected us in the ~2 years we've maintained both builds - >> > where adding stuff for Maven is typically just as annoying/difficult >> > with sbt. >> > >> > For (b) - Some non-specific concerns were raised about bugs with the >> > sbt assembly package - we should look into this and see what is going >> > on. Maven has better out-of-the-box support for publishing to Maven >> > central, we'd have to do some manual work on our end to make this work >> > well with sbt. >> >> >> Not non-specific concerns, assembly via sbt is fragile - the (manual) >> exclusion rules in sbt project are testament to this. >> >> In particular, I dont see any quantifiable benefits in using sbt over >> maven. >> Incremental compilation, compiling only a subproject, running specific >> tests, etc are all available even with maven - so are not >> differentiators. >> On other hand, sbt does introduce further manual overhead in >> dependency management for assembled/shaded jar creation. >> >> Regards, >> Mridul >> >> >> >> >> > >> > = Downstream Integrators = >> > On this one it seems that Maven is the universal favorite, largely >> > because of community awareness of Maven and comfort with Maven builds. >> > Some things like restructuring the Spark build to inherit config >> > values from a vendor build will be not possible with sbt (though >> > fairly straightforward to work around). Other cases where vendors have >> > directly modified or inherited the Spark build won't work anymore if >> > we standardize on SBT. These have no obvious work around at this point >> > as far as I see. >> > >> > - Patrick >> > >> > On Wed, Feb 26, 2014 at 7:09 PM, Mridul Muralidharan <mri...@gmail.com> >> wrote: >> >> On Feb 26, 2014 11:12 PM, "Patrick Wendell" <pwend...@gmail.com> >> wrote: >> >>> >> >>> @mridul - As far as I know both Maven and Sbt use fairly similar >> >>> processes for building the assembly/uber jar. We actually used to >> >>> package spark with sbt and there were no specific issues we >> >>> encountered and AFAIK sbt respects versioning of transitive >> >>> dependencies correctly. Do you have a specific bug listing for sbt >> >>> that indicates something is broken? >> >> >> >> Slightly longish ... >> >> >> >> The assembled jar, generated via sbt broke all over the place while I >> was >> >> adding yarn support in 0.6 - and I had to fix sbt project a fair bit >> to get >> >> it to work : we need the assembled jar to submit a yarn job. >> >> >> >> When I finally submitted those changes to 0.7, it broke even more - >> since >> >> dependencies changed : someone else had thankfully already added maven >> >> support by then - which worked remarkably well out of the box (with >> some >> >> minor tweaks) ! >> >> >> >> In theory, they might be expected to work the same, but practically >> they >> >> did not : as I mentioned, it must just have been luck that maven >> worked >> >> that well; but given multiple past nasty experiences with sbt, and the >> fact >> >> that it does not bring anything compelling or new in contrast, I am >> fairly >> >> against the idea of using only sbt - inspite of maven being >> unintuitive at >> >> times. >> >> >> >> Regards, >> >> Mridul >> >> >> >>> >> >>> @sandy - It sounds like you are saying that the CDH build would be >> >>> easier with Maven because you can inherit the POM. However, is this >> >>> just a matter of convenience for packagers or would standardizing on >> >>> sbt limit capabilities in some way? I assume that it would just mean a >> >>> bit more manual work for packagers having to figure out how to set the >> >>> hadoop version in SBT and exclude certain dependencies. For instance, >> >>> what does CDH about other components like Impala that are not based on >> >>> Maven at all? >> >>> >> >>> On Wed, Feb 26, 2014 at 9:31 AM, Evan Chan <e...@ooyala.com> wrote: >> >>> > I'd like to propose the following way to move forward, based on the >> >>> > comments I've seen: >> >>> > >> >>> > 1. Aggressively clean up the giant dependency graph. One ticket I >> >>> > might work on if I have time is SPARK-681 which might remove the >> giant >> >>> > fastutil dependency (~15MB by itself). >> >>> > >> >>> > 2. Take an intermediate step by having only ONE source of truth >> >>> > w.r.t. dependencies and versions. This means either: >> >>> > a) Using a maven POM as the spec for dependencies, Hadoop >> version, >> >>> > etc. Then, use sbt-pom-reader to import it. >> >>> > b) Using the build.scala as the spec, and "sbt make-pom" to >> >>> > generate the pom.xml for the dependencies >> >>> > >> >>> > The idea is to remove the pain and errors associated with manual >> >>> > translation of dependency specs from one system to another, while >> >>> > still maintaining the things which are hard to translate (plugins). >> >>> > >> >>> > >> >>> > On Wed, Feb 26, 2014 at 7:17 AM, Koert Kuipers <ko...@tresata.com> >> >> wrote: >> >>> >> We maintain in house spark build using sbt. We have no problem >> using >> >> sbt >> >>> >> assembly. We did add a few exclude statements for transitive >> >> dependencies. >> >>> >> >> >>> >> The main enemy of assemblies are jars that include stuff they >> shouldn't >> >>> >> (kryo comes to mind, I think they include logback?), new versions >> of >> >> jars >> >>> >> that change the provider/artifact without changing the package >> (asm), >> >> and >> >>> >> incompatible new releases (protobuf). These break the transitive >> >> resolution >> >>> >> process. I imagine that's true for any build tool. >> >>> >> >> >>> >> Besides shading I don't see anything maven can do sbt cannot, and >> if I >> >>> >> understand it correctly shading is not done currently using the >> build >> >> tool. >> >>> >> >> >>> >> Since spark is primarily scala/akka based the main developer base >> will >> >> be >> >>> >> familiar with sbt (I think?). Switching build tool is always >> painful. I >> >>> >> personally think it is smarter to put this burden on a limited >> number >> >> of >> >>> >> upstream integrators than on the community. However that said I >> don't >> >> think >> >>> >> its a problem for us to maintain an sbt build in-house if spark >> >> switched to >> >>> >> maven. >> >>> >> The problem is, the complete spark dependency graph is fairly >> large, >> >>> >> and there are lot of conflicting versions in there. >> >>> >> In particular, when we bump versions of dependencies - making >> managing >> >>> >> this messy at best. >> >>> >> >> >>> >> Now, I have not looked in detail at how maven manages this - it >> might >> >>> >> just be accidental that we get a decent out-of-the-box assembled >> >>> >> shaded jar (since we dont do anything great to configure it). >> >>> >> With current state of sbt in spark, it definitely is not a good >> >>> >> solution : if we can enhance it (or it already is ?), while keeping >> >>> >> the management of the version/dependency graph manageable, I dont >> have >> >>> >> any objections to using sbt or maven ! >> >>> >> Too many exclude versions, pinned versions, etc would just make >> things >> >>> >> unmanageable in future. >> >>> >> >> >>> >> >> >>> >> Regards, >> >>> >> Mridul >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> On Wed, Feb 26, 2014 at 8:56 AM, Evan chan <e...@ooyala.com> wrote: >> >>> >>> Actually you can control exactly how sbt assembly merges or >> resolves >> >>> >> conflicts. I believe the default settings however lead to order >> which >> >>> >> cannot be controlled. >> >>> >>> >> >>> >>> I do wish for a smarter fat jar plugin. >> >>> >>> >> >>> >>> -Evan >> >>> >>> To be free is not merely to cast off one's chains, but to live in >> a >> >> way >> >>> >> that respects & enhances the freedom of others. (#NelsonMandela) >> >>> >>> >> >>> >>>> On Feb 25, 2014, at 6:50 PM, Mridul Muralidharan < >> mri...@gmail.com> >> >>> >> wrote: >> >>> >>>> >> >>> >>>>> On Wed, Feb 26, 2014 at 5:31 AM, Patrick Wendell < >> pwend...@gmail.com >> >>> >> >>> >> wrote: >> >>> >>>>> Evan - this is a good thing to bring up. Wrt the shader plug-in >> - >> >>> >>>>> right now we don't actually use it for bytecode shading - we >> simply >> >>> >>>>> use it for creating the uber jar with excludes (which sbt >> supports >> >>> >>>>> just fine via assembly). >> >>> >>>> >> >>> >>>> >> >>> >>>> Not really - as I mentioned initially in this thread, sbt's >> assembly >> >>> >>>> does not take dependencies into account properly : and can >> overwrite >> >>> >>>> newer classes with older versions. >> >>> >>>> From an assembly point of view, sbt is not very good : we are >> yet to >> >>> >>>> try it after 2.10 shift though (and probably wont, given the >> mess it >> >>> >>>> created last time). >> >>> >>>> >> >>> >>>> Regards, >> >>> >>>> Mridul >> >>> >>>> >> >>> >>>> >> >>> >>>> >> >>> >>>> >> >>> >>>> >> >>> >>>>> >> >>> >>>>> I was wondering actually, do you know if it's possible to added >> >> shaded >> >>> >>>>> artifacts to the *spark jar* using this plug-in (e.g. not an >> uber >> >>> >>>>> jar)? That's something I could see being really handy in the >> future. >> >>> >>>>> >> >>> >>>>> - Patrick >> >>> >>>>> >> >>> >>>>>> On Tue, Feb 25, 2014 at 3:39 PM, Evan Chan <e...@ooyala.com> >> wrote: >> >>> >>>>>> The problem is that plugins are not equivalent. There is >> AFAIK no >> >>> >>>>>> equivalent to the maven shader plugin for SBT. >> >>> >>>>>> There is an SBT plugin which can apparently read POM XML files >> >>> >>>>>> (sbt-pom-reader). However, it can't possibly handle plugins, >> >> which >> >>> >>>>>> is still problematic. >> >>> >>>>>> >> >>> >>>>>>> On Tue, Feb 25, 2014 at 3:31 PM, yao <yaosheng...@gmail.com> >> >> wrote: >> >>> >>>>>>> I would prefer keep both of them, it would be better even if >> that >> >>> >> means >> >>> >>>>>>> pom.xml will be generated using sbt. Some company, like my >> current >> >>> >> one, >> >>> >>>>>>> have their own build infrastructures built on top of maven. >> It is >> >> not >> >>> >> easy >> >>> >>>>>>> to support sbt for these potential spark clients. But I do >> agree >> >> to >> >>> >> only >> >>> >>>>>>> keep one if there is a promising way to generate correct >> >>> >> configuration from >> >>> >>>>>>> the other. >> >>> >>>>>>> >> >>> >>>>>>> -Shengzhe >> >>> >>>>>>> >> >>> >>>>>>> >> >>> >>>>>>>> On Tue, Feb 25, 2014 at 3:20 PM, Evan Chan <e...@ooyala.com> >> wrote: >> >>> >>>>>>>> >> >>> >>>>>>>> The correct way to exclude dependencies in SBT is actually to >> >> declare >> >>> >>>>>>>> a dependency as "provided". I'm not familiar with Maven or >> its >> >>> >>>>>>>> dependencySet, but provided will mark the entire dependency >> tree >> >> as >> >>> >>>>>>>> excluded. It is also possible to exclude jar by jar, but >> this >> >> is >> >>> >>>>>>>> pretty error prone and messy. >> >>> >>>>>>>> >> >>> >>>>>>>>> On Tue, Feb 25, 2014 at 2:45 PM, Koert Kuipers < >> >> ko...@tresata.com> >> >>> >> wrote: >> >>> >>>>>>>>> yes in sbt assembly you can exclude jars (although i never >> had a >> >>> >> need for >> >>> >>>>>>>>> this) and files in jars. >> >>> >>>>>>>>> >> >>> >>>>>>>>> for example i frequently remove log4j.properties, because >> for >> >>> >> whatever >> >>> >>>>>>>>> reason hadoop decided to include it making it very >> difficult to >> >> use >> >>> >> our >> >>> >>>>>>>> own >> >>> >>>>>>>>> logging config. >> >>> >>>>>>>>> >> >>> >>>>>>>>> >> >>> >>>>>>>>> >> >>> >>>>>>>>>> On Tue, Feb 25, 2014 at 4:24 PM, Konstantin Boudnik < >> >> c...@apache.org >> >>> >>> >> >>> >>>>>>>>> wrote: >> >>> >>>>>>>>> >> >>> >>>>>>>>>>> On Fri, Feb 21, 2014 at 11:11AM, Patrick Wendell wrote: >> >>> >>>>>>>>>>> Kos - thanks for chiming in. Could you be more specific >> about >> >>> >> what is >> >>> >>>>>>>>>>> available in maven and not in sbt for these issues? I >> took a >> >> look >> >>> >> at >> >>> >>>>>>>>>>> the bigtop code relating to Spark. As far as I could tell >> [1] >> >> was >> >>> >> the >> >>> >>>>>>>>>>> main point of integration with the build system (maybe >> there >> >> are >> >>> >> other >> >>> >>>>>>>>>>> integration points)? >> >>> >>>>>>>>>>> >> >>> >>>>>>>>>>>> - in order to integrate Spark well into existing Hadoop >> >> stack it >> >>> >>>>>>>> was >> >>> >>>>>>>>>>>> necessary to have a way to avoid transitive >> dependencies >> >>> >>>>>>>>>> duplications and >> >>> >>>>>>>>>>>> possible conflicts. >> >>> >>>>>>>>>>>> >> >>> >>>>>>>>>>>> E.g. Maven assembly allows us to avoid adding _all_ >> Hadoop >> >>> >> libs >> >>> >>>>>>>>>> and later >> >>> >>>>>>>>>>>> merely declare Spark package dependency on standard >> Bigtop >> >>> >>>>>>>> Hadoop >> >>> >>>>>>>>>>>> packages. And yes - Bigtop packaging means the naming >> and >> >>> >> layout >> >>> >>>>>>>>>> would be >> >>> >>>>>>>>>>>> standard across all commercial Hadoop distributions >> that >> >> are >> >>> >>>>>>>> worth >> >>> >>>>>>>>>>>> mentioning: ASF Bigtop convenience binary packages, >> and >> >>> >>>>>>>> Cloudera or >> >>> >>>>>>>>>>>> Hortonworks packages. Hence, the downstream user >> doesn't >> >> need >> >>> >> to >> >>> >>>>>>>>>> spend any >> >>> >>>>>>>>>>>> effort to make sure that Spark "clicks-in" properly. >> >>> >>>>>>>>>>> >> >>> >>>>>>>>>>> The sbt build also allows you to plug in a Hadoop version >> >> similar >> >>> >> to >> >>> >>>>>>>>>>> the maven build. >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> I am actually talking about an ability to exclude a set of >> >>> >> dependencies >> >>> >>>>>>>>>> from an >> >>> >>>>>>>>>> assembly, similarly to what's happening in dependencySet >> >> sections >> >>> >> of >> >>> >>>>>>>>>> assembly/src/main/assembly/assembly.xml >> >>> >>>>>>>>>> If there is a comparable functionality in Sbt, that would >> help >> >>> >> quite a >> >>> >>>>>>>> bit, >> >>> >>>>>>>>>> apparently. >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> Cos >> >>> >>>>>>>>>> >> >>> >>>>>>>>>>>> - Maven provides a relatively easy way to deal with the >> >> jar-hell >> >>> >>>>>>>>>> problem, >> >>> >>>>>>>>>>>> although the original maven build was just Shader'ing >> >>> >> everything >> >>> >>>>>>>>>> into a >> >>> >>>>>>>>>>>> huge lump of class files. Oftentimes ending up with >> >> classes >> >>> >>>>>>>>>> slamming on >> >>> >>>>>>>>>>>> top of each other from different transitive >> dependencies. >> >>> >>>>>>>>>>> >> >>> >>>>>>>>>>> AFIAK we are only using the shade plug-in to deal with >> >> conflict >> >>> >>>>>>>>>>> resolution in the assembly jar. These are dealt with in >> sbt >> >> via >> >>> >> the >> >>> >>>>>>>>>>> sbt assembly plug-in in an identical way. Is there a >> >> difference? >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> I am bringing up the Sharder, because it is an awful hack, >> >> which is >> >>> >>>>>>>> can't >> >>> >>>>>>>>>> be >> >>> >>>>>>>>>> used in real controlled deployment. >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> Cos >> >>> >>>>>>>>>> >> >>> >>>>>>>>>>> [1] >> >>> >>>>>>>> >> >>> >> >> >> >> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob;f=bigtop-packages/src/common/spark/do-component-build;h=428540e0f6aa56cd7e78eb1c831aa7fe9496a08f;hb=master >> >>> >>>>>>>> >> >>> >>>>>>>> >> >>> >>>>>>>> >> >>> >>>>>>>> -- >> >>> >>>>>>>> -- >> >>> >>>>>>>> Evan Chan >> >>> >>>>>>>> Staff Engineer >> >>> >>>>>>>> e...@ooyala.com | >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> -- >> >>> >>>>>> -- >> >>> >>>>>> Evan Chan >> >>> >>>>>> Staff Engineer >> >>> >>>>>> e...@ooyala.com | >> >>> > >> >>> > >> >>> > >> >>> > -- >> >>> > -- >> >>> > Evan Chan >> >>> > Staff Engineer >> >>> > e...@ooyala.com | >> > >
