No, publishing a spark assembly jar is not fine. See the doc attached to https://issues.apache.org/jira/browse/SPARK-11157 and be aware that a likely goal of Spark 2.0 will be the elimination of assemblies.
On Thu, Dec 10, 2015 at 11:19 PM, [email protected] <[email protected]> wrote: > Using maven to download the assembly jar is fine. I would recommend to > deploy this > assembly jar to your local maven repo, i.e. nexus repo, Or more likey a > snapshot repository > > ------------------------------ > [email protected] > > > *From:* Xiaoyong Zhu <[email protected]> > *Date:* 2015-12-11 15:10 > *To:* Jeff Zhang <[email protected]> > *CC:* [email protected]; Zhaomin Xu <[email protected]>; Joe Zhang > (SDE) <[email protected]> > *Subject:* RE: Spark assembly in Maven repo? > > Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s > actually that we are building a certain plugin for IntelliJ where we want > to distribute this jar. But since the jar is updated frequently we don't > want to distribute it together with our plugin but we would like to > download it via Maven. > > > > In this case what’s the recommended way? > > > > Xiaoyong > > > > *From:* Jeff Zhang [mailto:[email protected]] > *Sent:* Thursday, December 10, 2015 11:03 PM > *To:* Xiaoyong Zhu <[email protected]> > *Cc:* [email protected] > *Subject:* Re: Spark assembly in Maven repo? > > > > I don't think make the assembly jar as dependency a good practice. You may > meet jar hell issue in that case. > > > > On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu <[email protected]> > wrote: > > Hi Experts, > > > > We have a project which has a dependency for the following jar > > > > spark-assembly-<version>-hadoop<version>.jar > > for example: > > spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar > > > > since this assembly might be updated in the future, I am not sure if there > is a Maven repo that has the above spark assembly jar? Or should we create > & upload it to Maven central? > > > > Thanks! > > > > Xiaoyong > > > > > > > > -- > > Best Regards > > Jeff Zhang > >
