@evan
>From the discussion in the JIRA, it seems that we still dont have a clear
solution for SPARK-1138. Nor do we have a sense of whether the solution is
going to small enough for a maintenance release. So I dont think we should
block the release of Spark 0.9.1 for this. We can make another Spark 0.9.2
release once the correct solution has been figured out.

@kevin
I understand the problem. I will try to port the solution for master
inthis PR <https://github.com/apache/spark/pull/100/> into
branch 0.9. Lets see if it works out.


On Tue, Mar 25, 2014 at 10:19 AM, Kevin Markey <kevin.mar...@oracle.com>wrote:

> TD:
>
> A correct shading of ASM should only affect Spark code unless someone is
> relying on ASM 4.0 in unrelated project code, in which case they can add
> org.ow2.asm:asm:4.x as a dependency.
>
> Our short term solution has been to repackage other libraries with a 3.2
> dependency or to exclude ASM when our use of a dependent library really
> doesn't need it.  As you probably know, the real problem arises in
> ClassVisitor, which is an Interface in 3.x and before, but in 4.x it is an
> abstract class that takes a version constant as its constructor.  The ASM
> folks of course had our best interests in mind when they did this,
> attempting to deal with the Java-version dependent  changes from one ASM
> release to the next.  Unfortunately, they didn't change the names or
> locations of their classes and interfaces, which would have helped.
>
> In our particular case, the only library from which we couldn't exclude
> ASM was org.glassfish.jersey.containers:jersey-container-servlet:jar:2.5.1.
> I added a new module to our project, including some dummy source code,
> because we needed the library to be self contained, made the servlet --
> minus some unrelated transitive dependencies -- the only module dependency,
> then used the Maven shade plugin to relocate "org.objectweb.asm" to an
> arbitrary target.  We added the new shaded module as a new project
> dependency, plus the unrelated transitive dependencies excluded above.
> This solved the problem. At least until we added WADL to the project.  Then
> we needed to deal with it on its own terms.
>
> As you can see, we left Spark alone in all its ASM 4.0 glory.  Why? Spark
> is more volatile than the other libraries.  Also, the way in which we
> needed to deploy Spark and other resources on our (Yarn) clusters suggested
> that it would be easier to shade the other libraries.  I wanted to avoid
> having to install a locally patched Spark library into our build, updating
> the cluster and individual developers whenever there's a new patch.
>  Individual developers such as me who are testing the impact of patches can
> handle it, but the main build goes to Maven Central via our corporate
> Artifactory mirror.
>
> If suddenly we had a Spark 0.9.1 with a shaded ASM, it would have no
> negative impact on us.  Only a positive impact.
>
> I just wish that all users of ASM would read FAQ entry 15!!!
>
> Thanks
> Kevin
>
>
>
> On 03/24/2014 06:30 PM, Tathagata Das wrote:
>
>> Hello Kevin,
>>
>> A fix for SPARK-782 would definitely simplify building against Spark.
>> However, its possible that a fix for this issue in 0.9.1 will break
>> the builds (that reference spark) of existing 0.9 users, either due to
>> a change in the ASM version, or for being incompatible with their
>> current workarounds for this issue. That is not a good idea for a
>> maintenance release, especially when 1.0 is not too far away.
>>
>> Can you (and others) elaborate more on the current workarounds that
>> you have for this issue? Its best to understand all the implications
>> of this fix.
>>
>> Note that in branch 0.9, it is not fixed, neither in SBT nor in Maven.
>>
>> TD
>>
>> On Mon, Mar 24, 2014 at 4:38 PM, Kevin Markey <kevin.mar...@oracle.com>
>> wrote:
>>
>>> Is there any way that [SPARK-782] (Shade ASM) can be included?  I see
>>> that
>>> it is not currently backported to 0.9.  But there is no single issue that
>>> has caused us more grief as we integrate spark-core with other project
>>> dependencies.  There are way too many libraries out there in addition to
>>> Spark 0.9 and before that are not well-behaved (ASM FAQ recommends
>>> shading),
>>> including some Hive and Hadoop libraries and a number of servlet
>>> libraries.
>>> We can't control those, but if Spark were well behaved in this regard, it
>>> would help.  Even for a maintenance release, and even if 1.0 is only 6
>>> weeks
>>> away!
>>>
>>> (For those not following 782, according to Jira comments, the SBT build
>>> shades it, but it is the Maven build that ends up in Maven Central.)
>>>
>>> Thanks
>>> Kevin Markey
>>>
>>>
>>>
>>>
>>> On 03/19/2014 06:07 PM, Tathagata Das wrote:
>>>
>>>>    Hello everyone,
>>>>
>>>> Since the release of Spark 0.9, we have received a number of important
>>>> bug
>>>> fixes and we would like to make a bug-fix release of Spark 0.9.1. We are
>>>> going to cut a release candidate soon and we would love it if people
>>>> test
>>>> it out. We have backported several bug fixes into the 0.9 and updated
>>>> JIRA
>>>>
>>>> accordingly<https://spark-project.atlassian.net/browse/
>>>> SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%
>>>> 20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%
>>>> 20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%
>>>> 20in%20(Resolved%2C%20Closed)>.
>>>>
>>>> Please let me know if there are fixes that were not backported but you
>>>> would like to see them in 0.9.1.
>>>>
>>>> Thanks!
>>>>
>>>> TD
>>>>
>>>>
>

Reply via email to