Hello Weichen, community I'm sorry, I'm feeling a little bit confused about this vote. Is this about the PR (https://github.com/apache/spark/pull/24490) that was merged in early June and introduced the spark-graph module including the okapi-shade dependency?
Regarding the okapi-shade dependency which was developed as part of the above PR work, some advice was offered by Scala experts at TripleQuote which helped find a satisfactory solution. The shading mechanism used is standard and very comparable to a Java library shading solution. The PR you link (https://github.com/apache/spark/pull/24297) is not meant for merging. It is just a proof-of-concept branch containing a full implementation of the system, which is kept up-to-date with the API discussion on the currently proposed PR: https://github.com/apache/spark/pull/24851. Thank you Mats On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu <weichen...@databricks.com> wrote: > Hi everyone, > > I'd like to call a new vote on the issue: should we add dependency > "org.opencypher:okapi-shade.okapi" into spark ? The issue background is: > > Spark is going to add a big feature "Spark Graph", the prototypical > implementation is here > https://github.com/apache/spark/pull/24297 > which will introduce dependency org.opencypher:okapi-shade.okapi > <https://github.com/opencypher/morpheus/blob/master/okapi-shade/build.gradle> > > Xiangrui already mentioned 2 concerns on this dependency change: > >> On the technical side, my main concern is the runtime dependency on >> org.opencypher:okapi-shade.okapi depends on several Scala libraries. We >> came out with the solution to shade a few Scala libraries to avoid >> pollution. However, I'm not super confident that the approach is >> sustainable for two reasons: a) there exists no proper shading libraries >> for Scala, 2) We will have to wait for upgrades from those Scala libraries >> before we can upgrade Spark to use a newer Scala version. So it would be >> great if some Scala experts can help review the current implementation and >> help assess the risk. > > > So let's discuss and vote whether this is a good choice. > Before this spark graph feature to get into spark ASAP, this issue should > be resolved first. > > This vote is open until next Tuseday (Oct. 22). > > [ ] +1: Accept the proposal > [ ] +0 > [ ] -1: I don't think this is a good idea because ... > > Thank you! > > Weichen > >
