I do not have a very informed opinion here, so take this with a grain of salt.
I'd say that we need to either commit a coherent version of this for Spark 3, or not at all. If it doesn't have support, I'd back out the existing changes. I was initially skeptical about how much this needs to be in Spark vs a third-party package, and that still stands. The addition of another dependency isn't that big a deal IMHO, but, yes, it does add something to the maintenance overhead. But that's all the more true of a new module. I don't feel strongly about it, but if this isn't obviously getting support from any committers, can we keep it as a third party library for now? On Tue, Oct 15, 2019 at 8:53 PM Weichen Xu <weichen...@databricks.com> wrote: > > Hi Mats Rydberg, > > Although this dependency "org.opencypher:okapi-shade.okapi" was added into > spark, but Xiangrui raised two concerns (see above mail) about it, so we'd > better rethink on this and consider whether this is a good choice, so I call > this vote. > > Thanks! > > On Tue, Oct 15, 2019 at 10:56 PM Mats Rydberg <m...@neo4j.org.invalid> wrote: >> >> Hello Weichen, community >> >> I'm sorry, I'm feeling a little bit confused about this vote. Is this about >> the PR (https://github.com/apache/spark/pull/24490) that was merged in early >> June and introduced the spark-graph module including the okapi-shade >> dependency? >> >> Regarding the okapi-shade dependency which was developed as part of the >> above PR work, some advice was offered by Scala experts at TripleQuote which >> helped find a satisfactory solution. The shading mechanism used is standard >> and very comparable to a Java library shading solution. >> >> The PR you link (https://github.com/apache/spark/pull/24297) is not meant >> for merging. It is just a proof-of-concept branch containing a full >> implementation of the system, which is kept up-to-date with the API >> discussion on the currently proposed PR: >> https://github.com/apache/spark/pull/24851. >> >> Thank you >> Mats >> >> >> On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu <weichen...@databricks.com> >> wrote: >>> >>> Hi everyone, >>> >>> I'd like to call a new vote on the issue: should we add dependency >>> "org.opencypher:okapi-shade.okapi" into spark ? The issue background is: >>> >>> Spark is going to add a big feature "Spark Graph", the prototypical >>> implementation is here >>> https://github.com/apache/spark/pull/24297 >>> which will introduce dependency org.opencypher:okapi-shade.okapi >>> >>> Xiangrui already mentioned 2 concerns on this dependency change: >>>> >>>> On the technical side, my main concern is the runtime dependency on >>>> org.opencypher:okapi-shade.okapi depends on several Scala libraries. We >>>> came out with the solution to shade a few Scala libraries to avoid >>>> pollution. However, I'm not super confident that the approach is >>>> sustainable for two reasons: a) there exists no proper shading libraries >>>> for Scala, 2) We will have to wait for upgrades from those Scala libraries >>>> before we can upgrade Spark to use a newer Scala version. So it would be >>>> great if some Scala experts can help review the current implementation and >>>> help assess the risk. >>> >>> >>> So let's discuss and vote whether this is a good choice. >>> Before this spark graph feature to get into spark ASAP, this issue should >>> be resolved first. >>> >>> This vote is open until next Tuseday (Oct. 22). >>> >>> [ ] +1: Accept the proposal >>> [ ] +0 >>> [ ] -1: I don't think this is a good idea because ... >>> >>> Thank you! >>> >>> Weichen >>> --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
