This is not the correct way to build Spark with sbt. Why ?
On Sun, Oct 15, 2017 at 11:54 PM, Mark Hamstra <m...@clearstorydata.com>
wrote:
> I am building Spark using build.sbt.
>
>
> Which just gets me back to my original question: Why? This is not the
> correct way to build Spark with sbt.
>
> On Sun, Oct 15, 2017 at 11:40 PM, patel kumar <patel.kumar...@gmail.com>
> wrote:
>
>> I am building Spark using build.sbt. Details are mentioned in the
>> original mail chain.
>>
>> When I was using Spark1.6 using scala 2.10, everything was working fine.
>>
>> Issue arises when I am updating my code to make it compatible with spark
>> 2.1.
>>
>> It is failing while doing sbt assembly.
>>
>>
>>
>> On Sun, Oct 15, 2017 at 11:29 PM, Mark Hamstra <m...@clearstorydata.com>
>> wrote:
>>
>>> I don't understand. Are you building Spark or something else?
>>>
>>> If you are trying to build Spark, it doesn't look like you are doing it
>>> the right way. As I mentioned before, and is explained in detail in the
>>> link I provided, building Spark with sbt is done via build/sbt, not by
>>> directly invoking your choice of an sbt version.
>>>
>>> On Sun, Oct 15, 2017 at 11:26 PM, patel kumar <patel.kumar...@gmail.com>
>>> wrote:
>>>
>>>> Because earlier, I was using sbt 0.13.13.1, and I was getting another
>>>> version conflict i.e.
>>>>
>>>> *[error] Modules were resolved with conflicting cross-version suffixes
>>>> in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp*
>>>> *arkTest/}newsparktest:*
>>>> *[error] org.json4s:json4s-ast _2.10, _2.11*
>>>> *[error] org.json4s:json4s-core _2.10, _2.11*
>>>> *[trace] Stack trace suppressed: run last *:update for the full output.*
>>>> *[error] (*:update) Conflicting cross-version suffixes in:
>>>> org.json4s:json4s-ast, org.json4s:json4s-core*
>>>>
>>>>
>>>> Actually issue is because of SBT compilation. sbt 0.13.x is originally
>>>> compiled with scala 2.10 and gives support to 2.11.x.
>>>>
>>>> When I execute sbt , all the default jars are automatically downloaded
>>>> with suffuix _10 (for scala 2.10). When execute assembly json4s jars for
>>>> 2.11 is also downloaded.
>>>>
>>>> I upgraded the sbt with the hope that this issue might be fixed with
>>>> new version of sbt.
>>>>
>>>> Sbt 1.02 introduced the same problem for XML related jars as sbt 1.0.2
>>>> originally compiled with scala 2.12.
>>>>
>>>> I looked up the sbt version which would by default download 2.11.x
>>>> jars, but failed to find any such version.
>>>>
>>>> So, now I am looking for a solution by which I could override the jars
>>>> that are downloaded by default during sbt.
>>>>
>>>>
>>>> On Sun, Oct 15, 2017 at 11:03 PM, Mark Hamstra <m...@clearstorydata.com
>>>> > wrote:
>>>>
>>>>> sbt version is 1.0.2.
>>>>>
>>>>>
>>>>> Why?
>>>>>
>>>>> Building Spark with sbt is done via build/sbt, which will give you sbt
>>>>> 0.13.11 when building Spark 2.1.0.
>>>>>
>>>>> https://spark.apache.org/docs/2.1.0/building-spark.html#buil
>>>>> ding-with-sbt
>>>>>
>>>>>
>>>>> On Sun, Oct 15, 2017 at 10:42 PM, patel kumar <
>>>>> patel.kumar...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using CDH cluster with Spark 2.1 with Scala Version 2.11.8.
>>>>>> sbt version is 1.0.2.
>>>>>>
>>>>>> While doing assembly , I am getting error as
>>>>>> *[error] java.lang.RuntimeException: Conflicting cross-version
>>>>>> suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.*
>>>>>> *modules:scala-parser-combinators*
>>>>>>
>>>>>> I tried to override the version mismatch using dependencyOverrides
>>>>>> and force(), but none of the solution worked.
>>>>>>
>>>>>> Please help me to resolve this version Conflict.
>>>>>>
>>>>>> Details of the configuration are mentioned below :-
>>>>>>
>>>>>> *build.sbt *
>>>>>>
>>>>>> ************************************************************
>>>>>> ***********
>>>>>> name := "newtest"
>>>>>> version := "0.0.2"
>>>>>>
>>>>>> scalaVersion := "2.11.8"
>>>>>>
>>>>>> sbtPlugin := true
>>>>>>
>>>>>> val sparkVersion = "2.1.0"
>>>>>>
>>>>>> mainClass in (Compile, run) := Some("com.testpackage.sq.newsp
>>>>>> arktest")
>>>>>>
>>>>>> assemblyJarName in assembly := "newtest.jar"
>>>>>>
>>>>>>
>>>>>> libraryDependencies ++= Seq(
>>>>>> "org.apache.spark" % "spark-core_2.11" % "2.1.1" % "provided",
>>>>>> "org.apache.spark" % "spark-sql_2.11" % "2.1.1" % "provided",
>>>>>> "com.databricks" % "spark-avro_2.11" % "3.2.0",
>>>>>> "org.apache.spark" % "spark-hive_2.11" % "2.1.1" % "provided"
>>>>>> )
>>>>>>
>>>>>>
>>>>>> libraryDependencies +=
>>>>>> "log4j" % "log4j" % "1.2.15" excludeAll(
>>>>>> ExclusionRule(organization = "com.sun.jdmk"),
>>>>>> ExclusionRule(organization = "com.sun.jmx"),
>>>>>> ExclusionRule(organization = "javax.jms")
>>>>>> )
>>>>>>
>>>>>> resolvers += "SparkPackages" at "https://dl.bintray.com/spark-
>>>>>> packages/maven/"
>>>>>> resolvers += Resolver.url("bintray-sbt-plugins", url("
>>>>>> http://dl.bintray.com/sbt/sbt-plugin-releases";))(Resolv
>>>>>> er.ivyStylePatterns)
>>>>>>
>>>>>> assemblyMergeStrategy in assembly := {
>>>>>> case PathList("META-INF", xs @ _*) => MergeStrategy.discard
>>>>>> case x => MergeStrategy.first
>>>>>> }
>>>>>>
>>>>>> ***************************************************
>>>>>> *plugins.sbt*
>>>>>>
>>>>>> ************************************
>>>>>>
>>>>>> dependencyOverrides += ("org.scala-lang.modules" % "scala-xml_2.11" %
>>>>>> "1.0.4")
>>>>>> dependencyOverrides += ("org.scala-lang.modules" %
>>>>>> "scala-parser-combinators_2.11" % "1.0.4")
>>>>>> addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
>>>>>> resolvers += Resolver.url("bintray-sbt-plugins", url("
>>>>>> https://dl.bintray.com/eed3si9n/sbt-plugins/";))(Resolve
>>>>>> r.ivyStylePatterns)
>>>>>>
>>>>>> *****************************************************
>>>>>> *Error Message after assembly*
>>>>>>
>>>>>> **************************************************
>>>>>> [error] Modules were resolved with conflicting cross-version suffixes
>>>>>> in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp
>>>>>> arkTest/}newsparktest:
>>>>>> [error] org.scala-lang.modules:scala-xml _2.11, _2.12
>>>>>> [error] org.scala-lang.modules:scala-parser-combinators _2.11,
>>>>>> _2.12
>>>>>> [error] java.lang.RuntimeException: Conflicting cross-version
>>>>>> suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
>>>>>> modules:scala-parser-combinators
>>>>>> [error] at scala.sys.package$.error(package.scala:27)
>>>>>> [error] at sbt.librarymanagement.Conflict
>>>>>> Warning$.processCrossVersioned(ConflictWarning.scala:39)
>>>>>> [error] at sbt.librarymanagement.Conflict
>>>>>> Warning$.apply(ConflictWarning.scala:19)
>>>>>> [error] at sbt.Classpaths$.$anonfun$ivyBa
>>>>>> seSettings$64(Defaults.scala:1971)
>>>>>> [error] at scala.Function1.$anonfun$compo
>>>>>> se$1(Function1.scala:44)
>>>>>> [error] at sbt.internal.util.$tilde$great
>>>>>> er.$anonfun$$u2219$1(TypeFunctions.scala:42)
>>>>>> [error] at sbt.std.Transform$$anon$4.work(System.scala:64)
>>>>>> [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
>>>>>> [error] at sbt.internal.util.ErrorHandlin
>>>>>> g$.wideConvert(ErrorHandling.scala:16)
>>>>>> [error] at sbt.Execute.work <http://sbt.execute.work/>
>>>>>> (Execute.scala:266)
>>>>>> [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
>>>>>> [error] at sbt.ConcurrentRestrictions$$an
>>>>>> on$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
>>>>>> [error] at sbt.CompletionService$$anon$2.
>>>>>> call(CompletionService.scala:32)
>>>>>> [error] at java.util.concurrent.FutureTas
>>>>>> k.run(FutureTask.java:266)
>>>>>> [error] at java.util.concurrent.Executors
>>>>>> $RunnableAdapter.call(Executors.java:511)
>>>>>> [error] at java.util.concurrent.FutureTas
>>>>>> k.run(FutureTask.java:266)
>>>>>> [error] at java.util.concurrent.ThreadPoo
>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>> [error] at java.util.concurrent.ThreadPoo
>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> [error] at java.lang.Thread.run(Thread.java:748)
>>>>>> [error] (*:update) Conflicting cross-version suffixes in:
>>>>>> org.scala-lang.modules:scala-xml, org.scala-lang.modules:scala-par
>>>>>> ser-combinators
>>>>>> [error] Total time: 413 s, completed Oct 12, 2017 3:28:02 AM
>>>>>>
>>>>>> **************************************************
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
