Hi,
I am running Spark in cluster mode with Hadoop YARN as the underlying
cluster manager. I get this error when trying to initialize the
SparkContext.
Exception in thread "main" org.apache.spark.SparkException: YARN mode not
available ?
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:1586)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:310)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:86)
at LascoScript$.main(LascoScript.scala:24)
at LascoScript.main(LascoScript.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.scheduler.cluster.YarnClientClusterScheduler
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:1580)
My build.sbt file looks like this:
name := "LascoScript"
version := "1.0"
scalaVersion := "2.10.4"
val excludeJBossNetty = ExclusionRule(organization = "org.jboss.netty")
val excludeMortbayJetty = ExclusionRule(organization = "org.eclipse.jetty",
artifact = "jetty-server")
val excludeAsm = ExclusionRule(organization = "org.ow2.asm")
val excludeCommonsLogging = ExclusionRule(organization = "commons-logging")
val excludeSLF4J = ExclusionRule(organization = "org.slf4j")
val excludeOldAsm = ExclusionRule(organization = "asm")
val excludeServletApi = ExclusionRule(organization = "javax.servlet",
artifact = "servlet-api")
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
excludeAll(
excludeServletApi, excludeMortbayJetty
)
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.5.1"
excludeAll(
excludeJBossNetty, excludeMortbayJetty, excludeAsm, excludeCommonsLogging,
excludeSLF4J, excludeOldAsm, excludeServletApi
)
libraryDependencies += "org.mortbay.jetty" % "servlet-api" % "3.0.20100224"
libraryDependencies += "org.eclipse.jetty" % "jetty-server" %
"8.1.16.v20140903"
unmanagedJars in Compile ++= {
val base = baseDirectory.value
val baseDirectories = (base / "lib") +++ (base)
val customJars = (baseDirectories ** "*.jar")
customJars.classpath
}
resolvers += "Akka Repository" at "http://repo.akka.io/releases/“
How can I fix this issue?
- Barrington
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/YARN-mode-not-available-error-tp14420.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
