Ah, sorry, forgot to talk about the second issue. On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell <matt.narr...@gmail.com> wrote: > However, now the Spark jobs running in the ApplicationMaster on a given node > fails to find the active resourcemanager. Below is a log excerpt from one > of the assigned nodes. As all the jobs fail, eventually YARN will move this > to execute on the node that co-locates the active resourcemanager and a > nodemanager, where the job will proceed a bit further. Then, the Spark job > itself will fail attempting to access HDFS files via the virtualized HA HDFS > URI.
> 14/08/20 11:34:27 INFO Client: Retrying connect to server: > 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) Here it seems you're running into the same issue but on the AM side. If you're comfortable with using a custom build of Spark, I'd recommend building it with the "-Phadoop-provided" profile enabled (note: I think that only works with maven currently). That way the Spark assembly does not include the Hadoop classes. A quick look at the code seems to indicate the Spark assembly is always added before the local Hadoop jars in the classpath. Unfortunately there's no workaround for that at the moment (aside from the above built-time fix), although we might be able to do something when SPARK-2848 is fixed. -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
