I have a very simple program that runs fine on my Linux server that runs
Spark master and worker in standalone mode.
public class SimpleSpark {
public int sum () {
SparkConf conf = new SparkConf()
.setAppName("Magellan")
.setMaster("spark://
ec2-nnn-nnn-nnn-nnn.compute-1.amazonaws.com:11407")
.setJars(new String[]
{"target/magellan-spark-1.0-SNAPSHOT.jar"});
JavaSparkContext sc = new JavaSparkContext(conf);
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
JavaRDD<Integer> distData = sc.parallelize(data);
int total = distData.reduce(new SumFunc());
return total;
}
public static class SumFunc implements Function2<Integer, Integer,
Integer> {
public Integer call(Integer a, Integer b) {
return a + b;
}
};
However, when I run the same driver from a Windows machine it outputs the
following message and never completes:
16/01/11 20:51:11 WARN TaskSchedulerImpl: Initial job has not accepted
any resources; check your cluster UI to ensure that workers are registered
and have sufficient resources
I have checked the cluster UI and the job is marked as RUNNING (so it does
not appears to be waiting on a worker). I do not see anything out of the
ordinary in the master and worker logs.
How do I debug a problem like this?
-Andrew