Hi Jakob The cluster was set up using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2 script
Given my limited knowledge I think this looks okay? Thanks Andy $ sudo netstat -peant | grep 7077 tcp 0 0 ::ffff:172-31-30-51:7077 :::* LISTEN 0 3116414 27355/java tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:57311 ESTABLISHED 0 3115919 27355/java tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:42333 ESTABLISHED 0 3736664 27355/java tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:49796 ESTABLISHED 0 3115925 27355/java tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:42290 ESTABLISHED 0 3115923 27355/java $ ps -aux | grep 27355 Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ ec2-user 23867 0.0 0.0 110404 872 pts/0 S+ 02:06 0:00 grep 27355 root 27355 0.5 6.7 3679096 515836 ? Sl Nov26 107:04 /usr/java/latest/bin/java -cp /root/spark/sbin/../conf/:/root/spark/lib/spark-assembly-1.5.1-hadoop1.2.1.j ar:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus -rdbms-3.2.9.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/ephemeral -hdfs/conf/ -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip ec2-54-215-217-122.us-west-1.compute.amazonaws.com --port 7077 --webui-port 8080 From: Jakob Odersky <[email protected]> Date: Thursday, December 10, 2015 at 5:55 PM To: Andrew Davidson <[email protected]> Cc: "user @spark" <[email protected]> Subject: Re: Warning: Master endpoint spark://ip:7077 was not a REST server. Falling back to legacy submission gateway instead. > Is there any other process using port 7077? > > On 10 December 2015 at 08:52, Andy Davidson <[email protected]> > wrote: >> Hi >> >> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. My job >> seems to run with out any problem. >> >> Kind regards >> >> Andy >> >> + /root/spark/bin/spark-submit --class com.pws.spark.streaming.IngestDriver >> --master spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> >> --total-executor-cores 2 --deploy-mode cluster >> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3 >> >> Running Spark using the REST application submission protocol. >> >> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server >> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> . >> >> Warning: Master endpoint >> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >> <http://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077> was not a >> REST server. Falling back to legacy submission gateway instead. >
