Hi Andy, You must be running in cluster mode. The Spark Master accepts client mode submissions on port 7077 and cluster mode submissions on port 6066. This is because standalone cluster mode uses a REST API to submit applications by default. If you submit to port 6066 instead the warning should go away.
-Andrew 2015-12-10 18:13 GMT-08:00 Andy Davidson <a...@santacruzintegration.com>: > Hi Jakob > > The cluster was set up using the spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2 > script > > Given my limited knowledge I think this looks okay? > > Thanks > > Andy > > $ sudo netstat -peant | grep 7077 > > tcp 0 0 ::ffff:172-31-30-51:7077 :::* > LISTEN 0 3116414 27355/java > > tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:57311 > ESTABLISHED 0 3115919 27355/java > > tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:42333 > ESTABLISHED 0 3736664 27355/java > > tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:49796 > ESTABLISHED 0 3115925 27355/java > > tcp 0 0 ::ffff:172-31-30-51:7077 ::ffff:172-31-30-51:42290 > ESTABLISHED 0 3115923 27355/java > > > $ ps -aux | grep 27355 > > Warning: bad syntax, perhaps a bogus '-'? See > /usr/share/doc/procps-3.2.8/FAQ > > ec2-user 23867 0.0 0.0 110404 872 pts/0 S+ 02:06 0:00 grep 27355 > > root 27355 0.5 6.7 3679096 515836 ? Sl Nov26 107:04 > /usr/java/latest/bin/java -cp > /root/spark/sbin/../conf/:/root/spark/lib/spark-assembly-1.5.1-hadoop1.2.1.jar:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/ephemeral-hdfs/conf/ > -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip > ec2-54-215-217-122.us-west-1.compute.amazonaws.com --port 7077 > --webui-port 8080 > > From: Jakob Odersky <joder...@gmail.com> > Date: Thursday, December 10, 2015 at 5:55 PM > To: Andrew Davidson <a...@santacruzintegration.com> > Cc: "user @spark" <user@spark.apache.org> > Subject: Re: Warning: Master endpoint spark://ip:7077 was not a REST > server. Falling back to legacy submission gateway instead. > > Is there any other process using port 7077? > > On 10 December 2015 at 08:52, Andy Davidson <a...@santacruzintegration.com > > wrote: > >> Hi >> >> I am using spark-1.5.1-bin-hadoop2.6. Any idea why I get this warning. >> My job seems to run with out any problem. >> >> Kind regards >> >> Andy >> >> + /root/spark/bin/spark-submit --class >> com.pws.spark.streaming.IngestDriver --master spark:// >> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 >> --total-executor-cores 2 --deploy-mode cluster >> hdfs:///home/ec2-user/build/ingest-all.jar --clusterMode --dirPath week_3 >> >> Running Spark using the REST application submission protocol. >> >> 15/12/10 16:46:33 WARN RestSubmissionClient: Unable to connect to server >> spark://ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077. >> >> Warning: Master endpoint >> ec2-54-205-209-122.us-west-1.compute.amazonaws.com:7077 was not a REST >> server. Falling back to legacy submission gateway instead. >> > >
