I am using the same exact code: https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java
Submitting like this: yarn: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/spark-submit --class org.sony.spark.stream.test.JavaRecoverableNetworkWordCount --master yarn-client --total-executor-cores 3 ./spark-streaming-test-0.0.1-SNAPSHOT-jar-with-dependencies.jar localhost 9999 /user/ec2-user/checkpoint/ /user/ec2-user/out local: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/spark-submit --class org.sony.spark.stream.test.JavaRecoverableNetworkWordCount --master spark://localhost:9966 --total-executor-cores 3 ./spark-streaming-test-0.0.1-SNAPSHOT-jar-with-dependencies.jar localhost 9999 /user/ec2-user/checkpoint/ /user/ec2-user/out Even though I am running as local I see it being scheduled and managed by yarn. On Mon, Aug 10, 2015 at 12:56 PM, Tathagata Das <t...@databricks.com> wrote: > Is it receiving any data? If so, then it must be listening. > Alternatively, to test these theories, you can locally running a spark > standalone cluster (one node standalone cluster in local machine), and > submit your app in client mode on that to see whether you are seeing the > process listening on 9999 or not. > > On Mon, Aug 10, 2015 at 12:43 PM, Mohit Anchlia <mohitanch...@gmail.com> > wrote: > >> I've verified all the executors and I don't see a process listening on >> the port. However, the application seem to show as running in the yarn UI >> >> On Mon, Aug 10, 2015 at 11:56 AM, Tathagata Das <t...@databricks.com> >> wrote: >> >>> In yarn-client mode, the driver is on the machine where you ran the >>> spark-submit. The executors are running in the YARN cluster nodes, and the >>> socket receiver listening on port 9999 is running in one of the executors. >>> >>> On Mon, Aug 10, 2015 at 11:43 AM, Mohit Anchlia <mohitanch...@gmail.com> >>> wrote: >>> >>>> I am running as a yarn-client which probably means that the program >>>> that submitted the job is where the listening is also occurring? I thought >>>> that the yarn is only used to negotiate resources in yarn-client master >>>> mode. >>>> >>>> On Mon, Aug 10, 2015 at 11:34 AM, Tathagata Das <t...@databricks.com> >>>> wrote: >>>> >>>>> If you are running on a cluster, the listening is occurring on one of >>>>> the executors, not in the driver. >>>>> >>>>> On Mon, Aug 10, 2015 at 10:29 AM, Mohit Anchlia < >>>>> mohitanch...@gmail.com> wrote: >>>>> >>>>>> I am trying to run this program as a yarn-client. The job seems to be >>>>>> submitting successfully however I don't see any process listening on this >>>>>> host on port 9999 >>>>>> >>>>>> >>>>>> https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java >>>>>> Active Jobs (2)Job IdDescriptionSubmittedDurationStages: >>>>>> Succeeded/TotalTasks (for all stages): Succeeded/Total1foreachRDD at >>>>>> JavaRecoverableNetworkWordCount.java:112 >>>>>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=1>2015/08/10 >>>>>> 13:27:3651 s0/2 >>>>>> 0/2 >>>>>> 0start at JavaRecoverableNetworkWordCount.java:152 >>>>>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=0>2015/08/10 >>>>>> 13:27:3551 s0/2 >>>>>> 0/70 >>>>>> >>>>> >>>>> >>>> >>> >> >
