+1
Regards,
Dhaval Modi
dhavalmod...@gmail.com
On 29 March 2018 at 19:57, Sidney Feiner <sidney.fei...@startapp.com> wrote:
> Hey,
>
> I have a Spark Streaming application processing some events.
>
> Sometimes, I want to stop the application if a get a specific event. I
> collect the executor's results in the driver and based on those results, I
> kill the StreamingContext using StreamingContext.stop(
> stopSparkContext=true).
>
> When I do that, I can see in the logs that the app is shutting down,
> closing receivers etc.
>
> But when I go to the master's web UI I can still see the app under
> "Running Applications". But if I click it, it says the endpoint doesn't
> exist.
>
> When I check the open processes on the machine, I can see that the job's
> process is still running.
>
> Am I closing the application wrong?
>
>
>
> Those are the logs once I call the stop() method:
>
>
>
> 2018-03-28 11:59:04 INFO KafkaProducer:615 - Closing the Kafka producer
> with timeoutMillis = 9223372036854775807 ms.
>
> 2018-03-28 11:59:04 INFO ReceiverTracker:54 - Sent stop signal to all 1
> receivers
>
> 2018-03-28 11:59:05 INFO BlockManagerInfo:54 - Added
> input-0-1522238344750 in memory on i-va-spark1:59059 (size: 1632.0 B, free:
> 579.2 MB)
>
> 2018-03-28 11:59:05 ERROR ReceiverTracker:70 - Deregistered receiver for
> stream 0: Stopped by driver
>
> 2018-03-28 11:59:05 INFO BlockManagerInfo:54 - Added
> input-0-1522238345000 in memory on i-va-spark1:59059 (size: 272.0 B, free:
> 579.2 MB)
>
> 2018-03-28 11:59:05 INFO TaskSetManager:54 - Finished task 0.0 in stage
> 2.0 (TID 70) in 30213 ms on i-va-spark1 (executor 0) (1/1)
>
> 2018-03-28 11:59:05 INFO TaskSchedulerImpl:54 - Removed TaskSet 2.0,
> whose tasks have all completed, from pool
>
> 2018-03-28 11:59:05 INFO DAGScheduler:54 - ResultStage 2 (start at
> UserLocationHistoryJob.scala:38) finished in 30.213 s
>
> 2018-03-28 11:59:05 INFO ReceiverTracker:54 - All of the receivers have
> deregistered successfully
>
> 2018-03-28 11:59:05 INFO ReceiverTracker:54 - ReceiverTracker stopped
>
> 2018-03-28 11:59:05 INFO JobGenerator:54 - Stopping JobGenerator
> immediately
>
> 2018-03-28 11:59:05 INFO RecurringTimer:54 - Stopped timer for
> JobGenerator after time 1522238340000
>
> 2018-03-28 11:59:05 INFO JobGenerator:54 - Stopped JobGenerator
>
> 2018-03-28 11:59:07 INFO JobScheduler:54 - Stopped JobScheduler
>
> 2018-03-28 11:59:07 INFO StreamingContext:54 - StreamingContext stopped
> successfully
>
> 2018-03-28 11:59:07 INFO BlockManagerInfo:54 - Removed broadcast_5_piece0
> on 10.0.0.243:41976 in memory (size: 2.4 KB, free: 488.4 MB)
>
> 2018-03-28 11:59:07 INFO BlockManagerInfo:54 - Removed broadcast_5_piece0
> on i-va-spark1:59059 in memory (size: 2.4 KB, free: 579.2 MB)
>
> 2018-03-28 11:59:07 INFO BlockManagerInfo:54 - Removed broadcast_4_piece0
> on 10.0.0.243:41976 in memory (size: 23.9 KB, free: 488.4 MB)
>
> 2018-03-28 11:59:07 INFO BlockManagerInfo:54 - Removed broadcast_4_piece0
> on i-va-spark1:59059 in memory (size: 23.9 KB, free: 579.2 MB)
>
> 2018-03-28 11:59:37 WARN QueuedThreadPool:178 -
> SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop
> Thread[SparkUI-171-selector-ServerConnectorManager@478b3e9/2,5,main]
>
> 2018-03-28 11:59:37 WARN QueuedThreadPool:178 -
> SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop
> Thread[SparkUI-172-selector-ServerConnectorManager@478b3e9/3,5,main]
>
> 2018-03-28 11:59:37 WARN QueuedThreadPool:178 -
> SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop
> Thread[SparkUI-169-selector-ServerConnectorManager@478b3e9/0,5,main]
>
> 2018-03-28 11:59:37 WARN QueuedThreadPool:178 -
> SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop
> Thread[SparkUI-170-selector-ServerConnectorManager@478b3e9/1,5,main]
>
> 2018-03-28 13:22:01 INFO DiskBlockManager:54 - Shutdown hook called
>
> 2018-03-28 13:22:01 INFO ShutdownHookManager:54 - Shutdown hook called
>
> 2018-03-28 13:22:01 INFO ShutdownHookManager:54 - Deleting directory
> /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581
>
> 2018-03-28 13:22:01 INFO ShutdownHookManager:54 - Deleting directory
> /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581/userFiles-
> 8a970eec-da41-442b-9ccf-1621b9e9e045
>
>
>
>
>
>
>
> *Sidney Feiner* */* SW Developer
>
> M: +972.528197720 */* Skype: sidney.feiner.startapp
>
>
>
> [image: emailsignature]
>
>
>
>
>
