Two weeks ago I have published a blogpost about our experiences running 24/7
Spark Streaming applications on YARN in production:
https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/
<https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/>
Amongst oth
Have you used awaitTermination() on your ssc ? --> Yes, i have used that.
Also try setting the deployment mode to yarn-client. --> Is this not
supported on yarn-cluster mode? I am trying to find root cause for
yarn-cluster mode.
Have you tested graceful shutdown on yarn-cluster mode?
On Fri, May
Rakesh
Have you used awaitTermination() on your ssc ?
If not , dd this and see if it changes the behavior.
I am guessing this issue may be related to yarn deployment mode.
Also try setting the deployment mode to yarn-client.
Thanks
Deepak
On Fri, May 13, 2016 at 10:17 AM, Rakesh H (Marketing Pla
Ping!!
Has anybody tested graceful shutdown of a spark streaming in yarn-cluster
mode?It looks like a defect to me.
On Thu, May 12, 2016 at 12:53 PM Rakesh H (Marketing Platform-BLR) <
rakes...@flipkart.com> wrote:
> We are on spark 1.5.1
> Above change was to add a shutdown hook.
> I am not addi
We are on spark 1.5.1
Above change was to add a shutdown hook.
I am not adding shutdown hook in code, so inbuilt shutdown hook is being
called.
Driver signals that it is going to to graceful shutdown, but executor sees
that Driver is dead and it shuts down abruptly.
Could this issue be related to y
This is happening because spark context shuts down without shutting down
the ssc first.
This was behavior till spark 1.4 ans was addressed in later releases.
https://github.com/apache/spark/pull/6307
Which version of spark are you on?
Thanks
Deepak
On Thu, May 12, 2016 at 12:14 PM, Rakesh H (Mar
Yes, it seems to be the case.
In this case executors should have continued logging values till 300, but
they are shutdown as soon as i do "yarn kill .."
On Thu, May 12, 2016 at 12:11 PM Deepak Sharma
wrote:
> So in your case , the driver is shutting down gracefully , but the
> executors are
So in your case , the driver is shutting down gracefully , but the
executors are not.
IS this the problem?
Thanks
Deepak
On Thu, May 12, 2016 at 11:49 AM, Rakesh H (Marketing Platform-BLR) <
rakes...@flipkart.com> wrote:
> Yes, it is set to true.
> Log of driver :
>
> 16/05/12 10:18:29 ERROR yar
Yes, it is set to true.
Log of driver :
16/05/12 10:18:29 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
16/05/12 10:18:29 INFO streaming.StreamingContext: Invoking
stop(stopGracefully=true) from shutdown hook
16/05/12 10:18:29 INFO scheduler.JobGenerator: Stopping JobGenerator graceful
Hi Rakesh
Did you tried setting *spark.streaming.stopGracefullyOnShutdown to true *for
your spark configuration instance?
If not try this , and let us know if this helps.
Thanks
Deepak
On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) <
rakes...@flipkart.com> wrote:
> Issue i a
Issue i am having is similar to the one mentioned here :
http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn
I am creating a rdd from sequence of 1 to 300 and creating streaming RDD
out of it.
val rdd = ssc.sparkContext.parallelize(1 to 300)
va
Hi all
Producing more data into Kafka is not effective in my situation,
because the speed of reading Kafka is consistent. I will adopt Saiai's
suggestion to add more receivers.
Kyle
2015-04-30 14:49 GMT+08:00 Saisai Shao :
> From the chart you pasted, I guess you only have one receiver with
>From the chart you pasted, I guess you only have one receiver with storage
level two copies, so mostly your taks are located on two executors. You
could use repartition to redistribute the data more evenly across the
executors. Also add more receiver is another solution.
2015-04-30 14:38 GMT+08:0
t; My environment info]Kyle Lin ---2015/04/30 14:39:32---Hi all My
> environment info
>
> From: Kyle Lin
> To: "user@spark.apache.org"
> Date: 2015/04/30 14:39
> Subject: The Processing loading of Spark streaming on YARN is not in
> balance
> -
/sets
From: Kyle Lin
To: "user@spark.apache.org"
Date: 2015/04/30 14:39
Subject:The Processing loading of Spark streaming on YARN is not in
balance
Hi all
My environment info
Hadoop release version: HDP 2.1
Kakfa: 0.8.1.2.1.4.0
Spark: 1.1.0
My question:
Hi all
My environment info
Hadoop release version: HDP 2.1
Kakfa: 0.8.1.2.1.4.0
Spark: 1.1.0
My question:
I ran Spark streaming program on YARN. My Spark streaming program will
read data from Kafka and doing some processing. But, I found there is
always only ONE executor under processing. As
I’m looking at various HA scenarios with Spark streaming. We’re currently
running a Spark streaming job that is intended to be long-lived, 24/7. We see
that if we kill node managers that are hosting Spark workers, new node managers
assume execution of the jobs that were running on the stopped
have you fixed this issue ?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-on-Yarn-Input-from-Flume-tp11755p22055.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-on-Yarn-tp19093.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
to flume.
>
> And then I run the FlumeEventCount example, when I run this example on
> Yarn, it also can not receive data from flume.
>
> And I will be very pleasure if some one can help me.
>
>
> XiaoQinyu
>
>
>
> --
> View this message in context:
> http://apach
lp me.
XiaoQinyu
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-on-Yarn-Input-from-Flume-tp11755.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---
21 matches
Mail list logo
