hi, dear
Thanks for the response. Some comments below. and yes, I am using spark on
yarn.
1. The release doc of spark says multi jobs can be submitted in one application
if the jobs(actions) are submit by different threads. I wrote some java thread
code in driver, one action in each thread, and the stages are run concurrently
which is observed on stages UI.
In my understanding the DAGscheduler generates different graph for each action.
Not sure correct or not.
Originally I was hoping the sparkcontext can generate different jobs for
none-relevant actions, but never try it successfully.
2. If DAGscheduler generates graph as below, can 1 and 2 run concurrently?
3. I want to reterive the original data out of RDD and have other computation
on the data. Like get the value of tempreture or other data, and works on them.
35597...@qq.com
From: linkpatrickliu
Date: 2014-08-29 14:01
To: user
Subject: RE: The concurrent model of spark job/stage/task
Hi,
Please see the answers following each question. If there's any mistake, please
let me know. Thanks!
I am not sure which mode you are running. So I will assume you are using
spark-submit script to submit spark applications to spark
cluster(spark-standalone or Yarn)
1. how to start 2 or more jobs in one spark driver, in java code.. I wrote 2
actions in the code, but the job still staged in index 0, 1, 2, 3... looks they
run secquencly.
A spark application is a job, you init the application by create a
SparkContext. The SparkContext will init the driver program for you.
So if you want to run multiple jobs simultaneously, you have to split the jobs
into different applications, and submit each of them.
The driver program is like an ApplicationMaster in yarn. It translate the spark
application into a DAG graph, and schedule each stage to workers. Each stage
consists of multiple Tasks.
The driver program handles the life cycle of a spark application.
2. are the stages run currently? because they always number in order 0, 1. 2.
3.. I obverserved on the spark stage UI.
No. Stages will run sequentially. It's a DAG graph, each stage depends on its
parent.
3. Can I retrieve the data out of RDD? like populate a pojo myself and compute
on it.
Not sure what you mean?
You can only retrieve a RDD related with your own SparkContext.
But once a spark application is finished, the SparkContext is released. RDDs
related with the SparkContext are released too.
Best regards,
Patrick Liu
Date: Thu, 28 Aug 2014 18:35:44 -0700
From: [hidden email]
To: [hidden email]
Subject: The concurrent model of spark job/stage/task
hi, guys
I am trying to understand how spark work on the concurrent model. I read
below from https://spark.apache.org/docs/1.0.2/job-scheduling.html
quote
" Inside a given Spark application (SparkContext instance), multiple parallel
jobs can run simultaneously if they were submitted from separate threads. By
“job”, in this section, we mean a Spark action (e.g. save, collect) and any
tasks that need to run to evaluate that action. Spark’s scheduler is fully
thread-safe and supports this use case to enable applications that serve
multiple requests (e.g. queries for multiple users)."
I searched everywhere but not get:
1. how to start 2 or more jobs in one spark driver, in java code.. I wrote 2
actions in the code, but the job still staged in index 0, 1, 2, 3... looks they
run secquencly.
2. are the stages run currently? because they always number in order 0, 1. 2.
3.. I obverserved on the spark stage UI.
3. Can I retrieve the data out of RDD? like populate a pojo myself and compute
on it.
Thanks in advance, guys.
[hidden email]
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/The-concurrent-model-of-spark-job-stage-task-tp13083.html
To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML
View this message in context: RE: The concurrent model of spark job/stage/task
Sent from the Apache Spark User List mailing list archive at Nabble.com.
