€ρ@Ҝ (๏̯͡๏)
Cc: Evo Eftimov; user
Subject: Re: How to increase the number of tasks
just multiply 2-4 with the cpu core number of the node .
2015-06-05 18:04 GMT+08:00 ÐΞ€ρ@Ҝ (๏̯͡๏) :
I did not change spark.default.parallelism,
What is recommended value for it.
On Fri, Jun 5, 2015 at 3
t;>> 1. RDD Partition = (Spark) Task
>>>
>>> 2. RDD Partition != (Spark) Executor
>>>
>>> 3. (Spark) Task != (Spark) Executor
>>>
>>> 4. (Spark) Task = JVM Thread
>>>
>>> 5. (Spark) Executor = JVM in
t;>
>> 4. (Spark) Task = JVM Thread
>>
>> 5. (Spark) Executor = JVM instance
>>
>>
>>
>> *From:* ÐΞ€ρ@Ҝ (๏̯͡๏) [mailto:deepuj...@gmail.com]
>> *Sent:* Friday, June 5, 2015 10:48 AM
>> *To:* user
>> *Subject:* How to increase the
Spark) Task
>
> 2. RDD Partition != (Spark) Executor
>
> 3. (Spark) Task != (Spark) Executor
>
> 4. (Spark) Task = JVM Thread
>
> 5. (Spark) Executor = JVM instance
>
>
>
> *From:* ÐΞ€ρ@Ҝ (๏̯͡๏) [mailto:deepuj...@gmail.com]
> *Sent:* Friday,
instance
From: ÐΞ€ρ@Ҝ (๏̯͡๏) [mailto:deepuj...@gmail.com]
Sent: Friday, June 5, 2015 10:48 AM
To: user
Subject: How to increase the number of tasks
I have a stage that spawns 174 tasks when i run repartition on avro data.
Tasks read between 512/317/316/214/173 MB of data. Even if i
I have a stage that spawns 174 tasks when i run repartition on avro data.
Tasks read between 512/317/316/214/173 MB of data. Even if i increase
number of executors/ number of partitions (when calling repartition) the
number of tasks launched remains fixed to 174.
1) I want to speed up this task.
