How to use Fair Scheduler Pools

2023-03-08 Thread 李杰
I have two questions to ask: I wrote a demo referring to the official website(https://spark.apache.org/docs/latest/job-scheduling.html), but it didn't meet my expectations. I don't know if there was a problem with my writing.I hope that when I use the following fairscheduler.xml, pool1 always

How to use Fair Scheduler Pools

2023-03-08 Thread 李杰
I have two questions to ask: I wrote a demo referring to the official website(https://spark.apache.org/docs/latest/job-scheduling.html), but it didn't meet my expectations. I don't know if there was a problem with my writing.I hope that when I use the following fairscheduler.xml, pool1 alw

Re: Spark fair scheduler pools vs. YARN queues

2017-04-05 Thread Nicholas Chammas
> > grrr... s/your/you're/ > > On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra > wrote: > > Your mixing up different levels of scheduling. Spark's fair scheduler > pools are about scheduling Jobs, not Applications; whereas YARN queues with > Spark are about scheduling

Re: Spark fair scheduler pools vs. YARN queues

2017-04-05 Thread Mark Hamstra
evant/useful in this context? > > On Wed, Apr 5, 2017 at 3:54 PM Mark Hamstra > wrote: > >> grrr... s/your/you're/ >> >> On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra >> wrote: >> >> Your mixing up different levels of scheduling. Spark's

Re: Spark fair scheduler pools vs. YARN queues

2017-04-05 Thread Nicholas Chammas
rrr... s/your/you're/ > > On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra > wrote: > > Your mixing up different levels of scheduling. Spark's fair scheduler > pools are about scheduling Jobs, not Applications; whereas YARN queues with > Spark are about scheduling Applic

Re: Spark fair scheduler pools vs. YARN queues

2017-04-05 Thread Mark Hamstra
grrr... s/your/you're/ On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra wrote: > Your mixing up different levels of scheduling. Spark's fair scheduler > pools are about scheduling Jobs, not Applications; whereas YARN queues with > Spark are about scheduling Applications, not Jo

Re: Spark fair scheduler pools vs. YARN queues

2017-04-05 Thread Mark Hamstra
Your mixing up different levels of scheduling. Spark's fair scheduler pools are about scheduling Jobs, not Applications; whereas YARN queues with Spark are about scheduling Applications, not Jobs. On Wed, Apr 5, 2017 at 12:27 PM, Nick Chammas wrote: > I'm having trouble unde

Spark fair scheduler pools vs. YARN queues

2017-04-05 Thread Nick Chammas
I'm having trouble understanding the difference between Spark fair scheduler pools <https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools> and YARN queues <https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>. Do they c

Re: Fair Scheduler Pools with Kafka Streaming

2016-02-16 Thread Sebastian Piu
Yes it is related to concurrentJobs, so you need to increase that. Salt that will mean that if you get overlapping batches then those will be executed in parallel too On Tue, 16 Feb 2016, 18:33 p pathiyil wrote: > Hi, > > I am trying to use Fair Scheduler Pools with Kafka Stream

Fair Scheduler Pools with Kafka Streaming

2016-02-16 Thread p pathiyil
Hi, I am trying to use Fair Scheduler Pools with Kafka Streaming. I am assigning each Kafka partition to its own pool. The attempt is to give each partition an equal share of compute time irrespective of the number of messages in each time window for each partition. However, I do not see fair

Fair Scheduler Pools

2015-02-24 Thread pnpritchard
Hi, I am trying to use the fair scheduler pools (http://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools) to schedule two jobs at the same time. In my simple example, I have configured spark in local mode with 2 cores ("local[2]"). I have also configured tw