[
https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938813#comment-15938813
]
Travis Hegner commented on SPARK-15176:
---------------------------------------
I'd like to have this feature available as well for a slightly different use
case.
Currently we have an application that reconciles data from a relational
database. It would be ideal to place any job within this app that touches the
RDB into a resource capped pool ("maxShare" was what I instinctively looked
for), to prevent overloading the remote RDB. The app has many other stages of
data manipulations which would benefit from utilizing a larger share of the
cluster, without the risk of hurting the RDB.
Limiting the number of executors and cores per executor works to prevent
overloading the RDB, but also limits the entire application, causing it to run
slower, and leaving potential idle resources on the table. Currently, the only
other option I can think of would be to split the RDB pull into a separate app
that persists the data to disk temporarily. This seems it would make the
overall process take even more time, however.
Is anyone still working this issue? Should I look at porting the existing PR to
2.1.x branch?
> Job Scheduling Within Application Suffers from Priority Inversion
> -----------------------------------------------------------------
>
> Key: SPARK-15176
> URL: https://issues.apache.org/jira/browse/SPARK-15176
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.1
> Reporter: Nick White
>
> Say I have two pools, and N cores in my cluster:
> * I submit a job to one, which has M >> N tasks
> * N of the M tasks are scheduled
> * I submit a job to the second pool - but none of its tasks get scheduled
> until a task from the other pool finishes!
> This can lead to unbounded denial-of-service for the second pool - regardless
> of `minShare` or `weight` settings. Ideally Spark would support a pre-emption
> mechanism, or an upper bound on a pool's resource usage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
