[
https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951194#comment-15951194
]
Imran Rashid commented on SPARK-20178:
--------------------------------------
Thanks for writing this up Tom.
The only way I see to have a pluggable interface in the current code is to
abstract out the *entire* thing -- DAGScheduler, TSM, TSI. perhaps also CGSB
and OCC. that would be pretty extreme though, I'd only consider that if we
actually have some reason to think we'd come up with a better version (eg. new
abstractions with less shared state).
In addition to not destabilizing the current scheduler, we should also think of
what the migration path would be for enabling these new changes. Will there be
a way for spark to auto-tune? Or will we need to create a number of new confs?
I know everyone hates having a huge set of configuration that needs to be
tuned, but at some point I think its OK if spark works reasonably well on small
clusters by default, and for large clusters you've just got to have somebody
that knows how to configure it carefully.
Another thing to keep in mind is that Spark is used on a huge variety of
workloads. I feel like right now we're very focused on large jobs on big
clusters with long tasks; but spark is also used with very small tasks,
especially streaming. I think all the ideas we're thinking of only effect
behavior after there is a failure, so hopefully it wouldn't matter. But we
need to be careful that we don't introduce complexity which effects performance
even before any failures.
> Improve Scheduler fetch failures
> --------------------------------
>
> Key: SPARK-20178
> URL: https://issues.apache.org/jira/browse/SPARK-20178
> Project: Spark
> Issue Type: Epic
> Components: Scheduler
> Affects Versions: 2.1.0
> Reporter: Thomas Graves
>
> We have been having a lot of discussions around improving the handling of
> fetch failures. There are 4 jira currently related to this.
> We should try to get a list of things we want to improve and come up with one
> cohesive design.
> SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753
> I will put my initial thoughts in a follow on comment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
