Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/6291#issuecomment-137161186
@squito Sure, if you want to finish off the test, that would be great. I
haven't been prioritizing that since the bug itself was fairly low priority and
I was wanting to use some of the new helpers that have been working their way
into the DAGSchedulerSuite. If this has now become a customer issue for you,
then that changes the prioritization a bit.
Your understanding of the prior behavior is essentially correct, but I'd
express it somewhat differently. Tasks for a Stage that was previously part of
a Job that is no longer active would be re-submitted as though they were part
of the prior Job and with no properties set. Since properties are what are
used to set an other-than-default scheduling pool, this would affect FAIR
scheduler usage, but it would also affect anything else that depends on the
settings of the properties (which would be just user code at this point, since
Spark itself doesn't really use the properties for anything else other than Job
Group and Description, which end up in the WebUI, can be used to kill by
JobGroup, etc.) Even the default, FIFO scheduling would be affected, however,
since the resubmission of the Tasks under the earlier jobId would effectively
give them a higher priority/greater urgency than the ActiveJob that now
actually needs them.
In any event, the Tasks would generate correct results, and that's why this
hasn't been a higher priority before now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
