Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2482#issuecomment-56315058
I've taken another pass at this. This time, I kept AsyncRDDActions but
re-implemented it using `runAsync`, but I'm actually on the fence about that
change. The one difference here is that the asynchronous jobs will now be
submitted with anonymous job groups rather than as part of the calling thread's
job group. This change might be observable by a user who writes a job that
fires off multiple asynchronous actions from a single driver control thread,
then attempts to cancel that thread's job group. Because job groups don't have
any hierarchy / nesting, this would break the cancellation of those jobs.
I'm beginning to get the sense that we might not have much room to change
anything about the implementation of AsyncRDDActions, so maybe we should just
let them be.
@rxin Based on our discussion, I added a check in DAGScheduler to reject
jobs submitted by cancelled threads.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
