Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2432#issuecomment-56756072
Actually, I have one more question / thought:
Someone has discovered a similar issue in HistoryServer where the current
code erroneously assumes that application names are unique:
https://issues.apache.org/jira/browse/SPARK-3610. I think the right approach
here is to use applicationIds. If we do that, though, we'll run up against the
same problem that we hit here, where some task scheduler backends don't provide
application ids and we end up using a random value. In those cases, it would
be nice if we used the _same_ random unique value that we use for the metrics
system.
What do you think about refactoring the `TaskScheduler` interface so that
`applicationId()` is no longer optional, and doing the same thing in
`SchedulerBackend`? We could add a note to the Scaladoc to say that
applicationIds can be assumed to be unique and change the default
implementation in `SchedulerBackend.scala` to return a random identifier.
After doing this, we could add a `getApplicationId()` method to
`SparkContext` to publicly-expose the applicationId and use this method when
configuring the metric server and in the HistoryServer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
