Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1486#issuecomment-49915888
Agree with Kay; given the code there, I'm sceptical that your current
solution works, since the ordering of hosts returned by
`getPreferredLocations()` doesn't seem to matter.
One way to abuse the current code without large API changes would be to
have `getPreferredLocations()` return *just* the hosts that have cached data
when data is cached. That means that, in most cases when data is cached, there
would be only one "preferred" host for each partition, so that might have
unwanted side-effects. The locality would still be NODE_LOCAL even though, in
practice, it's more like PROCESS_LOCAL.
Otherwise, choice 2 seems to be the closer to what I think would be ideal.
It seems weird to return (host, TaskLocality) though, since it wouldn't make
sense to return ("x", RACK_LOCAL) for example. But I think this is the most
flexible solution.
3 seems like it's just 2 in disguise; instead of enforcing type-safety in
the API, you'd have to do runtime checks that the returned strings make sense.
Probably not a huge deal in a private API, but as you say, a little ugly.
4 sounds like overkill (doesn't seem to add any benefit over 2 or 3, and
makes the API more complicated).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
