Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4509#issuecomment-73756818
Hey @growse,
I tested on a kerberos cluster and did see what you mention in the bug.
Thanks for that explanation.
I think I was a little too paranoid with the original patch, mainly to
cover the case of standalone. Since then, there has been a change in standalone
mode that should make this fix a lot simpler.
Basically, standalone mode now creates a per-app directory similarly to
Yarn. So the code can get away with just changing the permissions of that
"root" directory instead of every temp directory it creates.
Talking in code, this means that `Utils.createDirectory` doesn't need to do
`chmod 700` anymore. Instead, you can do it in
`Utils.getOrCreateLocalRootDirs`. That method already singles out Yarn and
doesn't change the original permissions. We just need to add the chmod for the
non-Yarn case.
That should cover both the driver in client mode (which calls that method
to create a local temp directory, and thus is not under the control of the NM
when running on Yarn) and also the containers in standalone and mesos.
(In fact, there's a little redundancy between the work done by
`Worker.scala` and `Utils.scala` now for standalone mode, but that shouldn't
cause any problems aside from looking a little weird when you look at the raw
files. We can clean that up separately if desired.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
