Hi,
I was doing programmatic submission of Spark yarn jobs and I saw code in
ClientBase.getDefaultYarnApplicationClasspath():
val field = classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
MRJobConfig doesn't have this field so the created launch env is incomplete.
Workaround is to set yarn.application.classpath with the value from
YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
This results in having the spark job hang if the submission config is different
from the default config. For example, if my resource manager port is 8050
instead of 8030, then the spark app is not able to register itself and stays in
ACCEPTED state.
I can easily fix this by changing this to YarnConfiguration instead of
MRJobConfig but was wondering what the steps are for submitting a fix.
Thanks,
Ron
Sent from my iPhone
