It is an interesting requirement. Usually, we suggest our users to add
these dependencies in
lib directory or ship via --yarnship cli option.
If you really do not want to do this, I think you could ship an updated
yarn-site.xml, set "yarn.application.classpath"
including the mapreduce jars. Then a
Hi Maxim,
Both Yarn per-job and session cluster should work. Since before the
JobManager and TaskManager
launcher, Yarn NodeManager could guarantee that all the local resources
have been localized and
accessible.
If do not want to use the getResource to read the file and use File
interface instead
Hi Yang,
I've just tried your suggestions, but, unfortunately, in yarn per job mode
it doesn't work, both commands return null.
I double checked that file is shipped to yarn container, but I feel that it
happens later in process.
At the moment I'm reading file with File interface, instead of getti
Hi Maxim,
I have verified that the following two ways could both work.
getClass().getClassLoader().getResource("lib/job.properties")
getClass().getClassLoader().getResource("job.properties")
Best,
Yang
Maxim Parkachov 于2020年2月17日周一 下午6:47写道：
> Hi Yang,
>
> thanks, this explains why classpath
Hi Yang,
thanks, this explains why classpath behavior changed, but now I struggle to
understand how I could overwrite resource, which is already shipped in job
jar.
Before I had job.properties files in JAR in under
resources/lib/job.properties
for local development and deploying on cluster it was
Hi Maxim Parkachov,
The users files also have been shipped to JobManager and TaskManager.
However, it
is not directly added to the classpath. Instead, the parent directory is
added to the
classpath. This changes are to make resource classloading work. You could
check more
information here[1].
[1
