I should also note that the above steps did get the Flink JobManager and
TaskManagers to save their tmp web dashboard files to /my/tmp/ and to show
in the Dashboard that the taskmanager.tmp.dirs property had been properly
set to /my/tmp/, but the tmp files I wrote in my jobs stubbornly wrote to
/tmp/ anyway.
On Fri, Jul 28, 2017 at 4:55 PM, Chris Hebert <
chris.hebert-...@digitalreasoning.com> wrote:
> Hi,
>
> My jobs create tmp files like so:
>
> java.nio.file.Path tmpFilePath = java.nio.file.Files.createTempFile("tmpFile",
> "txt");
>
> They currently appear in /tmp/, but I want them somewhere else, say
> /my/tmp/.
>
> The Flink on YARN docs say:
>
> Flink on YARN will overwrite the following configuration parameters
> jobmanager.rpc.address (because the JobManager is always allocated at
> different machines), taskmanager.tmp.dirs (we are using the tmp
> directories given by YARN) and parallelism.default if the number of slots
> has been specified.
>
> How would I specify a different tmp directory for a job without modifying
> my YARN tmp directories?
>
> I tried the taskmanager.tmp.dirs property in conf/flink-conf.yaml anyway,
> that failed.
>
> I appended -Djava.io.tmpdir=/my/tmp/ to JVM_ARGS and all three variations
> of DEFAULT_ENV_JAVA_OPTS in bin/config.sh, that failed.
>
> I passed -Djava.io.tmpdir=/my/tmp/ and variations as arguments to .
> /bin/yarn-session.sh and ./bin/flink run et cetera, that failed.
>
> Odd observation:
> The hadoop.tmp.dir property is set in my core-site.xml to /some/other/tmp/,
> yet Flink writes to /tmp/. My yarn-site.xml specifies no tmp.
>
> Side note:
> My Flink job is a Beam pipeline. I doubt that's relevant, but let me know
> if it is.
>
> Thanks,
> Chris
>
