Hi, I think this issue can be closed, because a probable reason have been found. It seems that https://github.com/apache/flink/blob/release-1.6.1-rc1/flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnJobClusterEntrypoint.java (https://link.getmailspring.com/link/[email protected]/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fflink%2Fblob%2Frelease-1.6.1-rc1%2Fflink-yarn%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fyarn%2Fentrypoint%2FYarnJobClusterEntrypoint.java&recipient=dXNlckBmbGluay5hcGFjaGUub3Jn) tracks current working directory location and assembles configuration with respect to various .conf files in this directory. Restarting job from directory, that doesn't contain any *.conf files results in proper application functioning. Cheers, Dmitry.
On Jun 10 2020, at 12:35 pm, Dmitry Mikhailov <[email protected]> wrote: > Hi > > I'm struggling with updating a flink application. While developing, I was > using a dev application configuration (application.conf). While updating > application.conf, and rerunning job in a production mode, I caught an > exception that stated, that i'm having a dev application configuration. Looks > that application.conf was somehow cached in a cluster. > Application configuration includes section of dynamic list of > RichMapFunction`s. > These map steps are assembled into a pipeline before starting a job. > Dev configuration have some map steps, that are not enabled in the production > configuration. > For example dev pipeline looks like: source -> dmpkit-cp-base -> > dmpkit-cp-attribute-mapper -> dmpkit-cp-tm -> sink > Production pipeline lacks first mapper: source -> dmpkit-cp-attribute-mapper > -> dmpkit-cp-tm -> sink > > Modifying application.conf from dev to prod and rerunning job results in > having dev configuration enabled, which causes errors. > Job is submitted with the parameters: > sudo -Eu dmpkit /usr/lib/flink/bin/flink run \ > -m yarn-cluster \ > --detached \ > --yarnname cleverdata-dmpkit-customerjourney-http-job \ > --yarnship /tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job \ > --class > ru.cleverdata.dmpkit.customerjourney.http.job.impl.ApplicationBootstrap \ > -C > file:///tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/jackson-databind-2.9.5.jar > \ > .... (other dependencies jars) \ > -C > file:///tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/etc.zip > \ > /tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/cleverdata-dmpkit-customerjourney-http-job-impl_2.11-2.18.0-93.jar > > application.conf is zipped into etc.zip archive and shipped in both > --yarnship /tmp/... directory and -C file:///.../etc.zip file. > Inspection of all deployed jars and configurations shows none dev > configurations. Started application shows that dev configuration loaded. App > crashes. > > However, some tracks of dev configuration can be found in the hdfs directory > of a deployed job: > some tmp file states, that dev pipeline map functions enabled > > hdfs dfs -cat > /user/dmpkit/.flink/application_1586040264218_59021/application_1586040264218_590217815615186225961213.tmp > | strings | less > contains some mentions of dmpkit-cp-base, which is dev-only map step. > I have no idea how this map function got into pipeline, when, as I said > earlier, it is not enabled in configuration and non of jars and\or *.conf > files contains this configuration. > I'm stuck with searching where this invalid configuration could be cached. > > Cheers, > Dmitry.
