Hi Mike! Flink does in fact cache jar files in the "blob server". But these are cached subject to the following conditions:
- No caching across "sessions", meaning start/stop of the cluster/jobmanager. If you run the per-job-yarn setup, the job does not cache anything. - Files are cached under a content hash, meaning as soon as the contents changes, the artifact is not reused. So if you actually change the jar file, no caching should happen. I cannot really explain what you are observing and have never seen that myself... Stephan On Mon, Jul 31, 2017 at 9:00 PM, Mike Accola <macc...@us.ibm.com> wrote: > No, I did not explicitly create an uber-jar. The mylib1.jar is very > light. It only contains my main application class (including > ProcessFunction). > > I have been specifying --classpath option on my flink run command to pull > in the mylib2.jar . > > Plus, I have been rebuilding mylib1.jar frequently just to be safe and it > hasn't made a difference. > > > Mike Accola > macc...@us.ibm.com > > > > > > From: Eron Wright <eronwri...@gmail.com> > To: dev@flink.apache.org > Date: 07/31/2017 01:47 PM > Subject: Re: Class Cache > > > > A Flink program is typically packaged as an 'uber-jar' containing its > dependencies. The Flink quickstart project illustrates this (see the use > of the shading plugin in pom.xml). Based on your description, the > classes > of mylib2.jar were copied into mylib1.jar when the latter was built. Try > rebuilding mylib1.jar to effect the change. > > -Eron > > On Mon, Jul 31, 2017 at 11:18 AM, Mike Accola <macc...@us.ibm.com> wrote: > > > Are classes cached somewhere in flink? I am running in a very basic, > > local environment on Linux (start_local.sh). I've somehow gotten my > > environment into a strange state that I don't understand. I feel like I > > am overlooking something simple, but I've checked everything I can think > > of. > > > > My main flink application with a ProcessFunction is embedded in > > mylib1.jar. Within my ProcessFunction I use another class that is > > embedded in mylib2.jar. > > > > When I made changes to function in mylib2.jar and rebuilt the jar, I > > realized the changes weren't taking affect. In fact, I then delete > > mylib2.jar entirely and my application still worked. I can't figure out > > where my application is picking up the function contained in mylib2.jar. > I > > have checked any temp directories, library paths, etc. I have > repeatedly > > stopped/started my flink environment just to be safe. > > > > I tried adding -verbose:class to env.java.opts. It output a lot of > class > > loading info to the stdout log, but there were no references to my class > > in mylib2.jar. > > > > This has to be caching this code somehow whether it is in flink or in > the > > jvm. Any ideas what could be happening or how to debug this further? > > > > Thanks > > > > > > > > > > >
