> On Sep 19, 2015, at 4:44 PM, Abdollahian Noghabi, Shadi > <abdol...@illinois.edu> wrote: > > As far as I know there is no notion of job version, and you should not run > two instances of a job with the same (job name , job id) pair since it will > mess up the checkpoint and etc. The job id is used to run the same job with > different instances at the same time.
I am aware of this. See my original post. > > However I think it might be useful to know how many times you have killed the > same job (i.e. same job name and id) and you are restarting it again, by > keeping a version. One way you can do this is to write a job.version to the > set of properties as you said. > > In the newer Samza version with the Job coordinator, you can fix this issue > as following. When submitting a job, in the bootstrap phase that it reads the > coordinator stream you can find the currently maximum value for the > job.version and then rewrite this property with job.version+1. Then later on > you can read this property from the coordinator stream topic, or from the job > coordinator server providing the whole list of config. However, I am not sure > how to do this in the older releases. I don’t want some sort of ‘job restart generation’. I’m thinking of a version like a build timestamp that is baked into the config file at packaging time. The code changed, and I want to replace the currently running job with the new (hopefully better) job, and pick up where the old one left off. My cluster monitoring scripts need to realize that the currently running job is old code, kill it, and replace it with the new job code. > Shadi > > > >> On Sep 19, 2015, at 6:09 PM, Richard Lee <rd...@tivo.com> wrote: >> >> I suppose it would be possible to add a custom ‘job.version’ field to the >> samza job properties file, and then query for it via the REST /config >> endpoint on the ApplicationMananger, but I’m unclear how I find the RPC port >> for the ApplicationManager from the ResourceManager. The ResourceManager >> seems to only list the UI tracker endpoint proxy in its application REST >> endpoint. >> >> I suppose I could scrape the port out of the HTML, but it seems like there >> should be a better way. >> >> Richard >> >>> On Sep 19, 2015, at 3:25 PM, Richard Lee <rd...@tivo.com> wrote: >>> >>> Hi there- >>> >>> How do people track which version of a samza job is running in yarn? The >>> job name and job id can’t be used, as they are used to create the >>> checkpoint topic, etc. I’m looking for a way of determining if the current >>> job running in yarn is the latest version, and if not, kill it and launch a >>> newer version, picking up where the previous version left off. >>> >>> There seems to be no ‘job version’ field anywhere obvious in either samza >>> or yarn. >>> >>> Is there another approach I should use? >>> >>> Richard >>> >> >
smime.p7s
Description: S/MIME cryptographic signature
