Two things we have done here. When troubleshooting the shutdown deadlock issue, we ran the app in docker container with remote debugging and I could step through all of the Samza (and our) related shutdown code. After the upgrade, none of the breakpoints are hit. Not even our own shutdownHook nor spring context shutdownHook.
As for logs, before upgrade we saw all shutdown related logging, after upgrade there is absolutely 0 logs. The JVM just immediately stops. Thanks in advance. I'll have a look at the ticket also. Thunder On Tue, Jan 7, 2020 at 1:55 PM Brett Konold <bkon...@linkedin.com.invalid> wrote: > Thunder, > > How were you able to determine that the shutdown hooks are not being > called? > > If you're able to share any of your shutdown logs from before and after > your 1.3 upgrade, that would be helpful while I try to reproduce this issue > myself. > > Brett > ________________________________ > From: Brett Konold <bkon...@linkedin.com> > Sent: Tuesday, January 7, 2020 1:25 PM > To: dev@samza.apache.org <dev@samza.apache.org> > Subject: Re: Problem: upgrade 1.2 to 1.3 caused loss of clean shutdown on > SIGTERM > > Hey Thunder, > > Thanks for reporting. Taking a look into this and will get back to you > when I have something. > > Brett > ________________________________ > From: Thunder Stumpges <thunder.stump...@gmail.com> > Sent: Monday, January 6, 2020 6:54 PM > To: dev@samza.apache.org <dev@samza.apache.org> > Subject: Problem: upgrade 1.2 to 1.3 caused loss of clean shutdown on > SIGTERM > > We are attempting to upgrade from samza 1.2 to 1.3 in hopes of fixing > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSAMZA-2198&data=02%7C01%7Cbkonold%40linkedin.com%7C3d5e0f0bf97e4cc633de08d7931d06ad%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637139625187723288&sdata=uwKMfKxQmS9uWS62Hew814P%2Fja7FpaViNTVzYrp03Ok%3D&reserved=0 > where there was a deadlock > in the shutdown code which prevented completing a clean shutdown. > > After the upgrade, it appears like NONE of the shutdown hooks / code are > being called and it just immediately shuts down. > > We are running stand alone Low Level Tasks with LocalApplicationRunner / > ZkJobCoordinator in Docker / K8s. > > When killing from docker for testing, we use docker kill -s SIGTERM > <container> to send SIGTERM instead of SIGKILL. This was working in samza > 1.2 (other than the deadlock from the above issue). > > Any ideas what changed? > > Thanks, > Thunder >
