Public bug reported: Earlier when we added config reloading functionality we made provision for the workers who were performing existing tasks should complete those before terminating them. The sequence was likely below;
On receipt of a SIGHUP signal the master process will: * reload the configuration * send a SIGHUP to the original workers * start (a potentially different number of) new workers with the new configuration * its listening socket will *not* be closed On receipt of a SIGHUP signal each original worker process will: * close the listening socket so as not to accept new requests * complete any in-flight requests * complete async requests (V1 create with copy-from option and V2 task api) * exit Recently while working on one of the functionality I found that the workers are not waiting to complete any in-flight requests due to some errors. Following are some important logs; [1] Logs for upload/import request which gets terminated on reload/sighup [2] Reproducer [1] https://paste.opendev.org/show/bTjORjtLtbhUe93xQFDu/ [2] https://paste.opendev.org/show/bFqCD4U4P0Q8MyIrsO4J/ ** Affects: glance Importance: High Status: New ** Changed in: glance Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1978971 Title: Graceful shutdown not working for wsgi and uwsgi Status in Glance: New Bug description: Earlier when we added config reloading functionality we made provision for the workers who were performing existing tasks should complete those before terminating them. The sequence was likely below; On receipt of a SIGHUP signal the master process will: * reload the configuration * send a SIGHUP to the original workers * start (a potentially different number of) new workers with the new configuration * its listening socket will *not* be closed On receipt of a SIGHUP signal each original worker process will: * close the listening socket so as not to accept new requests * complete any in-flight requests * complete async requests (V1 create with copy-from option and V2 task api) * exit Recently while working on one of the functionality I found that the workers are not waiting to complete any in-flight requests due to some errors. Following are some important logs; [1] Logs for upload/import request which gets terminated on reload/sighup [2] Reproducer [1] https://paste.opendev.org/show/bTjORjtLtbhUe93xQFDu/ [2] https://paste.opendev.org/show/bFqCD4U4P0Q8MyIrsO4J/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1978971/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
