Pressed the wrong button when I tried to edit the code example... here's
the correct (simplified) example:
def handle_sigterm(*_: Any) -> None :
"""Shutdown gracefully."""
done_event = server.stop(30)
done_event.wait(30)
print('Stop complete.')
server = grpc.server(
futures.ThreadPoolExecutor(max_workers = options['max_workers']),
)
add_Servicer_to_server(service, server)
server.add_insecure_port(options['address'])
server.start()
signal(SIGTERM, handle_sigterm)
server.wait_for_termination()
I'm deploying my service through kubernetes, which stops pods by first
sending a SIGTERM event, and, if the pod is still alive after a timeout, it
kills it using SIGKILL.
The behavior that I witness:
* the time it takes for the pod to terminate looks a lot like it's using
the full 30s that kubernetes gives it = it ends up getting killed by SIGKILL
* the log for the stop of the service inside the SIGTERM handler never
appears
So I think that one of following things might be happening here (that I can
think of):
1. the handler never gets called because the process gets killed before it
gets to handle it
2. the server takes so long to shut down, that kubernetes ends up killing
it before it gets to logging the message
3. the shutdown event doesn't get set even after the server is done
shutting down, causing kubernetes to kill it without logging the message
I see that the official examples contain a asyncio example for graceful
shutdown, can we maybe get something like for the regular, non-asyncio case
(https://github.com/lidizheng/grpc/tree/master/examples/python/helloworld)?
And for the long run: would it be possible to get something like
wait_for_termination that is capable of handling graceful shutdown
natively, without the developers having to add something on top of it?
Thanks for your help!
On Friday, January 14, 2022 at 9:20:22 PM UTC+1 M T wrote:
> Hi all,
>
> I'm currently trying to add some graceful shutdown logic into my gRPC
> server, but it seems that my shutdown handler never gets called:
>
> def handle_sigterm(*_: Any) -> None :
> """Shutdown gracefully."""
> done_event = server.stop(30)
> done_event.wait(30)
> self.stdout.write('Stop complete.')
>
> server = grpc.server(
> futures.ThreadPoolExecutor(max_workers = options['max_workers']),
> )
> add_
> server.add_insecure_port(options['address'])
> self.stdout.write(f'Starting gRPC server at {options["address"]}...')
> server.start()
> start_http_server(int(khaleesi_settings['MONITORING']['PORT']))
> signal(SIGTERM, handle_sigterm)
> self._log_server_state_event(
> action = Event.Action.ActionType.START,
> result = Event.Action.ResultType.SUCCESS,
> details = 'Server started successfully.'
> )
> server.wait_for_termination()
> except Exception as start_exception:
> self._log_server_state_event(
> action = Event.Action.ActionType.START,
> result = Event.Action.ResultType.ERROR,
> details = f'Server startup failed. {type(start_exception).__name__}:
> {str(start_exception)}'
> )
> raise start_exception from None
>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/70febd14-d278-4320-a941-aca3941dca63n%40googlegroups.com.
