How many server streaming calls are open to C at the time of the issue? Does C limits the number of maximum concurrent streams? (you can see it in the debug logs for the SETTINGS frame coming from C) On Tuesday, November 26, 2024 at 7:22:16 AM UTC-8 Jiří Krutil wrote:
> When running our Java application in MS Azure, we sometimes observe very > strange behavior, which appears as if a long-lived gRPC channel was working > only in one direction and was not delivering any RPC calls in the opposite > direction. > > Out setup is that we have three apps connected via gRPC: > > A -> B -> C > > B usually has a long-lived server streaming gRPC requests to C open, > consuming updates from C. When the issue occurs, updates from C are still > streaming to B, but no new unary requests made by B make it to C. > > The unary requests made by B are originated by A. B receives the request > from A and sends an unary request to C with a deadline copied from the > original request. After 20 seconds, B sees an "RPC cancelled" event, which > I believe comes from A in response to some kind of timeout. > > The problem occurs randomly and when it occurs, the channel never recovers. > > Debug logging seems to show that when B receives the request from A, it > creates a new stub using an existing (cached) channel and attempts to send > a request to C, but that request is actually never sent. > > If I make B forget the cached channel and create a new one, the unary > request to C works fine. > > We have keepAlive enabled on these channels, so I am surprised that > potential issues with the underlying connection are not detected by the > keepAlive mechanism. Is it possible that since traffic is steadily flowing > in the direction from C to B, that B never pings C to see if communication > in the opposite direction works as well? > > I suppose we could work around this by adding application-level health > checking for every channel, but I thought this is already taken care of by > gRPC. > > Any suggestions would be appreciated. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/grpc-io/0457a2a7-e1fd-4420-a01e-62db11437e4an%40googlegroups.com.
