In our case, we're only using gRPC over a local network, so we're fine with repeated and expensive attempts to reconnect. My first thought is to do do a new call to CreateChannel... but please let me know if there is anything else you'd suggest as a workaround.
On Tue, Jul 18, 2017 at 11:40 AM 'Carl Mastrangelo' via grpc.io < [email protected]> wrote: > This is indeed because gRPC does exponential connection backoff. To > reconnect immediate means we would have to know when the service is back > up. This is a problem because checking if the service is up is sometimes > expensive, sometimes even enough to keep the service down. > > At most it will wait 2 minutes after which it will re attach. I cannot > comment on whether those parameters were broken at that release. > > > On Friday, July 14, 2017 at 12:36:32 PM UTC-7, Bruno Bowden wrote: >> >> NOTE: we are currently using gRPC v1.1.4. For various reasons, it's more >> challenging for us to upgrade but please advise if this is a fixed issue in >> a more recent version or any workaround that you'd suggest. Thanks for all >> your work on gRPC. >> >> I'm trying to solve an issue with a persistent gRPC client where it takes >> 20 seconds to reconnect after a "Connect Failed" event. This is useful >> during testing when the service is repeatedly brought up and down, while >> the client is left running. See the console logs below that shows what >> repeatedly occurs. After bringing down the service, the client reports >> "Deadline Exceeded" - it has a 500ms and these failures are expected. When >> the service is restarted, including responding to other grpc clients, the >> already running grpc client continues to fail until 20 second after the >> original "Connect Failed". >> >> My impression is that this is part of the exponential backoff and retry. >> If I restart the persistent client, then it works immediately. I've tried >> playing with the grpc.max_reconnect_backoff_ms >> and grpc.initial_reconnect_backoff_ms settings without any success. >> >> # GRPC SERVICE BROUGHT DOWN >> I0714 12:13:17.325851 1793 xxxx.cc:118] gRPC status: Connect Failed >> I0714 12:13:18.826437 1794 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:19.826522 1792 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:20.826452 1795 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:21.826382 1798 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:22.826511 1797 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:23.826552 1791 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:24.826550 1795 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:25.826663 1793 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:26.826717 1794 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:27.826637 1792 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:28.826462 1798 xxxx.cc:118] gRPC status: Deadline Exceeded >> # GRPC SERVICE SUCCESSFULLY RESTARTED >> # expecting reconnect to work immediately >> I0714 12:13:29.826494 1795 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:30.826251 1791 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:31.827332 1792 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:32.826256 1796 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:33.826165 1795 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:34.827574 1791 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:35.826184 1792 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:36.825944 1798 xxxx.cc:118] gRPC status: Deadline Exceeded >> I0714 12:13:37.326781 1795 xxxx.cc:125] gRPC status: ok >> I0714 12:13:38.326337 1797 xxxx.cc:125] gRPC status: ok >> I0714 12:13:39.326292 1794 xxxx.cc:125] gRPC status: ok >> I0714 12:13:40.326501 1798 xxxx.cc:125] gRPC status: ok >> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "grpc.io" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/grpc-io/gmdyN2rukbY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/cd863687-5cb7-46c3-965f-adb19f77299a%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/cd863687-5cb7-46c3-965f-adb19f77299a%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAP7OsNTT_Q1_JxCF%2BNXKuXYE2r24v%3DUS930-Fu-QgWz7gfn4Bg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
