I took a stab at implementing a simplified exponential + randomized backoff policy here: https://github.com/apache/kafka/pull/1523
Rather than changing public interfaces / using plugins, it just adds a new client configuration "reconnect.backoff.max" that can be used to enable increasing backoff when node failures repeat. If this configuration is not set higher than reconnect.backoff.ms then the current constant backoff policy is retained. The default is to continue w/ current 50ms constant backoff. Thoughts? Would a change like this require a KIP? -Dana On Mon, May 2, 2016 at 10:29 AM, Guozhang Wang <wangg...@gmail.com> wrote: > For the specific problem of connection storm, randomized with normal > distribution at specified mean as "reconnect.backoff.ms" has been proved > pretty well. The most recent usage of it in my mind is RAFT, and it turns > out pretty effective in eliminating leader-election storms. > > > Guozhang > > On Fri, Apr 29, 2016 at 8:57 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > >> I'll agree w/ Jay and point out that the implementations of >> ReconnectionPolicy provided built-in with that driver are Constant, >> Exponential, and Counting. Constant and exponential can be combined with >> the right set of policy config parameters. I'm curious if there's a real >> need for something else, or if you're just looking for something >> exponential instead of non-constant? I think a fixed exponential backoff >> policy that defaults parameters to the current constant backoff policy >> would probably satisfy our needs. >> >> On Mon, Apr 4, 2016 at 1:25 PM, Jay Kreps <j...@confluent.io> wrote: >> >> > If I understand the problem we are fixing is a connection storm where >> when >> > a new broker comes online it is overwhelmed with connections. >> > >> > In general we try hard to avoid plugins where possible. Maybe instead of >> > adding another plugin interface we could just directly solve this problem >> > by doing some randomization in the backoff to space out the >> reconnections? >> > This seems like it would be good for anyone with a large client >> > environment? >> > >> > -Jay >> > >> > On Mon, Apr 4, 2016 at 12:54 PM, Florian Hussonnois < >> fhussonn...@gmail.com >> > > >> > wrote: >> > >> > > Hi Kafka Team, >> > > >> > > I have made a new Kafka Improvement Proposal. >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient >> > > >> > > This is my first proposal so I don't know if I have given enough >> > > information. >> > > Also I have already proposed an implementation : >> > > https://github.com/apache/kafka/pull/1179 >> > > >> > > Thanks >> > > >> > > -- >> > > Florian HUSSONNOIS >> > > >> > >> >> >> >> -- >> Thanks, >> Ewen >> > > > > -- > -- Guozhang
