>> we cannot just block the whole topic because there could partitions on other >> brokers which are at least available for reads That makes sense. So in order to not stop reading or writing from good partitions we'd need either:
1. This circuit breaker functionality to be in the client itself Or 2. The client to return which partition failed to read/write from. This would only be helpful if the clients are assigning partitions themselves. I use "partition" and not "broker" since each partition has only one primary broker at a time. -Dave -----Original Message----- From: Andrey Dyachkov [mailto:andrey.dyach...@gmail.com] Sent: Monday, October 31, 2016 3:17 AM To: users@kafka.apache.org Subject: Re: Kafka client circuit breaker R Krishna, A particular set of exceptions has to be defined. At least what I know: timeout exception, network exception. I am not sure I got the point about auto-recovery. About incoming messages, we can return the exception that the client can not access topic/broker/partition and user can properly process it. Dave, There is a question what the thing is we check failures for. If this is a topic we can not just block the whole topic because there could partitions on other brokers which are at least available for reads. From my point of view checking per broker is a good enough solution but as I already said it wasteful to implement it on the user side because Kafka client does the job to find out the brokers before the message is gonna be read/written. Thank you for support! On Mon, 31 Oct 2016 at 02:41 Tauzell, Dave <dave.tauz...@surescripts.com> wrote: > Once enough failures happen the circuit is marked open. The client > would then periodically try some messages until it works again. > Others would be failed. > > There are a number of existing circuit breaker libraries you can use > in the meantime like the Netflix one. > > Dave > > > On Oct 30, 2016, at 20:23, R Krishna <krishna...@gmail.com> wrote: > > > > So, basically, you want the clients to gracefully handle timeouts > > and > short > > circuit if it is unavailable broker exception and also clients > > should be able automatically recover when it is up? What about the > > messages that > come > > in when broker is not available? > > > > On Sun, Oct 30, 2016 at 12:25 PM, Andrey Dyachkov < > andrey.dyach...@gmail.com > >> wrote: > > > >> R Krishna, > >> > >> We have default timeout for producer and consumer it is 30 secs. > >> If Kafka is not available we will wait 30 secs on each request. > >> Decreasing timeout won't help much because we have a lot requests > >> but circuiting them will help us tremendous. We could implement > >> such logic before actually calling Kafka client but there is > >> problem what the trigger is to open the circuit. > >> I thought about circuiting by unavailable broker but it is quite an > >> expansive operation to figure out in the client's > code > >> on the flip side Kaka client already does all the job to make out > >> the broker. > >> > >> For us very important to stay live even if Kafka is unavailable (it > >> happens) that's why I think avoiding cascading issues will be very > useful. > >> > >>> On Sat, 29 Oct 2016 at 19:22 R Krishna <krishna...@gmail.com> wrote: > >>> > >>> What are the issues and problems you want to solve though? > >>> On Oct 29, 2016 3:59 AM, "Andrey Dyachkov" > >>> <andrey.dyach...@gmail.com> > >>> wrote: > >>> > >>>> Hi, > >>>> > >>>> We use Kafka for around half a year as a main backend system for > >> storing > >>>> events from big variety of internal microservices. In general It > >>>> works quite well and we experience issues only in some rare 'corner > >>>> cases' > >>> where > >>>> we either run against a bug or do not understand why Kafka > >>>> behaves in particular manner. The fact Kafka is the main system > >>>> is used under our software If it fails our system will fail too. > >>>> That's why I am writing > >> to > >>>> ask about the solution which can help us tolerate different > >>>> issues on > >>> Kafka > >>>> side. First of all could you tell me why doesn't Kafka client > >>>> have the solution for this kind of problems? Have you ever had > >>>> thoughts about > >> it? > >>>> Thank you in advance! > >>>> -- > >>>> > >>>> With great enthusiasm, > >>>> Andrey > >>>> > >>> > >> -- > >> > >> With great enthusiasm, > >> Andrey > >> > > > > > > > > -- > > Radha Krishna, Proddaturi > > 253-234-5657 > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of > the individual or entity to whom they are addressed. If you have > received this e-mail in error, please notify the sender by reply > e-mail immediately and destroy all copies of the e-mail and any attachments. > -- With great enthusiasm, Andrey This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
