Ismael, We are setting up a 0.10.0 test cluster now. I will report on whether this bug springs up in that cluster or not after a week or so.
Glad to hear that SocketServer bugs are being taken care of in 0.10.0 and hope this issue is ironed out as a result. Regards, Buvana -----Original Message----- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT Ismael Juma Sent: Tuesday, May 10, 2016 5:00 PM To: users@kafka.apache.org Subject: Re: ERROR Processor got uncaught exception. (kafka.network.Processor) Thanks Buvana. Is this happening in production only or can you also reproduce it in a test cluster? If the latter, would you be able to test the latest 0.10.0.0 release candidate? We fixed a few issues in the SocketServer. Ismael On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Ismael, > > Created bug: > https://issues.apache.org/jira/browse/KAFKA-3689 > > Hope to get a quick resolution. > > Thanks, > Buvana > > -----Original Message----- > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > Ismael Juma > Sent: Tuesday, May 10, 2016 10:54 AM > To: users@kafka.apache.org > Subject: Re: ERROR Processor got uncaught exception. > (kafka.network.Processor) > > OK, thanks. I suggest filing a bug in JIRA and please provide as much > information as possible (steps to reproduce would be ideal, but > sometimes that is hard to do). It does look like a Kafka bug. > > Ismael > > On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) < > buvana.rama...@nokia.com> wrote: > > > Ismael, > > > > Version 0.9.0.1 > > > > Do you have any idea how to prevent this from happening? Is it a > > Kafka issue? > > > > -Buvana > > > > -----Original Message----- > > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > > Ismael Juma > > Sent: Monday, May 09, 2016 8:02 PM > > To: users@kafka.apache.org > > Subject: Re: ERROR Processor got uncaught exception. > > (kafka.network.Processor) > > > > Hi Buvana, > > > > What is the version of the broker? It looks like the state of > > connection counts for quotas became inconsistent, that is, a given > > client ip address has no connections according to the internal > > state, but a disconnection > for > > that ip happened. > > > > Ismael > > > > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) < > > buvana.rama...@nokia.com> wrote: > > > > > Hello, > > > > > > We are trying to understand the following error in one of the > > > brokers > in > > > our cluster. It's a 3 broker cluster (all 3 servers are identical > > > - > > Intel > > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), > serving > > > about 8000 topics, single partitioned & replication factor = 3. > > > Each > > topic > > > gets data at a low rate - 200 bytes per sec. Cluster is managed > > > by 3 server ZK cluster (these servers are different from Kafka > > > broker > > servers). > > > All 6 servers are connected via 10G switch. Producers run from > > > external servers. > > > > > > Leaders are balanced across the topics. > > > > > > This error message occurs repeatedly in one of the servers. > > > Between the hours of 10:30am and 1:30pm today, there were about 10 > > > Million such occurrences. > > > > > > We tuned Linux network params (somaxconn, rmem_max, wmem_max, > > > etc). I > can > > > share the current settings if needed. > > > > > > Please let me know what is causing this error and a way to overcome it. > > > > > > Regards, > > > Buvana > > > > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > > (kafka.network.Processor) > > > java.lang.IllegalArgumentException: Attempted to decrease > > > connection > > count > > > for address with no connections, address: /X.Y.Z.144 (actual > > > network address masked) > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > > > at > scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > > at > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > > at kafka.network.Processor.run(SocketServer.scala:445) > > > at java.lang.Thread.run(Thread.java:745) > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > > (kafka.network.Processor) > > > java.lang.IllegalArgumentException: Attempted to decrease > > > connection > > count > > > for address with no connections, address: /X.Y.Z.144 > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > > at > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > > > at > scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > > at > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > > at kafka.network.Processor.run(SocketServer.scala:445) > > > at java.lang.Thread.run(Thread.java:745) > > > > > >
