Ismael, Created bug: https://issues.apache.org/jira/browse/KAFKA-3689
Hope to get a quick resolution. Thanks, Buvana -----Original Message----- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT Ismael Juma Sent: Tuesday, May 10, 2016 10:54 AM To: users@kafka.apache.org Subject: Re: ERROR Processor got uncaught exception. (kafka.network.Processor) OK, thanks. I suggest filing a bug in JIRA and please provide as much information as possible (steps to reproduce would be ideal, but sometimes that is hard to do). It does look like a Kafka bug. Ismael On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Ismael, > > Version 0.9.0.1 > > Do you have any idea how to prevent this from happening? Is it a Kafka > issue? > > -Buvana > > -----Original Message----- > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > Ismael Juma > Sent: Monday, May 09, 2016 8:02 PM > To: users@kafka.apache.org > Subject: Re: ERROR Processor got uncaught exception. > (kafka.network.Processor) > > Hi Buvana, > > What is the version of the broker? It looks like the state of connection > counts for quotas became inconsistent, that is, a given client ip address > has no connections according to the internal state, but a disconnection for > that ip happened. > > Ismael > > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) < > buvana.rama...@nokia.com> wrote: > > > Hello, > > > > We are trying to understand the following error in one of the brokers in > > our cluster. It's a 3 broker cluster (all 3 servers are identical - > Intel > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving > > about 8000 topics, single partitioned & replication factor = 3. Each > topic > > gets data at a low rate - 200 bytes per sec. Cluster is managed by 3 > > server ZK cluster (these servers are different from Kafka broker > servers). > > All 6 servers are connected via 10G switch. Producers run from external > > servers. > > > > Leaders are balanced across the topics. > > > > This error message occurs repeatedly in one of the servers. Between the > > hours of 10:30am and 1:30pm today, there were about 10 Million such > > occurrences. > > > > We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can > > share the current settings if needed. > > > > Please let me know what is causing this error and a way to overcome it. > > > > Regards, > > Buvana > > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > (kafka.network.Processor) > > java.lang.IllegalArgumentException: Attempted to decrease connection > count > > for address with no connections, address: /X.Y.Z.144 (actual network > > address masked) > > at > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > at > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > at > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > at > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > at > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > at kafka.network.Processor.run(SocketServer.scala:445) > > at java.lang.Thread.run(Thread.java:745) > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > (kafka.network.Processor) > > java.lang.IllegalArgumentException: Attempted to decrease connection > count > > for address with no connections, address: /X.Y.Z.144 > > at > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > at > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > at > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > at > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > at > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > at kafka.network.Processor.run(SocketServer.scala:445) > > at java.lang.Thread.run(Thread.java:745) > > >
