Ismael, Version 0.9.0.1
Do you have any idea how to prevent this from happening? Is it a Kafka issue? -Buvana -----Original Message----- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT Ismael Juma Sent: Monday, May 09, 2016 8:02 PM To: users@kafka.apache.org Subject: Re: ERROR Processor got uncaught exception. (kafka.network.Processor) Hi Buvana, What is the version of the broker? It looks like the state of connection counts for quotas became inconsistent, that is, a given client ip address has no connections according to the internal state, but a disconnection for that ip happened. Ismael On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Hello, > > We are trying to understand the following error in one of the brokers in > our cluster. It's a 3 broker cluster (all 3 servers are identical - Intel > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving > about 8000 topics, single partitioned & replication factor = 3. Each topic > gets data at a low rate - 200 bytes per sec. Cluster is managed by 3 > server ZK cluster (these servers are different from Kafka broker servers). > All 6 servers are connected via 10G switch. Producers run from external > servers. > > Leaders are balanced across the topics. > > This error message occurs repeatedly in one of the servers. Between the > hours of 10:30am and 1:30pm today, there were about 10 Million such > occurrences. > > We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can > share the current settings if needed. > > Please let me know what is causing this error and a way to overcome it. > > Regards, > Buvana > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.IllegalArgumentException: Attempted to decrease connection count > for address with no connections, address: /X.Y.Z.144 (actual network > address masked) > at > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > at > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > at > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > at > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at kafka.network.Processor.run(SocketServer.scala:445) > at java.lang.Thread.run(Thread.java:745) > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.IllegalArgumentException: Attempted to decrease connection count > for address with no connections, address: /X.Y.Z.144 > at > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > at > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > at > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > at > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > at scala.collection.Iterator$class.foreach(Iterator.scala:742) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at kafka.network.Processor.run(SocketServer.scala:445) > at java.lang.Thread.run(Thread.java:745) >
