Thanks. I had a quick look at the code and it's not obvious how this could happen. Let's see how your testing goes. :)
Ismael On Wed, May 11, 2016 at 2:04 AM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Ismael, > > We are setting up a 0.10.0 test cluster now. I will report on whether this > bug springs up in that cluster or not after a week or so. > > Glad to hear that SocketServer bugs are being taken care of in 0.10.0 and > hope this issue is ironed out as a result. > > Regards, > Buvana > > -----Original Message----- > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > Ismael Juma > Sent: Tuesday, May 10, 2016 5:00 PM > To: users@kafka.apache.org > Subject: Re: ERROR Processor got uncaught exception. > (kafka.network.Processor) > > Thanks Buvana. Is this happening in production only or can you also > reproduce it in a test cluster? If the latter, would you be able to test > the latest 0.10.0.0 release candidate? We fixed a few issues in the > SocketServer. > > Ismael > > On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) < > buvana.rama...@nokia.com> wrote: > > > Ismael, > > > > Created bug: > > https://issues.apache.org/jira/browse/KAFKA-3689 > > > > Hope to get a quick resolution. > > > > Thanks, > > Buvana > > > > -----Original Message----- > > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > > Ismael Juma > > Sent: Tuesday, May 10, 2016 10:54 AM > > To: users@kafka.apache.org > > Subject: Re: ERROR Processor got uncaught exception. > > (kafka.network.Processor) > > > > OK, thanks. I suggest filing a bug in JIRA and please provide as much > > information as possible (steps to reproduce would be ideal, but > > sometimes that is hard to do). It does look like a Kafka bug. > > > > Ismael > > > > On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) < > > buvana.rama...@nokia.com> wrote: > > > > > Ismael, > > > > > > Version 0.9.0.1 > > > > > > Do you have any idea how to prevent this from happening? Is it a > > > Kafka issue? > > > > > > -Buvana > > > > > > -----Original Message----- > > > From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT > > > Ismael Juma > > > Sent: Monday, May 09, 2016 8:02 PM > > > To: users@kafka.apache.org > > > Subject: Re: ERROR Processor got uncaught exception. > > > (kafka.network.Processor) > > > > > > Hi Buvana, > > > > > > What is the version of the broker? It looks like the state of > > > connection counts for quotas became inconsistent, that is, a given > > > client ip address has no connections according to the internal > > > state, but a disconnection > > for > > > that ip happened. > > > > > > Ismael > > > > > > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) < > > > buvana.rama...@nokia.com> wrote: > > > > > > > Hello, > > > > > > > > We are trying to understand the following error in one of the > > > > brokers > > in > > > > our cluster. It's a 3 broker cluster (all 3 servers are identical > > > > - > > > Intel > > > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), > > serving > > > > about 8000 topics, single partitioned & replication factor = 3. > > > > Each > > > topic > > > > gets data at a low rate - 200 bytes per sec. Cluster is managed > > > > by 3 server ZK cluster (these servers are different from Kafka > > > > broker > > > servers). > > > > All 6 servers are connected via 10G switch. Producers run from > > > > external servers. > > > > > > > > Leaders are balanced across the topics. > > > > > > > > This error message occurs repeatedly in one of the servers. > > > > Between the hours of 10:30am and 1:30pm today, there were about 10 > > > > Million such occurrences. > > > > > > > > We tuned Linux network params (somaxconn, rmem_max, wmem_max, > > > > etc). I > > can > > > > share the current settings if needed. > > > > > > > > Please let me know what is causing this error and a way to overcome > it. > > > > > > > > Regards, > > > > Buvana > > > > > > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > > > (kafka.network.Processor) > > > > java.lang.IllegalArgumentException: Attempted to decrease > > > > connection > > > count > > > > for address with no connections, address: /X.Y.Z.144 (actual > > > > network address masked) > > > > at > > > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > > at > > > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > > at > scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > > > at > > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > > > at > > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > > > at > scala.collection.Iterator$class.foreach(Iterator.scala:742) > > > > at > > scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > > > at > > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > > > at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > > > at kafka.network.Processor.run(SocketServer.scala:445) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. > > > > (kafka.network.Processor) > > > > java.lang.IllegalArgumentException: Attempted to decrease > > > > connection > > > count > > > > for address with no connections, address: /X.Y.Z.144 > > > > at > > > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > > at > > > > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565) > > > > at > scala.collection.MapLike$class.getOrElse(MapLike.scala:128) > > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59) > > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564) > > > > at > > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450) > > > > at > > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445) > > > > at > scala.collection.Iterator$class.foreach(Iterator.scala:742) > > > > at > > scala.collection.AbstractIterator.foreach(Iterator.scala:1194) > > > > at > > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > > > at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > > > at kafka.network.Processor.run(SocketServer.scala:445) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > >
