So I'm seeing CancelledKeyExceptions cropping up about the time that the connections get reset. Is this a zookeeper error that I'm hitting?
Karl On Apr 24, 2013, at 9:55 AM, Karl Kirch <kki...@wdtinc.com> wrote: > Just got logging cranked up. Will let you know when I see it again. > > Thanks, > Karl > > On Apr 23, 2013, at 8:11 PM, Jun Rao <jun...@gmail.com> > wrote: > >> This means that the broker closed the socket connection for some reason. >> The broker log around the same time should show the reason. Could you dig >> that out? >> >> Thanks, >> >> Jun >> >> >> On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch <kki...@wdtinc.com> wrote: >> >>> I occasionally am getting some batch send errors from the stock async >>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes. >>> Is there anyway to check what happens when those batch errors occur? >>> Or bump up the retry count? (looks like it only did a single retry). >>> >>> I need the speed of the async producer, but it needs to be reliable (I see >>> a handful of these a day but in a weather alerting system it only takes >>> missing one let alone 25 or 100/1000). >>> >>> Here's a stack trace of one of the errors that I'm seeing. >>> >>> 22:23:39.405 [ProducerSendThread-1824508747] WARN >>> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining >>> java.io.IOException: Connection reset by peer >>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] >>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) >>> ~[na:1.6.0_24] >>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] >>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) >>> ~[na:1.6.0_24] >>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360) >>> ~[na:1.6.0_24] >>> at >>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) >>> ~[apns-consumer-1.0.jar:na] >>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) >>> ~[apns-consumer-1.0.jar:na] >>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94) >>> ~[apns-consumer-1.0.jar:na] >>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) >>> [apns-consumer-1.0.jar:na] >>> at scala.collection.immutable.Stream.foreach(Stream.scala:260) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) >>> [apns-consumer-1.0.jar:na] >>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR >>> k.p.a.ProducerSendThread - Error in handling batch of 27 events >>> java.io.IOException: Connection reset by peer >>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] >>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) >>> ~[na:1.6.0_24] >>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] >>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) >>> ~[na:1.6.0_24] >>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360) >>> ~[na:1.6.0_24] >>> at >>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) >>> ~[apns-consumer-1.0.jar:na] >>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) >>> ~[apns-consumer-1.0.jar:na] >>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94) >>> ~[apns-consumer-1.0.jar:na] >>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) >>> ~[apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) >>> [apns-consumer-1.0.jar:na] >>> at scala.collection.immutable.Stream.foreach(Stream.scala:260) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) >>> [apns-consumer-1.0.jar:na] >>> at >>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) >>> [apns-consumer-1.0.jar:na] >>> >>> >>> >
