Here are the logs from broker id 0 and 1 and it was captured when broker 1 went down.
http://paste.ubuntu.com/9782553/ http://paste.ubuntu.com/9782554/ i'm using netty in storm and here are the configs storm.messaging.transport: "backtype.storm.messaging.netty.Context" storm.messaging.netty.buffer_size: 209715200 storm.messaging.netty.max_retries: 10 storm.messaging.netty.max_wait_ms: 5000 storm.messaging.netty.min_wait_ms: 10000 On Sat, Jan 17, 2015 at 1:24 AM, Harsha <ka...@harsha.io> wrote: > Tousif, > I meant to say if kafka broker is going down often its better to > analyze whats the root of cause of the crash. Using supervisord > to monitor kafka broker is fine, sorry about the confusion. > -Harsha > On Fri, Jan 16, 2015, at 11:25 AM, Gwen Shapira wrote: > > Those errors are expected - if broker 10.0.0.11 went down, it will > > reset the connection and the other broker will close the socket. > > However, it looks like 10.0.0.11 crashes every two minutes? > > > > Do you have the logs from 10.0.0.11? > > > > On Thu, Jan 15, 2015 at 9:51 PM, Tousif <tousif.pa...@gmail.com> wrote: > > > i'm using kafka 2.9.2-0.8.1.1 and zookeeper 3.4.6. > > > i noticed that only one broker is going down. > > > My message size is less thn 3 kb and KAFKA_HEAP_OPTS="-Xmx512M" > > > and KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops > > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled > > > -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC > > > -Djava.awt.headless=true" . > > > > > > Do you mean kafka broker never goes down and does broker start > > > automatically after failing ? > > > I see only these errors on both the brokers. > > > > > > 10.0.0.11 is the broker which is going down. > > > > > > ERROR Closing socket for /10.0.0.11 because of error > > > (kafka.network.Processor) > > > java.io.IOException: Connection reset by peer > > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) > > > at sun.nio.ch.IOUtil.read(IOUtil.java:171) > > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) > > > at kafka.utils.Utils$.read(Utils.scala:375) > > > at > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > > > at kafka.network.Processor.read(SocketServer.scala:347) > > > at kafka.network.Processor.run(SocketServer.scala:245) > > > at java.lang.Thread.run(Thread.java:662) > > > [2015-01-16 11:01:48,173] INFO Closing socket connection to /10.0.0.11 > . > > > (kafka.network.Processor) > > > [2015-01-16 11:03:08,164] ERROR Closing socket for /10.0.0.11 because > of > > > error (kafka.network.Processor) > > > java.io.IOException: Connection reset by peer > > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) > > > at sun.nio.ch.IOUtil.read(IOUtil.java:171) > > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) > > > at kafka.utils.Utils$.read(Utils.scala:375) > > > at > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > > > at kafka.network.Processor.read(SocketServer.scala:347) > > > at kafka.network.Processor.run(SocketServer.scala:245) > > > at java.lang.Thread.run(Thread.java:662) > > > [2015-01-16 11:03:08,280] INFO Closing socket connection to /10.0.0.11 > . > > > (kafka.network.Processor) > > > [2015-01-16 11:03:48,369] ERROR Closing socket for /10.0.0.11 because > of > > > error (kafka.network.Processor) > > > java.io.IOException: Connection reset by peer > > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) > > > at sun.nio.ch.IOUtil.read(IOUtil.java:171) > > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) > > > at kafka.utils.Utils$.read(Utils.scala:375) > > > at > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > > > at kafka.network.Processor.read(SocketServer.scala:347) > > > at kafka.network.Processor.run(SocketServer.scala:245) > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > > > > > On Thu, Jan 15, 2015 at 7:49 PM, Harsha <ka...@harsha.io> wrote: > > > > > >> Tousif, > > >> Which version of kafka and zookeeper are you using and whats > your > > >> message size and jvm size that you allocated for kafka brokers. > > >> There is only 1 zookeeper node , if its a production cluster I > recommend > > >> you to have quorum of zookeeper nodes. Both kafka & storm are heavy > > >> users of zookeeper. Also supervisord is recommended for storm I am not > > >> sure you need to have it for kafka, for storm its the fail-fast nature > > >> of workers that requires supervisord to restart. > > >> When kafka goes down first time , i.e before supervisord restarts it > do > > >> you see same OOM error. Check the logs to see why its going down for > the > > >> first time. > > >> -Harsha > > >> > > >> > > >> > > >> On Wed, Jan 14, 2015, at 10:50 PM, Tousif wrote: > > >> > Hello Chia-Chun Shih, > > >> > > > >> > There are multiple issues, > > >> > First thing is i don't see out of memory error every time and OOM > happens > > >> > after supervisord keep retrying to start kafka. > > >> > It goes down when it tries to add partition fetcher > > >> > > > >> > it starts with > > >> > > > >> > *conflict in /controller data: > > >> > {"version":1,"brokerid":0,"timestamp":"1421296052741"} stored data: > > >> > {"version":1,"brokerid":1,"timestamp":"1421291998088"} > > >> > (kafka.utils.ZkUtils$)* > > >> > > > >> > > > >> > ERROR Conditional update of path > > >> > /brokers/topics/realtimestreaming/partitions/1/state with data > > >> > > > >> > {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":54,"isr":[0]} > > >> > and expected version 90 failed due to > > >> > org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode > > >> > = > > >> > BadVersion for /brokers/topics/realtimestreaming/partitions/1/state > > >> > (kafka.utils.ZkUtils$) > > >> > > > >> > and then > > >> > > > >> > [ReplicaFetcherManager on broker 0] Removed fetcher for partitions > > >> > [realtimestreaming,0],[realtimestreaming,1] > > >> > (kafka.server.ReplicaFetcherManager) > > >> > [2015-01-15 09:57:34,350] INFO Truncating log realtimestreaming-0 to > > >> > offset > > >> > 846. (kafka.log.Log) > > >> > [2015-01-15 09:57:34,351] INFO Truncating log realtimestreaming-1 to > > >> > offset > > >> > 957. (kafka.log.Log) > > >> > [2015-01-15 09:57:34,650] INFO [ReplicaFetcherManager on broker 0] > *Added > > >> > fetcher for partitions ArrayBuffer([[realtimestreaming,0], > initOffset 846 > > >> > to broker id:1,host:realtimeslave1.novalocal,port:9092] , > > >> > [[realtimestreaming,1], initOffset 957 to broker > > >> > id:1,host:realtimeslave1.novalocal,port:9092] ) > > >> > (kafka.server.ReplicaFetcherManager)* > > >> > [2015-01-15 09:57:34,654] INFO [ReplicaFetcherThread-0-1], Starting > > >> > (kafka.server.ReplicaFetcherThread) > > >> > [2015-01-15 09:57:34,747] INFO [ReplicaFetcherThread-1-1], Starting > > >> > (kafka.server.ReplicaFetcherThread) > > >> > [2015-01-15 09:58:14,156] INFO Closing socket connection to / > 10.0.0.11. > > >> > (kafka.network.Processor) > > >> > > > >> > > > >> > > > >> > On Thu, Jan 15, 2015 at 12:01 PM, Chia-Chun Shih > > >> > <chiachun.s...@gmail.com> > > >> > wrote: > > >> > > > >> > > You can use tools (e.g., VisialVM) to diagnose OOM problem. > > >> > > > > >> > > 2015-01-15 14:15 GMT+08:00 Tousif Khazi <tou...@senseforth.com>: > > >> > > > > >> > > > i see this error > > >> > > > > > >> > > > ERROR [ReplicaFetcherThread-0-1], Error for partition > > >> > > > [realtimestreaming,1] to broker 1:class > > >> > > > kafka.common.NotLeaderForPartitionException > > >> > > > (kafka.server.ReplicaFetcherThread) > > >> > > > [2015-01-15 10:00:04,348] INFO [ReplicaFetcherManager on broker > 0] > > >> > > > Removed fetcher for partitions [realtimestreaming,1] > > >> > > > (kafka.server.ReplicaFetcherManager) > > >> > > > [2015-01-15 10:00:04,355] INFO Closing socket connection to > > >> > > > /10.0.0.11. (kafka.network.Processor) > > >> > > > [2015-01-15 10:00:04,444] WARN [KafkaApi-0] Fetch request with > > >> > > > correlation id 0 from client ReplicaFetcherThread-0-0 on > partition > > >> > > > [realtimestreaming,1] failed due to Leader not local for > partition > > >> > > > [realtimestreaming,1] on broker 0 (kafka.server.KafkaApis) > > >> > > > [2015-01-15 10:00:04,545] INFO [ReplicaFetcherThread-0-1], > Shutting > > >> > > > down (kafka.server.ReplicaFetcherThread) > > >> > > > [2015-01-15 10:00:04,848] INFO [ReplicaFetcherThread-0-1], > Stopped > > >> > > > (kafka.server.ReplicaFetcherThread) > > >> > > > [2015-01-15 10:00:04,849] INFO [ReplicaFetcherThread-0-1], > Shutdown > > >> > > > completed (kafka.server.ReplicaFetcherThread) > > >> > > > [2015-01-15 10:00:39,256] ERROR Closing socket for /10.0.0.11 > > >> because > > >> > > > of error (kafka.network.Processor) > > >> > > > java.io.IOException: Connection reset by peer > > >> > > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > >> > > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > >> > > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) > > >> > > > at sun.nio.ch.IOUtil.read(IOUtil.java:171) > > >> > > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) > > >> > > > > > >> > > > On Wed, Jan 14, 2015 at 10:12 PM, Tousif < > tousif.pa...@gmail.com> > > >> wrote: > > >> > > > > Thanks harsha for quick response. > > >> > > > > I don't see any other error. I used to see replica fetcher > error > > >> but > > >> > > > seems > > >> > > > > to be disappeared after setting replica fetcher threads to 2 > as I > > >> have > > >> > > 2 > > >> > > > > partitions. Some times I see zookeeper session expiration. > > >> > > > > On Jan 14, 2015 8:02 PM, "Harsha" <ka...@harsha.io> wrote: > > >> > > > > > > >> > > > >> Tousif, > > >> > > > >> Do you see any other errors in server.log > > >> > > > >> -Harsha > > >> > > > >> > > >> > > > >> On Wed, Jan 14, 2015, at 01:51 AM, Tousif wrote: > > >> > > > >> > Hello, > > >> > > > >> > > > >> > > > >> > I have configured kafka nodes to run via supervisord and > see > > >> > > > following > > >> > > > >> > exceptions > > >> > > > >> > and eventually brokers going out of memory. i have given > enough > > >> > > memory > > >> > > > >> > and > > >> > > > >> > process 1 event/second. kafka goes down every day > > >> > > > >> > > > >> > > > >> > i'm wondering what configurastion is missing or need to be > added > > >> > > > >> > > > >> > > > >> > Here are my cluster details: > > >> > > > >> > 2 brokers > > >> > > > >> > 1 zookeeper > > >> > > > >> > and 2 node apache storm > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > INFO zookeeper state changed (SyncConnected) > > >> > > > >> > (org.I0Itec.zkclient.ZkClient) > > >> > > > >> > ERROR Closing socket for /10.0.0.11 because of error > > >> > > > >> > (kafka.network.Processor) > > >> > > > >> > java.io.IOException: Connection reset by peer > > >> > > > >> > at sun.nio.ch.FileDispatcher.read0(Native Method) > > >> > > > >> > at > sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > >> > > > >> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) > > >> > > > >> > at sun.nio.ch.IOUtil.read(IOUtil.java:171) > > >> > > > >> > at > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) > > >> > > > >> > at kafka.utils.Utils$.read(Utils.scala:375) > > >> > > > >> > at > > >> > > > >> > > > >> > > > >> > > >> > > > > > >> > > > > >> > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > > >> > > > >> > at kafka.network.Processor.read(SocketServer.scala:347) > > >> > > > >> > at kafka.network.Processor.run(SocketServer.scala:245) > > >> > > > >> > at java.lang.Thread.run(Thread.java:662) > > >> > > > >> > [2015-01-13 23:43:37,962] INFO Closing socket connection > to / > > >> > > > 10.0.0.11. > > >> > > > >> > (kafka.network.Processor) > > >> > > > >> > Error occurred during initialization of VM > > >> > > > >> > Could not reserve enough space for object heap > > >> > > > >> > Error occurred during initialization of VM > > >> > > > >> > Could not reserve enough space for object heap > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > Regards, > > >> > > > >> > Tousif > > >> > > > >> > +918050227279 > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > Regards > > >> > > > >> > Tousif Khazi > > >> > > > >> > > >> > > > > > >> > > > > > >> > > > > > >> > > > -- > > >> > > > Regards, > > >> > > > Tousif > > >> > > > +918050227279 > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > > > >> > Regards > > >> > Tousif Khazi > > >> > > > > > > > > > > > > -- > > > > > > > > > Regards > > > Tousif Khazi > -- Regards Tousif Khazi
