Hello Chia-Chun Shih,
There are multiple issues,
First thing is i don't see out of memory error every time and OOM happens
after supervisord keep retrying to start kafka.
It goes down when it tries to add partition fetcher
it starts with
*conflict in /controller data:
{"version":1,"brokerid":0,"timestamp":"1421296052741"} stored data:
{"version":1,"brokerid":1,"timestamp":"1421291998088"}
(kafka.utils.ZkUtils$)*
ERROR Conditional update of path
/brokers/topics/realtimestreaming/partitions/1/state with data
{"controller_epoch":34,"leader":0,"version":1,"leader_epoch":54,"isr":[0]}
and expected version 90 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/realtimestreaming/partitions/1/state
(kafka.utils.ZkUtils$)
and then
[ReplicaFetcherManager on broker 0] Removed fetcher for partitions
[realtimestreaming,0],[realtimestreaming,1]
(kafka.server.ReplicaFetcherManager)
[2015-01-15 09:57:34,350] INFO Truncating log realtimestreaming-0 to offset
846. (kafka.log.Log)
[2015-01-15 09:57:34,351] INFO Truncating log realtimestreaming-1 to offset
957. (kafka.log.Log)
[2015-01-15 09:57:34,650] INFO [ReplicaFetcherManager on broker 0] *Added
fetcher for partitions ArrayBuffer([[realtimestreaming,0], initOffset 846
to broker id:1,host:realtimeslave1.novalocal,port:9092] ,
[[realtimestreaming,1], initOffset 957 to broker
id:1,host:realtimeslave1.novalocal,port:9092] )
(kafka.server.ReplicaFetcherManager)*
[2015-01-15 09:57:34,654] INFO [ReplicaFetcherThread-0-1], Starting
(kafka.server.ReplicaFetcherThread)
[2015-01-15 09:57:34,747] INFO [ReplicaFetcherThread-1-1], Starting
(kafka.server.ReplicaFetcherThread)
[2015-01-15 09:58:14,156] INFO Closing socket connection to /10.0.0.11.
(kafka.network.Processor)
On Thu, Jan 15, 2015 at 12:01 PM, Chia-Chun Shih <chiachun.s...@gmail.com>
wrote:
> You can use tools (e.g., VisialVM) to diagnose OOM problem.
>
> 2015-01-15 14:15 GMT+08:00 Tousif Khazi <tou...@senseforth.com>:
>
> > i see this error
> >
> > ERROR [ReplicaFetcherThread-0-1], Error for partition
> > [realtimestreaming,1] to broker 1:class
> > kafka.common.NotLeaderForPartitionException
> > (kafka.server.ReplicaFetcherThread)
> > [2015-01-15 10:00:04,348] INFO [ReplicaFetcherManager on broker 0]
> > Removed fetcher for partitions [realtimestreaming,1]
> > (kafka.server.ReplicaFetcherManager)
> > [2015-01-15 10:00:04,355] INFO Closing socket connection to
> > /10.0.0.11. (kafka.network.Processor)
> > [2015-01-15 10:00:04,444] WARN [KafkaApi-0] Fetch request with
> > correlation id 0 from client ReplicaFetcherThread-0-0 on partition
> > [realtimestreaming,1] failed due to Leader not local for partition
> > [realtimestreaming,1] on broker 0 (kafka.server.KafkaApis)
> > [2015-01-15 10:00:04,545] INFO [ReplicaFetcherThread-0-1], Shutting
> > down (kafka.server.ReplicaFetcherThread)
> > [2015-01-15 10:00:04,848] INFO [ReplicaFetcherThread-0-1], Stopped
> > (kafka.server.ReplicaFetcherThread)
> > [2015-01-15 10:00:04,849] INFO [ReplicaFetcherThread-0-1], Shutdown
> > completed (kafka.server.ReplicaFetcherThread)
> > [2015-01-15 10:00:39,256] ERROR Closing socket for /10.0.0.11 because
> > of error (kafka.network.Processor)
> > java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> >
> > On Wed, Jan 14, 2015 at 10:12 PM, Tousif <tousif.pa...@gmail.com> wrote:
> > > Thanks harsha for quick response.
> > > I don't see any other error. I used to see replica fetcher error but
> > seems
> > > to be disappeared after setting replica fetcher threads to 2 as I have
> 2
> > > partitions. Some times I see zookeeper session expiration.
> > > On Jan 14, 2015 8:02 PM, "Harsha" <ka...@harsha.io> wrote:
> > >
> > >> Tousif,
> > >> Do you see any other errors in server.log
> > >> -Harsha
> > >>
> > >> On Wed, Jan 14, 2015, at 01:51 AM, Tousif wrote:
> > >> > Hello,
> > >> >
> > >> > I have configured kafka nodes to run via supervisord and see
> > following
> > >> > exceptions
> > >> > and eventually brokers going out of memory. i have given enough
> memory
> > >> > and
> > >> > process 1 event/second. kafka goes down every day
> > >> >
> > >> > i'm wondering what configurastion is missing or need to be added
> > >> >
> > >> > Here are my cluster details:
> > >> > 2 brokers
> > >> > 1 zookeeper
> > >> > and 2 node apache storm
> > >> >
> > >> >
> > >> > INFO zookeeper state changed (SyncConnected)
> > >> > (org.I0Itec.zkclient.ZkClient)
> > >> > ERROR Closing socket for /10.0.0.11 because of error
> > >> > (kafka.network.Processor)
> > >> > java.io.IOException: Connection reset by peer
> > >> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > >> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > >> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > >> > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > >> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> > >> > at kafka.utils.Utils$.read(Utils.scala:375)
> > >> > at
> > >> >
> > >>
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > >> > at kafka.network.Processor.read(SocketServer.scala:347)
> > >> > at kafka.network.Processor.run(SocketServer.scala:245)
> > >> > at java.lang.Thread.run(Thread.java:662)
> > >> > [2015-01-13 23:43:37,962] INFO Closing socket connection to /
> > 10.0.0.11.
> > >> > (kafka.network.Processor)
> > >> > Error occurred during initialization of VM
> > >> > Could not reserve enough space for object heap
> > >> > Error occurred during initialization of VM
> > >> > Could not reserve enough space for object heap
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Tousif
> > >> > +918050227279
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> >
> > >> > Regards
> > >> > Tousif Khazi
> > >>
> >
> >
> >
> > --
> > Regards,
> > Tousif
> > +918050227279
> >
>
--
Regards
Tousif Khazi
