new broker instance can't sync up one partition from peers

I did a push. new instance comes up and tries to fetch log/data from other peers/replicas. Out of 60 partitions assigned for broker 0, it sync'ed up 59. but for whatever reason, it didn't try to fetch this partition/topic. [out-of-sync replica] BrokerId: 0, Topic: foo, PartitionId: 6, Leader: 5, Re

Re: Manual Leader Assignment

To avoid data loss, you will need to (1) use ack=-1 in the producer; (2) configure enough retries + backoff time in the producer (so that new leaders can be elected during failure). For controlled failure, you can reduce the unavailability window by using controlled shutdown. See http://kafka.apac

Re: Error in acceptor (kafka.network.Acceptor)

What type of error did you see? You may need to configure a larger open file handler limit. Thanks, Jun On Wed, Sep 3, 2014 at 12:01 PM, Shlomi Hazan wrote: > Hi, > > I am trying to load a cluster with over than 10K connections, and bumped > into the error in the subject. > Is there any limita

Re: message size limit

Are you using compression in the producer? If so, message.max.bytes applies to the compressed size of a batch of messages. Otherwise, message.max.bytes applies to the size of each individual message. Thanks, Jun On Wed, Sep 3, 2014 at 3:25 PM, Bhavesh Mistry wrote: > I am referring to wiki htt

zookeeper shows partition in-sync but local disk doesn't have the data dirs

did a push in cloud. after new instance for broker 0 comes up, I see a lot of WARNs in log file. 2014-09-09 04:21:09,271 WARN kafka.utils.Logging$class:83 [request-expiration-task] [warn] [KafkaApi-0] Fetch request with correlation id 51893 from client 1409779957450-6014fc32-0-0 on partition [foo

Re: new broker instance can't sync up one partition from peers

Which version of Kafka are you using? Any error in the controller/state-change log? Thanks, Jun On Mon, Sep 8, 2014 at 9:07 PM, Steven Wu wrote: > I did a push. new instance comes up and tries to fetch log/data from other > peers/replicas. Out of 60 partitions assigned for broker 0, it sync'ed

Re: new broker instance can't sync up one partition from peers

sorry. 0.8.1.1. couldn't find anything relevant to this issue in log files. but found a different issue that I will ask in a separate thread. On Mon, Sep 8, 2014 at 9:33 PM, Jun Rao wrote: > Which version of Kafka are you using? Any error in the > controller/state-change log? > > Thanks, > > J

Re: zookeeper shows partition in-sync but local disk doesn't have the data dirs

sorry. forgot to mention that I am running 0.8.1.1 On Mon, Sep 8, 2014 at 9:26 PM, Steven Wu wrote: > did a push in cloud. after new instance for broker 0 comes up, I see a lot > of WARNs in log file. > > 2014-09-09 04:21:09,271 WARN kafka.utils.Logging$class:83 > [request-expiration-task] [wa