Re: Created topic by 2 partitions, only can use the one partition
Anyone has idea? Thank you Sent from my iPhone > On Sep 30, 2014, at 1:45 PM, Jiang Jacky wrote: > > Hi, Guys > It is very weird, I created a topic with 2 partitions couple weeks ago, and I > can only production the message to partition 0, not partition 1, but for now, > I created a new topic again with 2 partitions, it does work. > So whats problem of the old topic? I tried to describe the old topic, I found > the following message > > Topic: Partition: 0Leader: 1 Replicas: 1 Isr: > 1 > Topic: Partition: 1Leader: 2 Replicas: 2 Isr: > 2 > > Please let me know, if the topic is screwed up. > > Thank you > >
Created topic by 2 partitions, only can use the one partition
Hi, Guys It is very weird, I created a topic with 2 partitions couple weeks ago, and I can only production the message to partition 0, not partition 1, but for now, I created a new topic again with 2 partitions, it does work. So whats problem of the old topic? I tried to describe the old topic, I found the following message Topic: Partition: 0Leader: 1 Replicas: 1 Isr: 1 Topic: Partition: 1Leader: 2 Replicas: 2 Isr: 2 Please let me know, if the topic is screwed up. Thank you
Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine
It is very weird, I have a kafka cluster in EC2, There is no any problem to produce message from one of node by same producer. But when I move the producer to my local machine at home, then it gives me the below error: Failed to send messages after 3 tries. Can anyone tell me how do I fix this issue? I have opened all ports of my machine at my home, and the security group is also opened for kafka server and zookeeper in EC2. Everything is fine, but I just cannot send any message to kafka server. Please help me. Thanks Everyone! Jacky
Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine
Hi, I tried to setup the host.name in servier.properties, it doesn't work. I believe it is the network security issue. However, I create a new instance in the same security group without kafka, zookeeper, it does work, it can still produce to kafka server. but when I change to another ec2 account, then create the same instance, and it cannot produce to kafka server. I pay attention that there is no outbound port setting in the security group configuration of kafka server ec2, I think if we shall have to set the outbound port for the firewall? Do you guys know which outbound port should be opened for kafka server? Thanks 2013/10/5 Guozhang Wang > Hello Jacky, > > Have you read this FAQ: > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F > > Guozhang > > > On Fri, Oct 4, 2013 at 10:41 PM, Jiang Jacky wrote: > > > It is very weird, I have a kafka cluster in EC2, There is no any problem > to > > produce message from one of node by same producer. But when I move the > > producer to my local machine at home, then it gives me the below error: > > Failed to send messages after 3 tries. > > > > Can anyone tell me how do I fix this issue? I have opened all ports of my > > machine at my home, and the security group is also opened for kafka > server > > and zookeeper in EC2. Everything is fine, but I just cannot send any > > message to kafka server. > > Please help me. > > Thanks Everyone! > > > > Jacky > > > > > > -- > -- Guozhang >
Kafka cannot produce message
Hi, I run into kafka issue recently, when I try to produce a message, it
gives me follow error
[2013-10-11 00:30:26,740] INFO Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s)
Set(my-replicated-topic) (kafka.client.ClientUtils$)
[2013-10-11 00:30:26,741] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2013-10-11 00:30:26,753] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2013-10-11 00:30:26,753] INFO Closing socket connection to /127.0.0.1.
(kafka.network.Processor)
[2013-10-11 00:30:26,754] WARN Error while fetching metadata
[{TopicMetadata for topic my-replicated-topic ->
No partition metadata for topic my-replicated-topic due to
kafka.common.LeaderNotAvailableException}] for topic [my-replicated-topic]:
class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2013-10-11 00:30:26,755] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic:
my-replicated-topic (kafka.producer.async.DefaultEventHandler)
[2013-10-11 00:30:26,755] INFO Back off for 100 ms before retrying send.
Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
[2013-10-11 00:30:26,856] INFO Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 48 for 1 topic(s)
Set(my-replicated-topic) (kafka.client.ClientUtils$)
[2013-10-11 00:30:26,857] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2013-10-11 00:30:26,874] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2013-10-11 00:30:26,874] INFO Closing socket connection to /127.0.0.1.
(kafka.network.Processor)
[2013-10-11 00:30:26,875] WARN Error while fetching metadata
[{TopicMetadata for topic my-replicated-topic ->
No partition metadata for topic my-replicated-topic due to
kafka.common.LeaderNotAvailableException}] for topic [my-replicated-topic]:
class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2013-10-11 00:30:26,876] ERROR Failed to send requests for topics
my-replicated-topic with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2013-10-11 00:30:26,876] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
I have no idea why the leader is not available.Can someone give me a hint?
Thanks a lot.
Re: Kafka cannot produce message
can anyone help me? Thanks a lot
2013/10/10 Jiang Jacky
> Hi, I run into kafka issue recently, when I try to produce a message, it
> gives me follow error
>
> [2013-10-11 00:30:26,740] INFO Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s)
> Set(my-replicated-topic) (kafka.client.ClientUtils$)
> [2013-10-11 00:30:26,741] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2013-10-11 00:30:26,753] INFO Disconnecting from localhost:9092
> (kafka.producer.SyncProducer)
> [2013-10-11 00:30:26,753] INFO Closing socket connection to /127.0.0.1.
> (kafka.network.Processor)
> [2013-10-11 00:30:26,754] WARN Error while fetching metadata
> [{TopicMetadata for topic my-replicated-topic ->
> No partition metadata for topic my-replicated-topic due to
> kafka.common.LeaderNotAvailableException}] for topic [my-replicated-topic]:
> class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2013-10-11 00:30:26,755] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic:
> my-replicated-topic (kafka.producer.async.DefaultEventHandler)
> [2013-10-11 00:30:26,755] INFO Back off for 100 ms before retrying send.
> Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
> [2013-10-11 00:30:26,856] INFO Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 48 for 1 topic(s)
> Set(my-replicated-topic) (kafka.client.ClientUtils$)
> [2013-10-11 00:30:26,857] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2013-10-11 00:30:26,874] INFO Disconnecting from localhost:9092
> (kafka.producer.SyncProducer)
> [2013-10-11 00:30:26,874] INFO Closing socket connection to /127.0.0.1.
> (kafka.network.Processor)
> [2013-10-11 00:30:26,875] WARN Error while fetching metadata
> [{TopicMetadata for topic my-replicated-topic ->
> No partition metadata for topic my-replicated-topic due to
> kafka.common.LeaderNotAvailableException}] for topic [my-replicated-topic]:
> class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2013-10-11 00:30:26,876] ERROR Failed to send requests for topics
> my-replicated-topic with correlation ids in [41,48]
> (kafka.producer.async.DefaultEventHandler)
> [2013-10-11 00:30:26,876] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
>
> I have no idea why the leader is not available.Can someone give me a hint?
> Thanks a lot.
>
Re: Kafka cannot produce message
yes, even I cannot create new topic, same error, leadernotavailable.
My zookeeper and storm all are working very well, except of Kafka.
2013/10/10 Neha Narkhede
> Are you trying to produce these messages to a newly created topic?
>
>
> On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky wrote:
>
> > can anyone help me? Thanks a lot
> >
> >
> > 2013/10/10 Jiang Jacky
> >
> > > Hi, I run into kafka issue recently, when I try to produce a message,
> it
> > > gives me follow error
> > >
> > > [2013-10-11 00:30:26,740] INFO Fetching metadata from broker
> > > id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s)
> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
> > > [2013-10-11 00:30:26,741] INFO Connected to localhost:9092 for
> producing
> > > (kafka.producer.SyncProducer)
> > > [2013-10-11 00:30:26,753] INFO Disconnecting from localhost:9092
> > > (kafka.producer.SyncProducer)
> > > [2013-10-11 00:30:26,753] INFO Closing socket connection to /127.0.0.1
> .
> > > (kafka.network.Processor)
> > > [2013-10-11 00:30:26,754] WARN Error while fetching metadata
> > > [{TopicMetadata for topic my-replicated-topic ->
> > > No partition metadata for topic my-replicated-topic due to
> > > kafka.common.LeaderNotAvailableException}] for topic
> > [my-replicated-topic]:
> > > class kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2013-10-11 00:30:26,755] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic:
> > > my-replicated-topic (kafka.producer.async.DefaultEventHandler)
> > > [2013-10-11 00:30:26,755] INFO Back off for 100 ms before retrying
> send.
> > > Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
> > > [2013-10-11 00:30:26,856] INFO Fetching metadata from broker
> > > id:0,host:localhost,port:9092 with correlation id 48 for 1 topic(s)
> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
> > > [2013-10-11 00:30:26,857] INFO Connected to localhost:9092 for
> producing
> > > (kafka.producer.SyncProducer)
> > > [2013-10-11 00:30:26,874] INFO Disconnecting from localhost:9092
> > > (kafka.producer.SyncProducer)
> > > [2013-10-11 00:30:26,874] INFO Closing socket connection to /127.0.0.1
> .
> > > (kafka.network.Processor)
> > > [2013-10-11 00:30:26,875] WARN Error while fetching metadata
> > > [{TopicMetadata for topic my-replicated-topic ->
> > > No partition metadata for topic my-replicated-topic due to
> > > kafka.common.LeaderNotAvailableException}] for topic
> > [my-replicated-topic]:
> > > class kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2013-10-11 00:30:26,876] ERROR Failed to send requests for topics
> > > my-replicated-topic with correlation ids in [41,48]
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2013-10-11 00:30:26,876] ERROR Error in handling batch of 1 events
> > > (kafka.producer.async.ProducerSendThread)
> > > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> > 3
> > > tries.
> > > at
> > >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > at
> > >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > at
> > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > at
> > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > > at
> > >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > at
> > >
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > >
> > >
> > > I have no idea why the leader is not available.Can someone give me a
> > hint?
> > > Thanks a lot.
> > >
> >
>
Re: Kafka cannot produce message
Another thing need is my instance comes with very small memory, just 683M,
I am not sure if it is related. I will try to use 8G memory instance.
2013/10/10 Jiang Jacky
> yes, even I cannot create new topic, same error, leadernotavailable.
> My zookeeper and storm all are working very well, except of Kafka.
>
>
> 2013/10/10 Neha Narkhede
>
>> Are you trying to produce these messages to a newly created topic?
>>
>>
>> On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky wrote:
>>
>> > can anyone help me? Thanks a lot
>> >
>> >
>> > 2013/10/10 Jiang Jacky
>> >
>> > > Hi, I run into kafka issue recently, when I try to produce a message,
>> it
>> > > gives me follow error
>> > >
>> > > [2013-10-11 00:30:26,740] INFO Fetching metadata from broker
>> > > id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s)
>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
>> > > [2013-10-11 00:30:26,741] INFO Connected to localhost:9092 for
>> producing
>> > > (kafka.producer.SyncProducer)
>> > > [2013-10-11 00:30:26,753] INFO Disconnecting from localhost:9092
>> > > (kafka.producer.SyncProducer)
>> > > [2013-10-11 00:30:26,753] INFO Closing socket connection to /
>> 127.0.0.1.
>> > > (kafka.network.Processor)
>> > > [2013-10-11 00:30:26,754] WARN Error while fetching metadata
>> > > [{TopicMetadata for topic my-replicated-topic ->
>> > > No partition metadata for topic my-replicated-topic due to
>> > > kafka.common.LeaderNotAvailableException}] for topic
>> > [my-replicated-topic]:
>> > > class kafka.common.LeaderNotAvailableException
>> > > (kafka.producer.BrokerPartitionInfo)
>> > > [2013-10-11 00:30:26,755] ERROR Failed to collate messages by topic,
>> > > partition due to: Failed to fetch topic metadata for topic:
>> > > my-replicated-topic (kafka.producer.async.DefaultEventHandler)
>> > > [2013-10-11 00:30:26,755] INFO Back off for 100 ms before retrying
>> send.
>> > > Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
>> > > [2013-10-11 00:30:26,856] INFO Fetching metadata from broker
>> > > id:0,host:localhost,port:9092 with correlation id 48 for 1 topic(s)
>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
>> > > [2013-10-11 00:30:26,857] INFO Connected to localhost:9092 for
>> producing
>> > > (kafka.producer.SyncProducer)
>> > > [2013-10-11 00:30:26,874] INFO Disconnecting from localhost:9092
>> > > (kafka.producer.SyncProducer)
>> > > [2013-10-11 00:30:26,874] INFO Closing socket connection to /
>> 127.0.0.1.
>> > > (kafka.network.Processor)
>> > > [2013-10-11 00:30:26,875] WARN Error while fetching metadata
>> > > [{TopicMetadata for topic my-replicated-topic ->
>> > > No partition metadata for topic my-replicated-topic due to
>> > > kafka.common.LeaderNotAvailableException}] for topic
>> > [my-replicated-topic]:
>> > > class kafka.common.LeaderNotAvailableException
>> > > (kafka.producer.BrokerPartitionInfo)
>> > > [2013-10-11 00:30:26,876] ERROR Failed to send requests for topics
>> > > my-replicated-topic with correlation ids in [41,48]
>> > > (kafka.producer.async.DefaultEventHandler)
>> > > [2013-10-11 00:30:26,876] ERROR Error in handling batch of 1 events
>> > > (kafka.producer.async.ProducerSendThread)
>> > > kafka.common.FailedToSendMessageException: Failed to send messages
>> after
>> > 3
>> > > tries.
>> > > at
>> > >
>> >
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> > > at
>> > >
>> >
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>> > > at
>> > >
>> >
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
>> > > at
>> > >
>> >
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
>> > > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
>> > > at
>> > >
>> >
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
>> > > at
>> > >
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>> > >
>> > >
>> > > I have no idea why the leader is not available.Can someone give me a
>> > hint?
>> > > Thanks a lot.
>> > >
>> >
>>
>
>
Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine
did anyone experience same issue? can you share with me? Thanks a lot. 2013/10/7 Jun Rao > Kafka port is documented in > http://kafka.apache.org/documentation.html#brokerconfigs > > Thanks, > > Jun > > > On Sat, Oct 5, 2013 at 12:05 PM, Jiang Jacky wrote: > > > Hi, I tried to setup the host.name in servier.properties, it doesn't > work. > > I believe it is the network security issue. However, I create a new > > instance in the same security group without kafka, zookeeper, it does > work, > > it can still produce to kafka server. but when I change to another ec2 > > account, then create the same instance, and it cannot produce to kafka > > server. > > I pay attention that there is no outbound port setting in the security > > group configuration of kafka server ec2, I think if we shall have to set > > the outbound port for the firewall? > > Do you guys know which outbound port should be opened for kafka server? > > Thanks > > > > > > 2013/10/5 Guozhang Wang > > > > > Hello Jacky, > > > > > > Have you read this FAQ: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F > > > > > > Guozhang > > > > > > > > > On Fri, Oct 4, 2013 at 10:41 PM, Jiang Jacky > > wrote: > > > > > > > It is very weird, I have a kafka cluster in EC2, There is no any > > problem > > > to > > > > produce message from one of node by same producer. But when I move > the > > > > producer to my local machine at home, then it gives me the below > error: > > > > Failed to send messages after 3 tries. > > > > > > > > Can anyone tell me how do I fix this issue? I have opened all ports > of > > my > > > > machine at my home, and the security group is also opened for kafka > > > server > > > > and zookeeper in EC2. Everything is fine, but I just cannot send any > > > > message to kafka server. > > > > Please help me. > > > > Thanks Everyone! > > > > > > > > Jacky > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > >
Re: Kafka cannot produce message
Hi, Everyone, I use the larger memory instance to replace that small one,
for now, I can create the topic, but when I try to produce message, it
still gives the same error.
Its so weird. can any one help me? Thanks
2013/10/10 Jiang Jacky
> Another thing need is my instance comes with very small memory, just 683M,
> I am not sure if it is related. I will try to use 8G memory instance.
>
>
> 2013/10/10 Jiang Jacky
>
>> yes, even I cannot create new topic, same error, leadernotavailable.
>> My zookeeper and storm all are working very well, except of Kafka.
>>
>>
>> 2013/10/10 Neha Narkhede
>>
>>> Are you trying to produce these messages to a newly created topic?
>>>
>>>
>>> On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky
>>> wrote:
>>>
>>> > can anyone help me? Thanks a lot
>>> >
>>> >
>>> > 2013/10/10 Jiang Jacky
>>> >
>>> > > Hi, I run into kafka issue recently, when I try to produce a
>>> message, it
>>> > > gives me follow error
>>> > >
>>> > > [2013-10-11 00:30:26,740] INFO Fetching metadata from broker
>>> > > id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s)
>>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
>>> > > [2013-10-11 00:30:26,741] INFO Connected to localhost:9092 for
>>> producing
>>> > > (kafka.producer.SyncProducer)
>>> > > [2013-10-11 00:30:26,753] INFO Disconnecting from localhost:9092
>>> > > (kafka.producer.SyncProducer)
>>> > > [2013-10-11 00:30:26,753] INFO Closing socket connection to /
>>> 127.0.0.1.
>>> > > (kafka.network.Processor)
>>> > > [2013-10-11 00:30:26,754] WARN Error while fetching metadata
>>> > > [{TopicMetadata for topic my-replicated-topic ->
>>> > > No partition metadata for topic my-replicated-topic due to
>>> > > kafka.common.LeaderNotAvailableException}] for topic
>>> > [my-replicated-topic]:
>>> > > class kafka.common.LeaderNotAvailableException
>>> > > (kafka.producer.BrokerPartitionInfo)
>>> > > [2013-10-11 00:30:26,755] ERROR Failed to collate messages by topic,
>>> > > partition due to: Failed to fetch topic metadata for topic:
>>> > > my-replicated-topic (kafka.producer.async.DefaultEventHandler)
>>> > > [2013-10-11 00:30:26,755] INFO Back off for 100 ms before retrying
>>> send.
>>> > > Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
>>> > > [2013-10-11 00:30:26,856] INFO Fetching metadata from broker
>>> > > id:0,host:localhost,port:9092 with correlation id 48 for 1 topic(s)
>>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
>>> > > [2013-10-11 00:30:26,857] INFO Connected to localhost:9092 for
>>> producing
>>> > > (kafka.producer.SyncProducer)
>>> > > [2013-10-11 00:30:26,874] INFO Disconnecting from localhost:9092
>>> > > (kafka.producer.SyncProducer)
>>> > > [2013-10-11 00:30:26,874] INFO Closing socket connection to /
>>> 127.0.0.1.
>>> > > (kafka.network.Processor)
>>> > > [2013-10-11 00:30:26,875] WARN Error while fetching metadata
>>> > > [{TopicMetadata for topic my-replicated-topic ->
>>> > > No partition metadata for topic my-replicated-topic due to
>>> > > kafka.common.LeaderNotAvailableException}] for topic
>>> > [my-replicated-topic]:
>>> > > class kafka.common.LeaderNotAvailableException
>>> > > (kafka.producer.BrokerPartitionInfo)
>>> > > [2013-10-11 00:30:26,876] ERROR Failed to send requests for topics
>>> > > my-replicated-topic with correlation ids in [41,48]
>>> > > (kafka.producer.async.DefaultEventHandler)
>>> > > [2013-10-11 00:30:26,876] ERROR Error in handling batch of 1 events
>>> > > (kafka.producer.async.ProducerSendThread)
>>> > > kafka.common.FailedToSendMessageException: Failed to send messages
>>> after
>>> > 3
>>> > > tries.
>>> > > at
>>> > >
>>> >
>>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>>> > > at
>>> > >
>>> >
>>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>>> > > at
>>> > >
>>> >
>>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
>>> > > at
>>> > >
>>> >
>>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
>>> > > at
>>> scala.collection.immutable.Stream.foreach(Stream.scala:254)
>>> > > at
>>> > >
>>> >
>>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
>>> > > at
>>> > >
>>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>>> > >
>>> > >
>>> > > I have no idea why the leader is not available.Can someone give me a
>>> > hint?
>>> > > Thanks a lot.
>>> > >
>>> >
>>>
>>
>>
>
Kafka server error
*Hi, Guys,* *I am currently running into the kafka server issue. * *I have a 5 nodes cluster and zookeeper running without any problem. when I manually boot each node by using* "*JMX_PORT=9997 bin/kafka-server-start.sh config/server-x.properties &*" command. *The scenario is:* *Then, first node, it can be booted.* *Once I boot the second node, it is crashed, the error is below:* [2013-10-11 04:02:17,200] INFO [Replica Manager on Broker 0]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) -> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 (kafka.server.ReplicaManager) [2013-10-11 04:02:17,204] WARN No previously checkpointed highwatermark value found for topic test-kafka partition 0. Returning 0 as the highwatermark (kafka.server.HighwaterMarkCheckpoint) [2013-10-11 04:02:17,205] INFO [ReplicaFetcherManager on broker 0] Removing fetcher for partition [test-kafka,0] (kafka.server.ReplicaFetcherManager) [2013-10-11 04:02:17,214] INFO [Kafka Log on Broker 0], Truncated log segment /tmp/kafka-logs/test-kafka-0/.log to target offset 0 (kafka.log.Log) [2013-10-11 04:02:17,235] INFO [ReplicaFetcherManager on broker 0] Adding fetcher for partition [test-kafka,0], initOffset 0 to broker 1 with fetcherId 0 (kafka.server.ReplicaFetcherManager) [2013-10-11 04:02:17,236] INFO [Replica Manager on Broker 0]: Handled leader and isr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) -> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 (kafka.server.ReplicaManager) [2013-10-11 04:02:17,240] INFO [ReplicaFetcherThread-0-1], Starting (kafka.server.ReplicaFetcherThread) [2013-10-11 04:02:17,266] INFO [Replica Manager on Broker 0]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) -> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:91,ControllerEpoch:30416),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 (kafka.server.ReplicaManager) [2013-10-11 04:02:17,267] INFO [ReplicaFetcherManager on broker 0] Removing fetcher for partition [test-kafka,0] (kafka.server.ReplicaFetcherManager) [2013-10-11 04:02:17,268] INFO [Kafka Log on Broker 0], Truncated log segment /tmp/kafka-logs/test-kafka-0/.log to target offset 0 (kafka.log.Log) [2013-10-11 04:02:17,268] INFO [ReplicaFetcherManager on broker 0] Adding fetcher for partition [test-kafka,0], initOffset 0 to broker 1 with fetcherId 0 (kafka.server.ReplicaFetcherManager) [2013-10-11 04:02:17,269] INFO [Replica Manager on Broker 0]: Handled leader and isr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) -> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:91,ControllerEpoch:30416),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 (kafka.server.ReplicaManager) [2013-10-11 04:02:17,269] ERROR [Kafka Request Handler 0 on Broker 0], Exception when handling request (kafka.server.KafkaRequestHandler) [2013-10-11 04:02:47,284] INFO Reconnect due to socket error: (kafka.consumer.SimpleConsumer) java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at kafka.utils.Utils$.read(Utils.scala:394) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109) at ka
Re: Kafka cannot produce message
Hi, Aniket
Thanks for your reply
I checked the log from /tmp/kafka-logs/
There are some empty files, likes .index,
.log. I cannot find any useful information there.
However, i found some other interesting things, I have posted here
https://mail.google.com/mail/ca/u/0/#label/TechDiscuss/141a5c074641dd70
Please have a look. Thanks a lot.
2013/10/11 Aniket Bhatnagar
> Hi Jiang
>
> Just thinking out loud, it could be because Kafka can't connect to
> Zookeeper. Can you check Kafka logs to validate the same?
>
>
> On 11 October 2013 08:22, Jiang Jacky wrote:
>
> > Hi, Everyone, I use the larger memory instance to replace that small one,
> > for now, I can create the topic, but when I try to produce message, it
> > still gives the same error.
> > Its so weird. can any one help me? Thanks
> >
> >
> > 2013/10/10 Jiang Jacky
> >
> > > Another thing need is my instance comes with very small memory, just
> > 683M,
> > > I am not sure if it is related. I will try to use 8G memory instance.
> > >
> > >
> > > 2013/10/10 Jiang Jacky
> > >
> > >> yes, even I cannot create new topic, same error, leadernotavailable.
> > >> My zookeeper and storm all are working very well, except of Kafka.
> > >>
> > >>
> > >> 2013/10/10 Neha Narkhede
> > >>
> > >>> Are you trying to produce these messages to a newly created topic?
> > >>>
> > >>>
> > >>> On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky
> > >>> wrote:
> > >>>
> > >>> > can anyone help me? Thanks a lot
> > >>> >
> > >>> >
> > >>> > 2013/10/10 Jiang Jacky
> > >>> >
> > >>> > > Hi, I run into kafka issue recently, when I try to produce a
> > >>> message, it
> > >>> > > gives me follow error
> > >>> > >
> > >>> > > [2013-10-11 00:30:26,740] INFO Fetching metadata from broker
> > >>> > > id:0,host:localhost,port:9092 with correlation id 47 for 1
> topic(s)
> > >>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
> > >>> > > [2013-10-11 00:30:26,741] INFO Connected to localhost:9092 for
> > >>> producing
> > >>> > > (kafka.producer.SyncProducer)
> > >>> > > [2013-10-11 00:30:26,753] INFO Disconnecting from localhost:9092
> > >>> > > (kafka.producer.SyncProducer)
> > >>> > > [2013-10-11 00:30:26,753] INFO Closing socket connection to /
> > >>> 127.0.0.1.
> > >>> > > (kafka.network.Processor)
> > >>> > > [2013-10-11 00:30:26,754] WARN Error while fetching metadata
> > >>> > > [{TopicMetadata for topic my-replicated-topic ->
> > >>> > > No partition metadata for topic my-replicated-topic due to
> > >>> > > kafka.common.LeaderNotAvailableException}] for topic
> > >>> > [my-replicated-topic]:
> > >>> > > class kafka.common.LeaderNotAvailableException
> > >>> > > (kafka.producer.BrokerPartitionInfo)
> > >>> > > [2013-10-11 00:30:26,755] ERROR Failed to collate messages by
> > topic,
> > >>> > > partition due to: Failed to fetch topic metadata for topic:
> > >>> > > my-replicated-topic (kafka.producer.async.DefaultEventHandler)
> > >>> > > [2013-10-11 00:30:26,755] INFO Back off for 100 ms before
> retrying
> > >>> send.
> > >>> > > Remaining retries = 0 (kafka.producer.async.DefaultEventHandler)
> > >>> > > [2013-10-11 00:30:26,856] INFO Fetching metadata from broker
> > >>> > > id:0,host:localhost,port:9092 with correlation id 48 for 1
> topic(s)
> > >>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
> > >>> > > [2013-10-11 00:30:26,857] INFO Connected to localhost:9092 for
> > >>> producing
> > >>> > > (kafka.producer.SyncProducer)
> > >>> > > [2013-10-11 00:30:26,874] INFO Disconnecting from localhost:9092
> > >>> > > (kafka.producer.SyncProducer)
> > >>> > > [2013-10-11 00:30:26,874] INFO Closing socket connection to /
> > >>> 127.0.0.1.
> > >>> > > (kafka.network.Processor)
> > >>> > > [2013-10-11 00:30:26,875] WARN Error wh
Re: Kafka server error
Yes, it just says "INFO Reconnect due to socket error" But why and how come it comes? my zookeeper and storm have no any problem to collaborate each other. 2013/10/11 Jun Rao > The log you posted for the second broker didn't say why it crashed. Is that > all you got? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky wrote: > > > *Hi, Guys,* > > *I am currently running into the kafka server issue. * > > *I have a 5 nodes cluster and zookeeper running without any problem. > when I > > manually boot each node by using* "*JMX_PORT=9997 > bin/kafka-server-start.sh > > config/server-x.properties &*" command. > > > > *The scenario is:* > > *Then, first node, it can be booted.* > > *Once I boot the second node, it is crashed, the error is below:* > > > > [2013-10-11 04:02:17,200] INFO [Replica Manager on Broker 0]: Handling > > LeaderAndIsr request > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > -> > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > (kafka.server.ReplicaManager) > > [2013-10-11 04:02:17,204] WARN No previously checkpointed highwatermark > > value found for topic test-kafka partition 0. Returning 0 as the > > highwatermark (kafka.server.HighwaterMarkCheckpoint) > > [2013-10-11 04:02:17,205] INFO [ReplicaFetcherManager on broker 0] > Removing > > fetcher for partition [test-kafka,0] (kafka.server.ReplicaFetcherManager) > > [2013-10-11 04:02:17,214] INFO [Kafka Log on Broker 0], Truncated log > > segment /tmp/kafka-logs/test-kafka-0/.log to target > > offset 0 (kafka.log.Log) > > [2013-10-11 04:02:17,235] INFO [ReplicaFetcherManager on broker 0] Adding > > fetcher for partition [test-kafka,0], initOffset 0 to broker 1 with > > fetcherId 0 (kafka.server.ReplicaFetcherManager) > > [2013-10-11 04:02:17,236] INFO [Replica Manager on Broker 0]: Handled > > leader and isr request > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > -> > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > (kafka.server.ReplicaManager) > > [2013-10-11 04:02:17,240] INFO [ReplicaFetcherThread-0-1], Starting > > (kafka.server.ReplicaFetcherThread) > > [2013-10-11 04:02:17,266] INFO [Replica Manager on Broker 0]: Handling > > LeaderAndIsr request > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > -> > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:91,ControllerEpoch:30416),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > (kafka.server.ReplicaManager) > > [2013-10-11 04:02:17,267] INFO [ReplicaFetcherManager on broker 0] > Removing > > fetcher for partition [test-kafka,0] (kafka.server.ReplicaFetcherManager) > > [2013-10-11 04:02:17,268] INFO [Kafka Log on Broker 0], Truncated log > > segment /tmp/kafka-logs/test-kafka-0/.log to target > > offset 0 (kafka.log.Log) > > [2013-10-11 04:02:17,268] INFO [ReplicaFetcherManager on broker 0] Adding > > fetcher for partition [test-kafka,0], initOffset 0 to broker 1 with > > fetcherId 0 (kafka.server.ReplicaFetcherManager) > > [2013-10-11 04:02:17,269] INFO [Replica Manager on Broker 0]: Handled > > leader and isr request > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > -> > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:91,ControllerEpoch:30416),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > (kafka.server.ReplicaManager) > > [2013-10-11 04:02:17,269] ERROR [Kafka Request Handler 0 on Broker 0], > > Exception when handling request (kafka.server.KafkaRequestHandler) > > [2013-10-11 04:02:47,284] INFO Reconnect due to socket error: > > (kafka.consumer.SimpleConsumer) > > java.net.SocketTimeoutException > > at > > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229) > > at > sun.nio.ch.ChannelInputStream.read(Channe
Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine
Hi, Thanks Joe I have disabled the firewall, but I did not tried that again, I changed another way by using Netty sending the socket package to kafka server, then it will produce in the local. But recently I suddenly ran into below issue, can you have a look? https://mail.google.com/mail/ca/u/0/#label/TechDiscuss/141a5c074641dd70 Thanks. 2013/10/10 Joseph Lawson > another thing to note is that your brokers may be running on port 6667 > instead of 9092. it appears that the default recently changed from 9092 to > 6667 so make sure if you have firewall rules for 9092 that you specifically > set the port setting in your configuration to what you expect. -Joe > > Sent from my Droid Charge on Verizon 4G LTE Joseph Lawson wrote: > it may be that the broker is reporting its internal ip address to > zookeeper which then may interfere with your ability to submit the the > broker with a producer that isn't on AWS's network. you can make the > broker report the public dns name instead of the internal host name which i > believe it reports by default. just update the host.name setting on the > broker to the public dns address. hope this helps .. > > -Joe > > Sent from my Droid Charge on Verizon 4G LTE Jiang Jacky wrote: > did anyone experience same issue? can you share with me? > Thanks a lot. > > > 2013/10/7 Jun Rao > > > Kafka port is documented in > > http://kafka.apache.org/documentation.html#brokerconfigs > > > > Thanks, > > > > Jun > > > > > > On Sat, Oct 5, 2013 at 12:05 PM, Jiang Jacky > wrote: > > > > > Hi, I tried to setup the host.name in servier.properties, it doesn't > > work. > > > I believe it is the network security issue. However, I create a new > > > instance in the same security group without kafka, zookeeper, it does > > work, > > > it can still produce to kafka server. but when I change to another ec2 > > > account, then create the same instance, and it cannot produce to kafka > > > server. > > > I pay attention that there is no outbound port setting in the security > > > group configuration of kafka server ec2, I think if we shall have to > set > > > the outbound port for the firewall? > > > Do you guys know which outbound port should be opened for kafka server? > > > Thanks > > > > > > > > > 2013/10/5 Guozhang Wang > > > > > > > Hello Jacky, > > > > > > > > Have you read this FAQ: > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F > > > > > > > > Guozhang > > > > > > > > > > > > On Fri, Oct 4, 2013 at 10:41 PM, Jiang Jacky > > > wrote: > > > > > > > > > It is very weird, I have a kafka cluster in EC2, There is no any > > > problem > > > > to > > > > > produce message from one of node by same producer. But when I move > > the > > > > > producer to my local machine at home, then it gives me the below > > error: > > > > > Failed to send messages after 3 tries. > > > > > > > > > > Can anyone tell me how do I fix this issue? I have opened all ports > > of > > > my > > > > > machine at my home, and the security group is also opened for kafka > > > > server > > > > > and zookeeper in EC2. Everything is fine, but I just cannot send > any > > > > > message to kafka server. > > > > > Please help me. > > > > > Thanks Everyone! > > > > > > > > > > Jacky > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > >
Re: Kafka cannot produce message
Thanks Jun Rao
Where can I find those logs?
2013/10/11 Jun Rao
> Any error in the controller and state-change log?
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 10, 2013 at 9:38 PM, Jiang Jacky wrote:
>
> > Hi, Aniket
> > Thanks for your reply
> > I checked the log from /tmp/kafka-logs/
> > There are some empty files, likes .index,
> > .log. I cannot find any useful information there.
> > However, i found some other interesting things, I have posted here
> > https://mail.google.com/mail/ca/u/0/#label/TechDiscuss/141a5c074641dd70
> > Please have a look. Thanks a lot.
> >
> >
> > 2013/10/11 Aniket Bhatnagar
> >
> > > Hi Jiang
> > >
> > > Just thinking out loud, it could be because Kafka can't connect to
> > > Zookeeper. Can you check Kafka logs to validate the same?
> > >
> > >
> > > On 11 October 2013 08:22, Jiang Jacky wrote:
> > >
> > > > Hi, Everyone, I use the larger memory instance to replace that small
> > one,
> > > > for now, I can create the topic, but when I try to produce message,
> it
> > > > still gives the same error.
> > > > Its so weird. can any one help me? Thanks
> > > >
> > > >
> > > > 2013/10/10 Jiang Jacky
> > > >
> > > > > Another thing need is my instance comes with very small memory,
> just
> > > > 683M,
> > > > > I am not sure if it is related. I will try to use 8G memory
> instance.
> > > > >
> > > > >
> > > > > 2013/10/10 Jiang Jacky
> > > > >
> > > > >> yes, even I cannot create new topic, same error,
> leadernotavailable.
> > > > >> My zookeeper and storm all are working very well, except of Kafka.
> > > > >>
> > > > >>
> > > > >> 2013/10/10 Neha Narkhede
> > > > >>
> > > > >>> Are you trying to produce these messages to a newly created
> topic?
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky <
> jiang0...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > can anyone help me? Thanks a lot
> > > > >>> >
> > > > >>> >
> > > > >>> > 2013/10/10 Jiang Jacky
> > > > >>> >
> > > > >>> > > Hi, I run into kafka issue recently, when I try to produce a
> > > > >>> message, it
> > > > >>> > > gives me follow error
> > > > >>> > >
> > > > >>> > > [2013-10-11 00:30:26,740] INFO Fetching metadata from broker
> > > > >>> > > id:0,host:localhost,port:9092 with correlation id 47 for 1
> > > topic(s)
> > > > >>> > > Set(my-replicated-topic) (kafka.client.ClientUtils$)
> > > > >>> > > [2013-10-11 00:30:26,741] INFO Connected to localhost:9092
> for
> > > > >>> producing
> > > > >>> > > (kafka.producer.SyncProducer)
> > > > >>> > > [2013-10-11 00:30:26,753] INFO Disconnecting from
> > localhost:9092
> > > > >>> > > (kafka.producer.SyncProducer)
> > > > >>> > > [2013-10-11 00:30:26,753] INFO Closing socket connection to /
> > > > >>> 127.0.0.1.
> > > > >>> > > (kafka.network.Processor)
> > > > >>> > > [2013-10-11 00:30:26,754] WARN Error while fetching metadata
> > > > >>> > > [{TopicMetadata for topic my-replicated-topic ->
> > > > >>> > > No partition metadata for topic my-replicated-topic due to
> > > > >>> > > kafka.common.LeaderNotAvailableException}] for topic
> > > > >>> > [my-replicated-topic]:
> > > > >>> > > class kafka.common.LeaderNotAvailableException
> > > > >>> > > (kafka.producer.BrokerPartitionInfo)
> > > > >>> > > [2013-10-11 00:30:26,755] ERROR Failed to collate messages by
> > > > topic,
> > > > >>> > > partition due to: Failed to fetch topic metadata for topic:
> > > > >>> > > my-replicated-topic
> (kafka.producer.async.DefaultEventHandler
Re: Kafka server error
Yes, everytime, I send message, it throws that error. 2013/10/11 Jun Rao > Is that reproducible? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky wrote: > > > Yes, it just says "INFO Reconnect due to socket error" > > But why and how come it comes? my zookeeper and storm have no any problem > > to collaborate each other. > > > > > > 2013/10/11 Jun Rao > > > > > The log you posted for the second broker didn't say why it crashed. Is > > that > > > all you got? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky > > wrote: > > > > > > > *Hi, Guys,* > > > > *I am currently running into the kafka server issue. * > > > > *I have a 5 nodes cluster and zookeeper running without any problem. > > > when I > > > > manually boot each node by using* "*JMX_PORT=9997 > > > bin/kafka-server-start.sh > > > > config/server-x.properties &*" command. > > > > > > > > *The scenario is:* > > > > *Then, first node, it can be booted.* > > > > *Once I boot the second node, it is crashed, the error is below:* > > > > > > > > [2013-10-11 04:02:17,200] INFO [Replica Manager on Broker 0]: > Handling > > > > LeaderAndIsr request > > > > > > > > > > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > > > -> > > > > > > > > > > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > > > (kafka.server.ReplicaManager) > > > > [2013-10-11 04:02:17,204] WARN No previously checkpointed > highwatermark > > > > value found for topic test-kafka partition 0. Returning 0 as the > > > > highwatermark (kafka.server.HighwaterMarkCheckpoint) > > > > [2013-10-11 04:02:17,205] INFO [ReplicaFetcherManager on broker 0] > > > Removing > > > > fetcher for partition [test-kafka,0] > > (kafka.server.ReplicaFetcherManager) > > > > [2013-10-11 04:02:17,214] INFO [Kafka Log on Broker 0], Truncated log > > > > segment /tmp/kafka-logs/test-kafka-0/.log to > target > > > > offset 0 (kafka.log.Log) > > > > [2013-10-11 04:02:17,235] INFO [ReplicaFetcherManager on broker 0] > > Adding > > > > fetcher for partition [test-kafka,0], initOffset 0 to broker 1 with > > > > fetcherId 0 (kafka.server.ReplicaFetcherManager) > > > > [2013-10-11 04:02:17,236] INFO [Replica Manager on Broker 0]: Handled > > > > leader and isr request > > > > > > > > > > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > > > -> > > > > > > > > > > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > > > (kafka.server.ReplicaManager) > > > > [2013-10-11 04:02:17,240] INFO [ReplicaFetcherThread-0-1], Starting > > > > (kafka.server.ReplicaFetcherThread) > > > > [2013-10-11 04:02:17,266] INFO [Replica Manager on Broker 0]: > Handling > > > > LeaderAndIsr request > > > > > > > > > > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > > > -> > > > > > > > > > > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:91,ControllerEpoch:30416),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > > > (kafka.server.ReplicaManager) > > > > [2013-10-11 04:02:17,267] INFO [ReplicaFetcherManager on broker 0] > > > Removing > > > > fetcher for partition [test-kafka,0] > > (kafka.server.ReplicaFetcherManager) > > > > [2013-10-11 04:02:17,268] INFO [Kafka Log on Broker 0], Truncated log > > > > segment /tmp/kafka-logs/test-kafka-0/.log to > target > > > > offset 0 (kafka.log.Log) > > > > [2013-10-11 04:02:17,268] INFO [ReplicaFetcherManager on br
Re: Kafka server error
Sorry, even I don't send message, it shows up that message about every 1 minute 2013/10/11 Jiang Jacky > Yes, everytime, I send message, it throws that error. > > > 2013/10/11 Jun Rao > >> Is that reproducible? >> >> Thanks, >> >> Jun >> >> >> On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky wrote: >> >> > Yes, it just says "INFO Reconnect due to socket error" >> > But why and how come it comes? my zookeeper and storm have no any >> problem >> > to collaborate each other. >> > >> > >> > 2013/10/11 Jun Rao >> > >> > > The log you posted for the second broker didn't say why it crashed. Is >> > that >> > > all you got? >> > > >> > > Thanks, >> > > >> > > Jun >> > > >> > > >> > > On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky >> > wrote: >> > > >> > > > *Hi, Guys,* >> > > > *I am currently running into the kafka server issue. * >> > > > *I have a 5 nodes cluster and zookeeper running without any problem. >> > > when I >> > > > manually boot each node by using* "*JMX_PORT=9997 >> > > bin/kafka-server-start.sh >> > > > config/server-x.properties &*" command. >> > > > >> > > > *The scenario is:* >> > > > *Then, first node, it can be booted.* >> > > > *Once I boot the second node, it is crashed, the error is below:* >> > > > >> > > > [2013-10-11 04:02:17,200] INFO [Replica Manager on Broker 0]: >> Handling >> > > > LeaderAndIsr request >> > > > >> > > > >> > > >> > >> Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) >> > > > -> >> > > > >> > > > >> > > >> > >> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 >> > > > (kafka.server.ReplicaManager) >> > > > [2013-10-11 04:02:17,204] WARN No previously checkpointed >> highwatermark >> > > > value found for topic test-kafka partition 0. Returning 0 as the >> > > > highwatermark (kafka.server.HighwaterMarkCheckpoint) >> > > > [2013-10-11 04:02:17,205] INFO [ReplicaFetcherManager on broker 0] >> > > Removing >> > > > fetcher for partition [test-kafka,0] >> > (kafka.server.ReplicaFetcherManager) >> > > > [2013-10-11 04:02:17,214] INFO [Kafka Log on Broker 0], Truncated >> log >> > > > segment /tmp/kafka-logs/test-kafka-0/.log to >> target >> > > > offset 0 (kafka.log.Log) >> > > > [2013-10-11 04:02:17,235] INFO [ReplicaFetcherManager on broker 0] >> > Adding >> > > > fetcher for partition [test-kafka,0], initOffset 0 to broker 1 with >> > > > fetcherId 0 (kafka.server.ReplicaFetcherManager) >> > > > [2013-10-11 04:02:17,236] INFO [Replica Manager on Broker 0]: >> Handled >> > > > leader and isr request >> > > > >> > > > >> > > >> > >> Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) >> > > > -> >> > > > >> > > > >> > > >> > >> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 >> > > > (kafka.server.ReplicaManager) >> > > > [2013-10-11 04:02:17,240] INFO [ReplicaFetcherThread-0-1], Starting >> > > > (kafka.server.ReplicaFetcherThread) >> > > > [2013-10-11 04:02:17,266] INFO [Replica Manager on Broker 0]: >> Handling >> > > > LeaderAndIsr request >> > > > >> > > > >> > > >> > >> Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) >> > > > -> >> > > > >> > > > >> > > >> > >> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:91,ControllerEpoch:30416),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 >>
Re: Kafka server error
no, it is not reproducible. Thank you. 2013/10/11 Jun Rao > I meant the broker crash. Is that reproducible? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 10:10 PM, Jiang Jacky wrote: > > > Yes, everytime, I send message, it throws that error. > > > > > > 2013/10/11 Jun Rao > > > > > Is that reproducible? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky > > wrote: > > > > > > > Yes, it just says "INFO Reconnect due to socket error" > > > > But why and how come it comes? my zookeeper and storm have no any > > problem > > > > to collaborate each other. > > > > > > > > > > > > 2013/10/11 Jun Rao > > > > > > > > > The log you posted for the second broker didn't say why it crashed. > > Is > > > > that > > > > > all you got? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > > > > > > On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky > > > > wrote: > > > > > > > > > > > *Hi, Guys,* > > > > > > *I am currently running into the kafka server issue. * > > > > > > *I have a 5 nodes cluster and zookeeper running without any > > problem. > > > > > when I > > > > > > manually boot each node by using* "*JMX_PORT=9997 > > > > > bin/kafka-server-start.sh > > > > > > config/server-x.properties &*" command. > > > > > > > > > > > > *The scenario is:* > > > > > > *Then, first node, it can be booted.* > > > > > > *Once I boot the second node, it is crashed, the error is below:* > > > > > > > > > > > > [2013-10-11 04:02:17,200] INFO [Replica Manager on Broker 0]: > > > Handling > > > > > > LeaderAndIsr request > > > > > > > > > > > > > > > > > > > > > > > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > > > > > -> > > > > > > > > > > > > > > > > > > > > > > > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > > > > > (kafka.server.ReplicaManager) > > > > > > [2013-10-11 04:02:17,204] WARN No previously checkpointed > > > highwatermark > > > > > > value found for topic test-kafka partition 0. Returning 0 as the > > > > > > highwatermark (kafka.server.HighwaterMarkCheckpoint) > > > > > > [2013-10-11 04:02:17,205] INFO [ReplicaFetcherManager on broker > 0] > > > > > Removing > > > > > > fetcher for partition [test-kafka,0] > > > > (kafka.server.ReplicaFetcherManager) > > > > > > [2013-10-11 04:02:17,214] INFO [Kafka Log on Broker 0], Truncated > > log > > > > > > segment /tmp/kafka-logs/test-kafka-0/.log to > > > target > > > > > > offset 0 (kafka.log.Log) > > > > > > [2013-10-11 04:02:17,235] INFO [ReplicaFetcherManager on broker > 0] > > > > Adding > > > > > > fetcher for partition [test-kafka,0], initOffset 0 to broker 1 > with > > > > > > fetcherId 0 (kafka.server.ReplicaFetcherManager) > > > > > > [2013-10-11 04:02:17,236] INFO [Replica Manager on Broker 0]: > > Handled > > > > > > leader and isr request > > > > > > > > > > > > > > > > > > > > > > > > > > > Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) > > > > > > -> > > > > > > > > > > > > > > > > > > > > > > > > > > > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:90,ControllerEpoch:30411),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:localhost,port:9092 > > > > > > (kafka.server.ReplicaManager) > > > > > > [2013-10-11 04:02:17,240] INFO [
Re: Kafka server error
Hi, Since I changed the port to 9093, it seems works, I can produce message, and another node can consume that. But there is still exception [2013-10-11 05:32:42,706] ERROR [KafkaApi-2] Error while fetching metadata for partition [my-replicated-topic,0] (kafka.server.KafkaApis) kafka.common.ReplicaNotAvailableException at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:471) at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456) at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.immutable.Set$Set1.foreach(Set.scala:81) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.Set$Set1.map(Set.scala:68) at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:452) at kafka.server.KafkaApis.handle(KafkaApis.scala:69) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) at java.lang.Thread.run(Thread.java:724) [2013-10-11 05:32:42,710] INFO Closing socket connection to /10.0.0.42. (kafka.network.Processor) 2013/10/11 Jiang Jacky > no, it is not reproducible. Thank you. > > > 2013/10/11 Jun Rao > >> I meant the broker crash. Is that reproducible? >> >> Thanks, >> >> Jun >> >> >> On Thu, Oct 10, 2013 at 10:10 PM, Jiang Jacky >> wrote: >> >> > Yes, everytime, I send message, it throws that error. >> > >> > >> > 2013/10/11 Jun Rao >> > >> > > Is that reproducible? >> > > >> > > Thanks, >> > > >> > > Jun >> > > >> > > >> > > On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky >> > wrote: >> > > >> > > > Yes, it just says "INFO Reconnect due to socket error" >> > > > But why and how come it comes? my zookeeper and storm have no any >> > problem >> > > > to collaborate each other. >> > > > >> > > > >> > > > 2013/10/11 Jun Rao >> > > > >> > > > > The log you posted for the second broker didn't say why it >> crashed. >> > Is >> > > > that >> > > > > all you got? >> > > > > >> > > > > Thanks, >> > > > > >> > > > > Jun >> > > > > >> > > > > >> > > > > On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky > > >> > > > wrote: >> > > > > >> > > > > > *Hi, Guys,* >> > > > > > *I am currently running into the kafka server issue. * >> > > > > > *I have a 5 nodes cluster and zookeeper running without any >> > problem. >> > > > > when I >> > > > > > manually boot each node by using* "*JMX_PORT=9997 >> > > > > bin/kafka-server-start.sh >> > > > > > config/server-x.properties &*" command. >> > > > > > >> > > > > > *The scenario is:* >> > > > > > *Then, first node, it can be booted.* >> > > > > > *Once I boot the second node, it is crashed, the error is >> below:* >> > > > > > >> > > > > > [2013-10-11 04:02:17,200] INFO [Replica Manager on Broker 0]: >> > > Handling >> > > > > > LeaderAndIsr request >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:30416;CorrelationId:5;ClientId:id_0-host_null-port_9092;PartitionState:(test-kafka,0) >> > > > > > -> >> > > > > > >> > > > > > >> > > > > >> > > > >>
Re: Kafka server error
Ok, finally, I find the problem, in the /etc/hosts, I set the hostname as 127.0.0.1 ip-10-0-0-108 It seems cannot use its internal hostname, and caused the connection to close. Thanks guys, if I encounter any further issue, I will let you know. 2013/10/11 Jiang Jacky > Hi, Since I changed the port to 9093, it seems works, I can produce > message, and another node can consume that. > But there is still exception > > [2013-10-11 05:32:42,706] ERROR [KafkaApi-2] Error while fetching metadata > for partition [my-replicated-topic,0] (kafka.server.KafkaApis) > kafka.common.ReplicaNotAvailableException > at > kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:471) > at > kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) > at scala.collection.immutable.List.foreach(List.scala:45) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.immutable.List.map(List.scala:45) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456) > at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.immutable.Set$Set1.map(Set.scala:68) > at > kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:452) > at kafka.server.KafkaApis.handle(KafkaApis.scala:69) > at > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) > at java.lang.Thread.run(Thread.java:724) > [2013-10-11 05:32:42,710] INFO Closing socket connection to /10.0.0.42. > (kafka.network.Processor) > > > 2013/10/11 Jiang Jacky > >> no, it is not reproducible. Thank you. >> >> >> 2013/10/11 Jun Rao >> >>> I meant the broker crash. Is that reproducible? >>> >>> Thanks, >>> >>> Jun >>> >>> >>> On Thu, Oct 10, 2013 at 10:10 PM, Jiang Jacky >>> wrote: >>> >>> > Yes, everytime, I send message, it throws that error. >>> > >>> > >>> > 2013/10/11 Jun Rao >>> > >>> > > Is that reproducible? >>> > > >>> > > Thanks, >>> > > >>> > > Jun >>> > > >>> > > >>> > > On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky >>> > wrote: >>> > > >>> > > > Yes, it just says "INFO Reconnect due to socket error" >>> > > > But why and how come it comes? my zookeeper and storm have no any >>> > problem >>> > > > to collaborate each other. >>> > > > >>> > > > >>> > > > 2013/10/11 Jun Rao >>> > > > >>> > > > > The log you posted for the second broker didn't say why it >>> crashed. >>> > Is >>> > > > that >>> > > > > all you got? >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > Jun >>> > > > > >>> > > > > >>> > > > > On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky < >>> jiang0...@gmail.com> >>> > > > wrote: >>> > > > > >>> > > > > > *Hi, Guys,* >>> > > > > > *I am currently running into the kafka server issue. * >>> > > > > > *I have a 5 nodes cluster and zookeeper running without any >>> > problem. >>> > > > > when I >>> > > > > > manually boot each node by using* "*JMX_PORT=9997 >>> > > > > bin/kafka-server-start.sh >>> > > > > > config/server-x.properties &*" command. >>> > > > > > >>> > > > > > *The scenario is:* >>> > > > > > *Then, first node, it can be booted.* >>> > > > > > *On
Robust of cassandra
Hi, Everyone For now, I have a kafka cluster, and there is only 1 partition in each node. i just also produce lots of messages to 1 topic only. Then, I have a problem, I used it for a while, the topic is crashed, the error message is "NoAvailableLeader", but it is good for other topics that I do not often use. So I have to cleanup the zookeeper cache, and restart it again, then that topic comes back normal. Can someone tell me how to fix that? I don't want to clean the zookeeper sometimes in order to make the topic available. It seems not stable. Thanks
Re: Robust of cassandra
Hi, Neha Thanks for replying, I am currently using the 0.8 version. I did not set any replication factor, it keeps the default from package. When I check the leader for that topic, it is none. It is weird, There is no problem for any other topics except that topic that I am using. For now, I cannot produce the bug, but it is produced in random time, can you tell me what setting shall I set to prevent from this? Thanks 2013/10/29 Neha Narkhede > What is the replication factor and have you checked if there are leader > elections around the time you see the NoAvailableLeader exception? It is > important to figure out the root cause of NoAvailableLeader, but it is > transient and should fix itself. Are you using 0.8 HEAD ? > > > On Tue, Oct 29, 2013 at 5:57 PM, Jiang Jacky wrote: > > > Hi, Everyone > > For now, I have a kafka cluster, and there is only 1 partition in each > > node. i just also produce lots of messages to 1 topic only. > > Then, I have a problem, I used it for a while, the topic is crashed, the > > error message is "NoAvailableLeader", but it is good for other topics > that > > I do not often use. So I have to cleanup the zookeeper cache, and restart > > it again, then that topic comes back normal. > > Can someone tell me how to fix that? I don't want to clean the zookeeper > > sometimes in order to make the topic available. It seems not stable. > > Thanks > > >
Re: Robust of cassandra
I am sorry Neha, so late to reply you. I will try that and let you know anyway. Thanks Neha, 2013/10/31 Neha Narkhede > The default replication factor is 1, so even if you have one broker > failure, you can run into a situation with no leader for some partitions. > You probably want to use a higher replication factor. > > Thanks, > Neha > > > On Thu, Oct 31, 2013 at 12:07 PM, Jiang Jacky wrote: > > > Hi, Neha > > Thanks for replying, I am currently using the 0.8 version. > > I did not set any replication factor, it keeps the default from package. > > When I check the leader for that topic, it is none. > > It is weird, There is no problem for any other topics except that topic > > that I am using. > > For now, I cannot produce the bug, but it is produced in random time, can > > you tell me what setting shall I set to prevent from this? > > Thanks > > > > > > 2013/10/29 Neha Narkhede > > > > > What is the replication factor and have you checked if there are leader > > > elections around the time you see the NoAvailableLeader exception? It > is > > > important to figure out the root cause of NoAvailableLeader, but it is > > > transient and should fix itself. Are you using 0.8 HEAD ? > > > > > > > > > On Tue, Oct 29, 2013 at 5:57 PM, Jiang Jacky > > wrote: > > > > > > > Hi, Everyone > > > > For now, I have a kafka cluster, and there is only 1 partition in > each > > > > node. i just also produce lots of messages to 1 topic only. > > > > Then, I have a problem, I used it for a while, the topic is crashed, > > the > > > > error message is "NoAvailableLeader", but it is good for other topics > > > that > > > > I do not often use. So I have to cleanup the zookeeper cache, and > > restart > > > > it again, then that topic comes back normal. > > > > Can someone tell me how to fix that? I don't want to clean the > > zookeeper > > > > sometimes in order to make the topic available. It seems not stable. > > > > Thanks > > > > > > > > > >
coding hangs there
Hi, I just installed the kafka 0.9, and my code stuck on the following command for producer: it is stuck on producer.flush(); for consume, it is stuck on ConsumerRecords records = consumer.poll(200); I did directly downloaded the code from github, and tried some other resources for 0.9, nothing is working. https://github.com/mapr-demos/kafka-sample-programs Have configured all the parameter from the *.props files. Thanks Jacky
