Re: Question about recovering from outage

I'm running Kafka 0.8. / Jonas 2014-05-18 23:45 GMT+02:00 Jonas Bergström : > Hi all, and thanks for a fantastic product. > > The other day our kafka node in our test environment went down due to full > disc. I reconfigured kafka to save fewer messages, and restarted the node. > It is a single

Re: ISR not updating

The value of under replicated partitions is 0 across the cluster. Thanks, Shone On Mon, May 19, 2014 at 12:23 AM, Jun Rao wrote: > What's the value of under replicated partitions JMX in each broker? > > Thanks, > > Jun > > > On Sat, May 17, 2014 at 6:16 PM, Paul Mackles wrote: > > > Today we

RE: starting of at a small scale, single ec2 instance with 7.5 GB RAM with kafka

Hi, I like how kafka operates, but I'm wondering if it is possible to run everything on a single ec2 instance with 7.5 GB RAM. So that would be zookeeper and a single kafka broker. I would have a separate server to consume from the broker. Producers would be from my web servers. I don't want

Make kafka storage engine pluggable and provide a HDFS plugin?

Hi there, I recently started to use Kafka for our data analysis pipeline and it works very well. One problem to us so far is expanding our cluster when we need more storage space. Kafka provides some scripts for helping do this but the process wasn't smooth. To make it work perfectly, seems Kafk

Re: ISR not updating

Ok. That does indicate the ISR should include all replicas. Which version of ZK server are you using? Could you check the ZK server log to see if there if the ISR is being updated? Thanks, Jun On Mon, May 19, 2014 at 1:30 AM, Shone Sadler wrote: > The value of under replicated partitions is 0

Re: Question about recovering from outage

Do you think you could upgrade to 0.8.1.1? It fixed a bunch of corner cases in the controller. Thanks, Jun On Mon, May 19, 2014 at 12:00 AM, Jonas Bergström wrote: > I'm running Kafka 0.8. > > / Jonas > > > 2014-05-18 23:45 GMT+02:00 Jonas Bergström : > > > Hi all, and thanks for a fantastic p

Kafka Migration Tool

Hi all, I'm trying to migrate from Kafka 0.7.2-2.9.2 (with Zookeeper 3.3.4 from Cloudera) to Kafka 0.8.1.1-2.9.2 (with official Zookeeper 3.4.5 ) - However hitting brick walls with a very mysterious problem: 6) at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217) Caused by: java.la

Re: What happens to Kafka when ZK lost its quorum?

Hi Weide/Connie, I have added this entry in the FAQ, please let me know if anything on the wiki is not clear to you. https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowareKafkabrokersdependonZookeeper ? Guozhang On Wed, May 14, 2014 at 8:15 AM, Neha Narkhede wrote: > Kafka requires

Cause and Recovery

Hi, I got this from the log: [exec] 03:50:06.216 [ProducerSendThread-] [1;31mERROR [0;39m [1;35mk.producer.async.ProducerSendThread [0;39m - Error in handling batch of 100 events [exec] kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. [exec] at kafka.pr

Re: Cause and Recovery

Hi Mingtao, 1. Do you see any error/warn log before this entry? 2. The producer will re-try on sending when the previous trial is not successful. In your case it has exhausted all the retries and hence messages are dropped on this floor and lost. Guozhang On Mon, May 19, 2014 at 8:32 AM, Mingta

Re: Question about recovering from outage

Ok, I'll upgrade. Is there a way to see the status of a node that is recovering, e.g. in zookeeper or via jmx? / Jonas 2014-05-19 16:49 GMT+02:00 Jun Rao : > Do you think you could upgrade to 0.8.1.1? It fixed a bunch of corner cases > in the controller. > > Thanks, > > Jun > > > On Mon, May 19

Consistent replication of an event stream into Kafka

Hello, We have a use case where we want to replicate an event stream which exists outside of kafka into a kafka topic (single partition). The event stream has sequence ids which always increase by 1. We want to preserve this ordering. The difficulty is that we want to be able to have the process

SocketServerStats not reporting bytes written or read

Hi all, I have an intermittent problem with the JMX SocketServer stats on my 0.7.2 Kafka cluster. I'm collecting the SocketServerStats with jmxstats and everything seems to be working fine except kafka.SocketServerStats:BytesWrittenPerSecond and kafka.SocketServerStats:BytesReadPerSecond are no

Re: Kafka Migration Tool

It seems that you may have set the zk connection string to one used by 0.8 Kafka brokers. Thanks, Jun On Mon, May 19, 2014 at 8:34 AM, Mo Firouz wrote: > Hi all, > > I'm trying to migrate from Kafka 0.7.2-2.9.2 (with Zookeeper 3.3.4 from > Cloudera) to Kafka 0.8.1.1-2.9.2 (with official Zooke

Re: Question about recovering from outage

In trunk, we have a JMX monitoring the states of each broker. One of the states is log recovery. Thanks, Jun On Mon, May 19, 2014 at 11:15 AM, Jonas Bergström wrote: > Ok, I'll upgrade. > Is there a way to see the status of a node that is recovering, e.g. in > zookeeper or via jmx? > > / Jonas

Re: SocketServerStats not reporting bytes written or read

Is the problem with the jmx beans themselves or jmxstats? Thanks, Jun On Mon, May 19, 2014 at 2:48 PM, Xuyen On wrote: > Hi all, > > I have an intermittent problem with the JMX SocketServer stats on my 0.7.2 > Kafka cluster. > I'm collecting the SocketServerStats with jmxstats and everything

Re: Consistent replication of an event stream into Kafka

Hello Bob, What you described is similar to the idempotent producer design that we are now discussing about: https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer Do you think this new feature will solve your case? Guozhang On Mon, May 19, 2014 at 2:40 PM, Bob Potter wrote: