Re: Consumer Events Problem

2013-12-09 Thread Sanket Maru
For mine topic the offset isn't increasing which means the consumer has stopped. I wanted to get the count(#) of events that are still remaining to be processed. Is that possible ? On Mon, Dec 9, 2013 at 9:44 PM, Jun Rao wrote: > By default, each topic is kept on the broker for 7 days. Older d

Re: Migration Process from Kafka 0.7 to 0.8

2013-12-09 Thread Jun Rao
Yes, that's the wiki for migration from 0.7 to 0.8. In terms of the migration path, the process hasn't changed since the 0.8.0 beta release. Thanks, Jun On Mon, Dec 9, 2013 at 12:46 PM, Arnaud Lawson wrote: > Hi, I am working on a migration plan - moving from kafka 0.7 to kafka 0.8 > - and ca

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
That's good to know. Thanks for your help! 2013/12/9 Neha Narkhede > We will announce it on this mailing list. It is probably a month away from > a release. > > Thanks, > Neha > > > On Mon, Dec 9, 2013 at 12:02 PM, Maxime Nay wrote: > > > Ok, thanks for your help. When 0.8.1 will be production

Re: Presentation on our use of Kafka at Flurry

2013-12-09 Thread Neha Narkhede
That's great, thanks for sharing! On Mon, Dec 9, 2013 at 1:49 PM, Ian Friedman wrote: > Hello fellow kafka users, > > I just wanted to share a presentation one of my colleagues made at SK > Planet's Tech Planet conference. It's about how Flurry transitioned from a > map-reduce batching data pro

Presentation on our use of Kafka at Flurry

2013-12-09 Thread Ian Friedman
Hello fellow kafka users, I just wanted to share a presentation one of my colleagues made at SK Planet's Tech Planet conference. It's about how Flurry transitioned from a map-reduce batching data processing system to using Kafka to do continuous processing in our data ingestion pipeline. I thou

Re: Partition reassignment stucked

2013-12-09 Thread Neha Narkhede
We will announce it on this mailing list. It is probably a month away from a release. Thanks, Neha On Mon, Dec 9, 2013 at 12:02 PM, Maxime Nay wrote: > Ok, thanks for your help. When 0.8.1 will be production ready, will you > announce it somewhere (will you release it right away) ? > > Thanks,

Migration Process from Kafka 0.7 to 0.8

2013-12-09 Thread Arnaud Lawson
Hi, I am working on a migration plan - moving from kafka 0.7 to kafka 0.8 - and came across the webpage below that was last edited on April 26, 2013. Is this the most up-to-date information page regarding the migration process? And are there other useful webpages? https://cwiki.apache.org/conflue

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread Benjamin Black
You might look at Curator http://curator.apache.org/ On Mon, Dec 9, 2013 at 12:36 PM, S Ahmed wrote: > Say am I doing this, a scenerio that I just came up with that demonstrates > #2. > > Someone signs up on a website, and you have to: > > 1. create the user profile > 2. send email confirmation

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread S Ahmed
Say am I doing this, a scenerio that I just came up with that demonstrates #2. Someone signs up on a website, and you have to: 1. create the user profile 2. send email confirmation email 3. resize avatar Now once a person registers on a website, I write a message to Kafka. Now I have 3 differe

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread Philip O'Toole
We use Zookeeper, as is standard with Kafka. Our systems are idempotent, so we only store offsets when the message is fully processed. If this means we occasionally replay a message due to some corner-case, or simply a restart, it doesn't matter. Philip On Mon, Dec 9, 2013 at 12:28 PM, S Ahmed

RE: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread S Ahmed
I was hoping people could comment on how they handle the following scenerios: 1. Storing the last successfully processed messageId/Offset. Are people using mysql, redis, etc.? What are the tradeoffs here? 2. How do you handle recovering from an error while processesing a given event? There are

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
Ok, thanks for your help. When 0.8.1 will be production ready, will you announce it somewhere (will you release it right away) ? Thanks, Maxime 2013/12/9 Neha Narkhede > I wouldn't call 0.8.1 production ready just yet. We are still in the > process of deploying it at LinkedIn. Until it is rea

Re: Partition reassignment stucked

2013-12-09 Thread Neha Narkhede
I wouldn't call 0.8.1 production ready just yet. We are still in the process of deploying it at LinkedIn. Until it is ready, there isn't a good cluster expansion solution other than spinning up a new cluster. This is probably a little easier if you have a VIP in front of your kafka cluster. Thanks

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
Hi, We used the code checked in this branch a few hours before the official 0.8.0 final release : https://github.com/apache/kafka/tree/0.8 So hopefully it should be the exact same code as the official release. The controller logs are empty. In a previous exchange you advised us to not use trunk

Re: How do you keep track of offset in a partition

2013-12-09 Thread Jun Rao
Consumer offsets are typically checkpointed into ZK. To check # of unconsumed messages, you can either use the ConsumerOffsetChecker tool (see https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F) or use the maxlag jmx in the consumer application (see htt

Re: Consumer Events Problem

2013-12-09 Thread Jun Rao
By default, each topic is kept on the broker for 7 days. Older data, whether consumed or not, will be deleted. To check # of unconsumed messages, you can either use the ConsumerOffsetChecker tool (see https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F)

Re: How do you keep track of offset in a partition

2013-12-09 Thread Bhargav
Hey, "Keeping track of/what/has been consumed, is, surprisingly, one of the key performance points of a messaging system." -- Kafka documentation. How to know what has not been consumed yet? How to retrieve them? Regards, Bhargav

Re: How do you keep track of offset in a partition

2013-12-09 Thread Bhargav
Hi, How to know where to start consuming from, if we have already consumed a few messages. Some where in between latest and the earliest? How to identify the timestamp or offset from the api. Thanks, Bhargav --

Re: Consumer Events Problem

2013-12-09 Thread Philip O'Toole
OK, I am only familiar with 0.72. Philip On Mon, Dec 9, 2013 at 4:54 AM, Sanket Maru wrote: > I am using kafka 0.8.0 > > > On Mon, Dec 9, 2013 at 6:09 PM, Philip O'Toole wrote: > > > What version are you running? > > > > Philip > > > > > > On Mon, Dec 9, 2013 at 4:30 AM, Sanket Maru wrote: >

Re: Consumer Events Problem

2013-12-09 Thread Sanket Maru
I am using kafka 0.8.0 On Mon, Dec 9, 2013 at 6:09 PM, Philip O'Toole wrote: > What version are you running? > > Philip > > > On Mon, Dec 9, 2013 at 4:30 AM, Sanket Maru wrote: > > > I am working on a small project and discovered that our consumer hasn't > > been executed for over a month now.

Re: Consumer Events Problem

2013-12-09 Thread Philip O'Toole
What version are you running? Philip On Mon, Dec 9, 2013 at 4:30 AM, Sanket Maru wrote: > I am working on a small project and discovered that our consumer hasn't > been executed for over a month now. > > How can i check the unprocessed events ? From which date the events are > available and wh

Consumer Events Problem

2013-12-09 Thread Sanket Maru
I am working on a small project and discovered that our consumer hasn't been executed for over a month now. How can i check the unprocessed events ? From which date the events are available and what is the retention policy on the producer side ? Thanks, Sanket Maru