Re: NFS and/or local filesystem consumer?

2014-05-17 Thread Jun Rao
Ken, We don't have sth like that now. It shouldn't be too hard to write one though. You probably need some kind of time-based partitioning to different files. Thanks, Jun On Fri, May 16, 2014 at 4:40 PM, Carlile, Ken wrote: > Hi Jun, > > I was wondering if there was something out there alread

Re: New consumer APIs

2014-05-17 Thread Jun Rao
Eric, With the new proposal, it seems what you want can be achieved by just instantiating N Consumer instances. Then, you can wrap each in a thread and call poll() in it. Will that work for you? Thanks Jun On Fri, May 16, 2014 at 4:05 PM, Eric Sammer wrote: > Neha: > > Here's the basic pseudo

Re: ISR not updating

2014-05-17 Thread Jun Rao
Do you see constant ISR shrinking/expansion of those two partitions in the leader broker's log ? Thanks, Jun On Fri, May 16, 2014 at 4:25 PM, Paul Mackles wrote: > Hi - We are running kafka_2.8.0-0.8.0-beta1 (we are a little behind in > upgrading). > > From what I can tell, connectivity to ZK

Re: who is using kafka to stare large messages?

2014-05-17 Thread Jun Rao
You just need to make sure that fetch size is larger than max.message.size set in the brokers. Thanks, Jun On Thu, May 15, 2014 at 3:55 PM, MB JA wrote: > Hi. > > I´m using Kafka, Can you help me please? :) > > I´m have the problem to read the next message available when is larger > than the

Re: ISR not updating

2014-05-17 Thread Shone Sadler
Hi Jun, I work with Paul and am monitoring the cluster as well. The status has not changed. When we execute kafka-list-topic we are seeing the following (showing one of two partitions having the problem) topic: t1 partition: 33 leader: 1 replicas: 1,2,3 isr: 1 when inspecting the logs of lead

Re: ISR not updating

2014-05-17 Thread Paul Mackles
Today we did a rolling restart of ZK. We also restarted the kafka controller and ISRs are still not being updated in ZK. Again, the cluster seems fine and the replicas in question do appear to be getting updated. I am guessing there must be some bad state persisted in ZK. On 5/17/14 7:50 PM, "Shon