state of replication in 0.8

2013-01-19 Thread Stan Rosenberg
Hi, Could someone please comment on the state of replication in 0.8, i.e., what's implemented and what remains to be implemented? On a first blush I see similarity between what's in the high-level documentation and Jun's paper, "Using Paxos to Build a Scalable, Consistent, and Highly Available Da

Re: resilient producer

2013-01-15 Thread Stan Rosenberg
On Tue, Jan 15, 2013 at 3:12 PM, Corbin Hoenes wrote: > +1 how about posting yours to GitHub? > Sounds like a good contrib project. > There is nothing to post at the moment as we're currently in the requirements gathering phase :) Potentially, we might have a contrib project along the lines of

Re: resilient producer

2013-01-15 Thread Stan Rosenberg
Jay, Thanks for your insight! More comments are below. On Tue, Jan 15, 2013 at 3:18 PM, Jay Kreps wrote: > I can't speak for all users, but at LinkedIn we don't do this. We just run > Kafka as a high-availability system (i.e. something not allowed to be > down). These kind of systems require

Re: Is this a good overview of kafka?

2013-01-14 Thread Stan Rosenberg
Hi Felix, Would you mind elaborating on what you said regarding the ordering guaranteed; inlined below. Thanks, stan On Mon, Jan 14, 2013 at 6:08 PM, Felix GV wrote: > > For example if you partitioned using a User ID field within the messages, > you would be > guaranteed that all messages per

Re: partitioning

2013-01-14 Thread Stan Rosenberg
On Fri, Jan 11, 2013 at 12:37 AM, Jun Rao wrote: > Our current partitioning strategy is to mod key by # of partitions, not # > brokers. For better balancing partitions over brokers, one simple strategy > is to over partition, i.e., you have a few times of more partitions than > brokers. That way,

partitioning

2013-01-10 Thread Stan Rosenberg
Hi, I apologize if this question has been addressed before. We are currently evaluating kafka for our high volume data ingestion infrastructure. I would like to understand why consistent hashing was not implemented given its inherent ability to dynamically balance the load across brokers. The cur