Maxime, First of all, in 0.8, you can choose to have a replication factor of 2. It just means that one can tolerate only one broker failure.
Second, our producer logic supports retries on failure. If a message can't be sent, on retry, our default partitioner will select another random partition to send the message to. So, with a replication factor of 1, assuming that you have enough partitions (partitions are typically spread over all brokers) and enough retries, the message is likely to be delivered. To improve this, another possibility is to create a new type of partitioner that will route messages randomly to only those partitions with a leader. However, this likely requires a new interface of the partitioner. In addition to number of partitions, the partitioner needs to know whether each partition is available or not. Thanks, Jun On Wed, Jan 9, 2013 at 8:43 AM, Maxime Brugidou <maxime.brugi...@gmail.com>wrote: > Thanks for your response. I think the work-around is not really acceptable > for me since it will consume 3x the resources (because replication of 3 is > the minimum acceptable) and it will still make the cluster less available > anyway (unless i have only 3 brokers). > > The thing is that 0.7 was making the cluster 100% available (for my use > case, accepting data loss) as long a single broker was alive. > > A way to handle this would be to: > 1. Have a lot of partitions per topic (more than the # of brokers) > 2. Have something that rebalances the partitions and make sure a broker has > a at least a partition for each topic (to make every topic "available") > 3. Have a setting in the consumer/producer that say "I don't care about > partitioning, just produce/consume wherever you can" > > This is probably not simple to implement, I'll add these ideas in the JIRA > and will pursue the discussion there. > > Maxime > > On Wed, Jan 9, 2013 at 5:18 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > As a work around in the meantime you can probably run with > > replication--although it sounds like you don't really need it, it > shouldn't > > hurt. > > >
