Makes sense. Thanks :)
On Wed, Oct 22, 2014 at 11:10 AM, Jonathan Weeks <jonathanbwe...@gmail.com> wrote: > There are various costs when a broker fails, including broker leader election > for each partition, etc., as well as exposing possible issues for in-flight > messages, and client rebalancing etc. > > So even though replication provides partition redundancy, RAID 10 on each > broker is usually a good tradeoff to prevent the typical most common cause of > broker server failure (e.g. disk failure) as well, and overall smoother > operation. > > Best Regards, > > -Jonathan > > > On Oct 22, 2014, at 11:01 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> RAID-10? >> Interesting choice for a system where the data is already replicated >> between nodes. Is it to avoid the cost of large replication over the >> network? how large are these disks? >> >> On Wed, Oct 22, 2014 at 10:00 AM, Todd Palino <tpal...@gmail.com> wrote: >>> In fact there are many more than 4000 open files. Many of our brokers run >>> with 28,000+ open files (regular file handles, not network connections). In >>> our case, we're beefing up the disk performance as much as we can by >>> running in a RAID-10 configuration with 14 disks. >>> >>> -Todd >>> >>> On Tue, Oct 21, 2014 at 7:58 PM, Xiaobin She <xiaobin...@gmail.com> wrote: >>> >>>> Todd, >>>> >>>> Actually I'm wondering how kafka handle so much partition, with one >>>> partition there is at least one file on disk, and with 4000 partition, >>>> there will be at least 4000 files. >>>> >>>> When all these partitions have write request, how did Kafka make the write >>>> operation on the disk to be sequential (which is emphasized in the design >>>> document of Kafka) and make sure the disk access is effective? >>>> >>>> Thank you for your reply. >>>> >>>> xiaobinshe >>>> >>>> >>>> >>>> 2014-10-22 5:10 GMT+08:00 Todd Palino <tpal...@gmail.com>: >>>> >>>>> As far as the number of partitions a single broker can handle, we've set >>>>> our cap at 4000 partitions (including replicas). Above that we've seen >>>> some >>>>> performance and stability issues. >>>>> >>>>> -Todd >>>>> >>>>> On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She <xiaobin...@gmail.com> >>>>> wrote: >>>>> >>>>>> hello, everyone >>>>>> >>>>>> I'm new to kafka, I'm wondering what's the max num of partition can one >>>>>> siggle machine handle in Kafka? >>>>>> >>>>>> Is there an sugeest num? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> xiaobinshe >>>>>> >>>>> >>>> >
