In fact there are many more than 4000 open files. Many of our brokers run with 28,000+ open files (regular file handles, not network connections). In our case, we're beefing up the disk performance as much as we can by running in a RAID-10 configuration with 14 disks.
-Todd On Tue, Oct 21, 2014 at 7:58 PM, Xiaobin She <xiaobin...@gmail.com> wrote: > Todd, > > Actually I'm wondering how kafka handle so much partition, with one > partition there is at least one file on disk, and with 4000 partition, > there will be at least 4000 files. > > When all these partitions have write request, how did Kafka make the write > operation on the disk to be sequential (which is emphasized in the design > document of Kafka) and make sure the disk access is effective? > > Thank you for your reply. > > xiaobinshe > > > > 2014-10-22 5:10 GMT+08:00 Todd Palino <tpal...@gmail.com>: > > > As far as the number of partitions a single broker can handle, we've set > > our cap at 4000 partitions (including replicas). Above that we've seen > some > > performance and stability issues. > > > > -Todd > > > > On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She <xiaobin...@gmail.com> > > wrote: > > > > > hello, everyone > > > > > > I'm new to kafka, I'm wondering what's the max num of partition can one > > > siggle machine handle in Kafka? > > > > > > Is there an sugeest num? > > > > > > Thanks. > > > > > > xiaobinshe > > > > > >
