Hmm, I haven't read the design doc lately, but I'm surprised that there's even a discussion of sequential disk access. I suppose for small subsets of the writes you can write larger blocks of sequential data, but that's about the extent of it. Maybe one of the developers can speak more to that aspect.
As far as the number of files goes, it really doesn't matter that much whether you have a few or a lot. Once you have more than one, the disk access is random, so the performance is more like a cliff than a gentle slope. As I said, we've found issues once we go above 4000 partitions, and that's probably a combination of what the software can handle and the number of open files. -Todd On Thu, Oct 23, 2014 at 11:19 PM, Xiaobin She <xiaobin...@gmail.com> wrote: > Todd, > > Thank you very much for your reply. My understanding of RAID 10 is wrong. > > I understand that one can not get absolute sequential disk access even on > one single disk, the reason I'm interested with this question is that the > design document of Kafka emphasize that Kafka make advantage of the > sequential disk acceess to improve the disk performance, and I can' t > understand how to achive this with thounds of open files. > > I thought that compare to one or fewer files, thounds of open files will > make the disk access much more random, and make the disk performance much > more weak. > > You mentioned that to increase overall IO cpapcity, one will have to use > multiple spindles with sufficiently fast disk speed, but will it be more > effective for the disk with fewer files? Or does the num of files is not an > important factor for the entire performance of Kafka? > > Thanks again. > > xiaobinshe > > > > 2014-10-23 22:01 GMT+08:00 Todd Palino <tpal...@gmail.com>: > > > Your understanding of RAID 10 is slightly off. Because it is a > combination > > of striping and mirroring, trying to say that there are 4000 open files > per > > pair of disks is not accurate. The disk, as far as the system is > concerned, > > is the entire RAID. Files are striped across all mirrors, so any open > file > > will cross all 7 mirror sets. > > > > Even if you were to operate on a single disk, you're never going to be > able > > to ensure sequential disk access with Kafka. Even if you have a single > > partition on a disk, there will be multiple log files for that partition > > and you will have to seek to read older data. What you have to do is use > > multiple spindles, with sufficiently fast disk speeds, to increase your > > overall IO capacity. You can also tune to get a little more. For example, > > we use a 120 second commit on that mount point to reduce the frequency of > > flushing to disk. > > > > -Todd > > > > > > On Wed, Oct 22, 2014 at 10:09 PM, Xiaobin She <xiaobin...@gmail.com> > > wrote: > > > > > Todd, > > > > > > Thank you for the information. > > > > > > With 28,000+ files and 14 disks, that makes there are averagely about > > 4000 > > > open files on two disk ( which is treated as one single disk) , am I > > right? > > > > > > How do you manage to make the all the write operation to thest 4000 > open > > > files be sequential to the disk? > > > > > > As far as I know, write operation to different files on the same disk > > will > > > cause random write, which is not good for performance. > > > > > > xiaobinshe > > > > > > > > > > > > > > > 2014-10-23 1:00 GMT+08:00 Todd Palino <tpal...@gmail.com>: > > > > > > > In fact there are many more than 4000 open files. Many of our brokers > > run > > > > with 28,000+ open files (regular file handles, not network > > connections). > > > In > > > > our case, we're beefing up the disk performance as much as we can by > > > > running in a RAID-10 configuration with 14 disks. > > > > > > > > -Todd > > > > > > > > On Tue, Oct 21, 2014 at 7:58 PM, Xiaobin She <xiaobin...@gmail.com> > > > wrote: > > > > > > > > > Todd, > > > > > > > > > > Actually I'm wondering how kafka handle so much partition, with one > > > > > partition there is at least one file on disk, and with 4000 > > partition, > > > > > there will be at least 4000 files. > > > > > > > > > > When all these partitions have write request, how did Kafka make > the > > > > write > > > > > operation on the disk to be sequential (which is emphasized in the > > > design > > > > > document of Kafka) and make sure the disk access is effective? > > > > > > > > > > Thank you for your reply. > > > > > > > > > > xiaobinshe > > > > > > > > > > > > > > > > > > > > 2014-10-22 5:10 GMT+08:00 Todd Palino <tpal...@gmail.com>: > > > > > > > > > > > As far as the number of partitions a single broker can handle, > > we've > > > > set > > > > > > our cap at 4000 partitions (including replicas). Above that we've > > > seen > > > > > some > > > > > > performance and stability issues. > > > > > > > > > > > > -Todd > > > > > > > > > > > > On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She < > > xiaobin...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > hello, everyone > > > > > > > > > > > > > > I'm new to kafka, I'm wondering what's the max num of partition > > can > > > > one > > > > > > > siggle machine handle in Kafka? > > > > > > > > > > > > > > Is there an sugeest num? > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > xiaobinshe > > > > > > > > > > > > > > > > > > > > > > > > > > > >
