>> After performing some tests, I noticed that after removing >> SD_IOSTATS functionality from the kernel, the write >> performances of Lustre was 4 times faster. On a Lustre 1.8.7 >> client, I untared a tarfile of a directory containing about >> 225,000 files (6.4GB).
> That is about 28kB/file, so it is causing a LOT of IO requests > to the underlying disks. Well, at least one per file, but I more suspicious of the MDS when there is a lot of metadata traffic, and here there is file creation and file closure (with update of the file size) every 28KiB. The size of each write IO is small, so the size of each RPC, but that is reflected in the low overall transfer rate. [ ... ] >> With both servers, it took about 4 minutes to untar the >> tarfile when running the kernel without SD_IOSTATS compared >> to about 16 minutes when running the kernel with SD_IOSTATS. >> Is that normal that SD_IOSTATS has a so big impact on Lustre >> write performances? > The sd_iostats code uses spin_lock_irqsave() to ensure > consistent data structure access in interrupt context, so > possibly this is causing the performance impact. The impact > might be more severe if there are many cores on the server. It would be really unfortunate that it were severe to the point that it takes 3 times as long to do that as to do 28KiB of IO. Note the transfer rate: 6.4GB in 4 minutes is around 25-26MiB/s, while in 16 minutes it reduces to around 6-7MiB/s. Theese are not so good figures in absolute terms, and more interestingly imply around ~1ms per file (~1,000 files/s) and written per second) vs. 4ms per file (~250 files/s), with an extra 3ms per file (allegedly) with 'SD_IOSTATS'. Considering the lack of obviously useful information in the original post, I am skeptical that 'SD_IOSTATS' directly accounts for 3ms on top of 1ms of IO per file, that's why I suspect at most some collateral effect. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
