Public bug reported: RAID-5/6 write performance utterly sucks. Initially we were on RAID-6, since RAID write performance was really really poor I converted it to RAID-5. This only took around 9 days...
After I pointed out write performance was bad on the LIO mailing list (export it as iSCSI volume through LIO) that the write performance was horribly bad they stated I should have a look at /sys/block/md?/queue/max_sectors_kb (and oc max_hw_sectors_kb). To my surprise both were set at 127, which is horrible performance wise as it's not a binary multiple. Since max_hw_sectors_kb has 127 as value as well I can't set it higher. All block devices used in the RAID set have their values set at 4096. I think the optimum value would be something like (D-P)*512/1024/2048/4096. Where D is the number of disks in the RAID and P is the number of parity disks (1 in case of RAID-5, 2 in case of RAID-6). Or at least something that's a binary multiple (which 127 is definitely not). Can't find much about it. Some guy reported his issues were gone once he upgraded the kernel to 3.3. Also saw this: http://www.spinics.net/lists/raid/msg38609.html In any case, I'm quite shocked it hasn't been noticed/fixed. I get like 20-30MiB/s write sequential sustained (when doing over 10GB writes with dd for example, the first ~8G or so go fast due to buffers, once they're full performance crumbles down to utter shit) on a 8 disk RAID set. Individual disks do nearly 100MiB/s sequential each (even when they're all under load at the same time), CPU's are hardly loaded at all so it's not a checksumming thing. 1) 12.04 LTS server 64-bit (upgraded from 8.04 -> 10.04 -> 11.?? (had issues with mvsas controllers - they're replaced (LSI) now, not that this version helped it :)) -> 12.04) 2) 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 17:49:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux 3) At least 200-300MiB/s write performance sustained (sequential oc). Note the at least :). 4) Crappy performance like 20-30MiB/s sustained sequential. ProblemType: Bug DistroRelease: Ubuntu 12.04 Package: linux-image-3.2.0-26-generic 3.2.0-26.41 ProcVersionSignature: Ubuntu 3.2.0-26.41-generic 3.2.19 Uname: Linux 3.2.0-26-generic x86_64 AlsaDevices: total 0 crw-rw---T 1 root audio 116, 1 Jul 8 14:29 seq crw-rw---T 1 root audio 116, 33 Jul 8 14:29 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.0.1-0ubuntu8 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found. Date: Tue Jul 31 10:30:29 2012 HibernationDevice: RESUME=UUID=3680d1bc-fe0c-4584-98b3-4cfc8bb50c60 InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427) IwConfig: lo no wireless extensions. eth1 no wireless extensions. eth0 no wireless extensions. Lsusb: Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd MachineType: Supermicro X8SIL PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-26-generic root=UUID=3615935e-453f-448c-a6ea-bd595a49da9c ro quiet RelatedPackageVersions: linux-restricted-modules-3.2.0-26-generic N/A linux-backports-modules-3.2.0-26-generic N/A linux-firmware 1.79 RfKill: Error: [Errno 2] No such file or directory SourcePackage: linux UpgradeStatus: Upgraded to precise on 2012-07-06 (24 days ago) dmi.bios.date: 02/25/2010 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1.0c dmi.board.asset.tag: To Be Filled By O.E.M. dmi.board.name: X8SIL dmi.board.vendor: Supermicro dmi.board.version: 0123456789 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 24 dmi.chassis.vendor: Supermicro dmi.chassis.version: 0123456789 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.0c:bd02/25/2010:svnSupermicro:pnX8SIL:pvr0123456789:rvnSupermicro:rnX8SIL:rvr0123456789:cvnSupermicro:ct24:cvr0123456789: dmi.product.name: X8SIL dmi.product.version: 0123456789 dmi.sys.vendor: Supermicro ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug precise -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1031260 Title: RAID write performance is horrible. max_sectors_kb is set to the odd value of 127 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1031260/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
