Here are the stats results with Bonnie++, in HTML format (see attachment) The first 4 tests are in VM. The 2 last are just test on 2 workstations with similar options. An entry level one and a high end one, both with SSDs. DRBD is of course not used there.
Test command : ------------------- bonnie++ -d ./scratchdir/ -c 2 -s 2G -n 64:4096:512 (Changed to -s 16G on my high end workstation ) Virtualization software and Hardware for the first 4 tests (localssd1, ssdvg1, localdir, satavg1) : ------------------------------------------------------------------------------------------------------------------------ - Virtulization : Proxmox VE 1.9 (KVM) + VirtIO network & disk drivers, 10GB Storage and 256MB Ram - Hardware : 2 HP Pavilion G5415be workstation with QuadCore Athlon X4 645 3.2GHz+ SSD Chronos SA3 MSK 240GB ( in SATA2 mode because of Mother board ) - Limited to 1GB of Ram with the mem=1024M boot option. - Additional GBits PCIe 1X network adapters with a cross cable dedicated to DRBD replication (when used). Test conditions (rest of Stack description ): localssd1 ======= - VM OS : Ubuntu 11.10 Server with default options (ext4) - VM Storage : LVM on (SSD PV of 40GB) ssdvg1 ===== - VM OS : Ubuntu 11.10 Server with default options (ext4) with 256MB Ram - VM Storage : LVM on DRBD Active-Active (Protocol C) (SSD PV of 200GB in each server) localdir ===== - VM OS : Ubuntu 11.10 Server with default options (ext4) with 256MB Ram - VM Storage : RAW Image on a local SATA disk (2TB/7200RPM) formated in EXT3 satavg1 ===== - VM OS : Ubuntu 11.10 Server with default options (ext4) with 256MB Ram - VM Storage : LVM on DRBD Active-Active (Protocol C) (SATA disk 2TB/7200RPM PV of 750GB in each server) - With Meta-Disk on a small partition of 128MB on the skinner ===== - Not a VM, just my workstation, a very light workstation with 2GB Ram, Dual Core ATOM 1.8GHz, and OCZ Vertex2 SSD. - Kubuntu 11.10 on ext4 westvleteren ========= - Not a VM, an other workstation, high end, with 8GB Ram, Quad Core intel Corei5, SSD 120GB SATA3 ( more than 500MB/s read/writes with 90000 IOPS ) Conclusions : ----------------- - on the satavg1 test, using meta-disk on SSD did not fix my problem. Maybe it's better than meta-disk internal, but clearly unusable. Machine unresponsive, host with load of 10 during the test in the VM, ... So unfortunately, SSD meta-disk did not save my 750GB partitions from being far too slow to be used. - on the ssdvg1, the performances, are really good. Strangely, it even competes to the results without DRBD (localssd1). The move to SSD for my VMs seems to fix my issue. My totally useless cluster on SATA Drives, becomes now really usable with SSD. As all my VMs can keep into 200GB storage. My performance problem is fixed, it's even really good. Regards, François I noticed I did not reply to the list. So, here is the copy : Hi Eduardo, When I had these horrible performances I was not in a good situations : - One 2GB disk SSD 7200RPM per node. - DRBD 8.3, Protocol C, Active-Active - 2 Network interfaces on a 100Mbits network switch. Now that I added the SSDs, I'll test again. I'm currently looking on how to test. As I'm not interested in testing my OS cache, I'll reduce the memory to 1GB or less (with the mem=1024M kernel boot option), Install an ubuntu VM with 512MRam and PV disk drivers, and run either bonnie++ or iozone in these 4 conditions : - Local SSD - DRBD SSD in Active-Active - Local SATA - DRBD Sata in Active-Active (but with the meta-disk on SSD). (Where I was getting 30IOPS with bonnie++ in my VM before I moved meta-disk to SSD) Regards, François Delpierre ----- Mail original ----- De: "Eduardo Diaz - Gmail" <[email protected]> À: "François Delpierre" <[email protected]> Cc: [email protected] Envoyé: Mardi 3 Janvier 2012 14:19:39 Objet: Re: [DRBD-user] drbd performance issue on SATA I check the Network link too, because some times the network are the real problem. Did you test the network interface?. Did you change the the protocol to A for example.. and make more test. In my experience the SSD disk has problems for broken. http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid-state-drive-scale.html What configuration are you creating? Active/active, Active/passive? What filesystem? regards! On Sun, Jan 1, 2012 at 6:04 PM, François Delpierre <[email protected]> wrote: Hi, I'm building a very low cost virtualization server for a Non Profit organization using 2 HP workstations (500€ each) (8GB Ram, 1.5TB, QuadCore AMD), Software Proxmox and DRBD for the data replication with dedicated gigabit & cross cable for the replication. After setting up the replication (Protocol C), I had catastrophic performances, ( dropping to 30 IOPS after 60s of random writes ). I also found a lot of reading, not fully explaining my terrific 30 IOPS on 7200RPM drives, but clearly pointing the necessity to have a battery backed up cache controller to sustain random writes generated by the meta-disk. As the hardware makrket did change quite a lot recently, especially towards SSD, I immediately realized that 240GB SSD are half price than a small raid controller, a battery for the cache, and 2 small disks with reasonable performances (10rpm). So, I choose SSD. I created 2 partitions on my SSD : - 128MB for external meta-disk of my slow SATA partition (750GB) - 240GB for a second DRBD resource, with internal meta-disk. My questions : - Is there some recommendations on a SSD setup ? - Why is the external meta-disk that few documented ? I spent more than 1 hour to find the syntax of the directive : meta-disk /dev/disk/by-id/scsi-SATA_MKNSSDCR240GB_MKN1146A0000032XXX-part1[0]; And it's not clear to my what is the [0] at the end ? Regards, François Delpierre _______________________________________________ drbd-user mailing list [email protected] http://lists.linbit.com/mailman/listinfo/drbd-userTitle: Bonnie++ Benchmark results
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | ||||||||||||||||||||||||
| Concurrency | Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Max Size | Min Size | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | ||||||
| localssd1 | 2 | 2G | 1230 | 96 | 58189 | 10 | 48222 | 11 | 4860 | 95 | 94024 | 16 | 3908 | 160 | 64 | 4096 | 512 | 14832 | 30 | 2445 | 24 | 48493 | 71 | 14796 | 27 | 1632 | 23 | 38912 | 64 |
| Latency | 6439us | 4175ms | 196ms | 7219us | 12001us | 500ms | Latency | 125ms | 160ms | 5625us | 123ms | 155ms | 227ms | ||||||||||||||||
| ssdvg1 | 2 | 2G | 1242 | 98 | 86313 | 10 | 50924 | 9 | 4871 | 95 | 100476 | 14 | 3525 | 140 | 64 | 4096 | 512 | 14350 | 28 | 2416 | 23 | 46369 | 68 | 14962 | 28 | 1637 | 23 | 38555 | 63 |
| Latency | 6474us | 238ms | 202ms | 8873us | 8403us | 314ms | Latency | 127ms | 173ms | 6087us | 122ms | 167ms | 194ms | ||||||||||||||||
| localdir | 2 | 2G | 1273 | 99 | 63908 | 6 | 20017 | 3 | 4162 | 82 | 60767 | 8 | 122.7 | 6 | 64 | 4096 | 512 | 8504 | 15 | 160 | 1 | 44076 | 69 | 9184 | 21 | 138 | 1 | 43661 | 75 |
| Latency | 6411us | 597ms | 485ms | 31939us | 80564us | 1097ms | Latency | 507ms | 822ms | 63905us | 470ms | 593ms | 14710us | ||||||||||||||||
| satavg1 | 2 | 2G | 1158 | 91 | 18852 | 3 | 6508 | 1 | 2107 | 41 | 11803 | 2 | 105.9 | 5 | 64 | 4096 | 512 | 3355 | 5 | 114 | 1 | 29084 | 42 | 3708 | 6 | 90 | 1 | 32682 | 53 |
| Latency | 6581us | 1409ms | 1227ms | 288ms | 198ms | 1460ms | Latency | 1123ms | 4182ms | 164ms | 1020ms | 2164ms | 34131us | ||||||||||||||||
| skinner | 2 | 2G | 158 | 99 | 230966 | 88 | 106531 | 42 | 626 | 99 | 260127 | 44 | 13160 | 562 | 64 | 4096 | 512 | 9440 | 88 | 35102 | 97 | 15392 | 92 | 10437 | 88 | 38199 | 99 | 15105 | 93 |
| Latency | 85011us | 127ms | 69985us | 32114us | 26642us | 15884us | Latency | 61986us | 4029us | 101ms | 101ms | 2100us | 102ms | ||||||||||||||||
| westvleteren | 2 | 16G | 1031 | 99 | 191447 | 11 | 100037 | 9 | 4045 | 87 | 291026 | 14 | 2290 | 89 | 64 | 4096 | 512 | 4913 | 22 | 9374 | 3 | 13011 | 88 | 5066 | 21 | 9304 | 3 | 12288 | 90 |
| Latency | 11543us | 422ms | 423ms | 13569us | 216ms | 180ms | Latency | 910ms | 61910us | 5023us | 805ms | 252ms | 19067us | ||||||||||||||||
_______________________________________________ drbd-user mailing list [email protected] http://lists.linbit.com/mailman/listinfo/drbd-user
