>>>>> "Conrad" == Conrad Wood <[email protected]> writes:
Conrad> On Sat, 2012-03-24 at 01:08 -0700, Anton Cohen wrote: >> On Fri, Mar 23, 2012 at 10:01 AM, Conrad Wood <[email protected]> >> wrote: >> on storage: >> dd if=/dev/zero of=volume bs=1M : ~ 1,600MByte/s >> cp file1 file2 : ~ 300MByte/s (both files on same volume) >> >> on server >> dd if=/dev/zero of=volume bs=1M : ~ 1,100MByte/s >> cp file1 file2 : ~ 30MByte/s (both files on same volume) >> >> It's not clear to me whether of=volume is writing to the volume or a >> file on a file system, i.e., is it of=/dev/vol or >> of=/mountpoint/file1? To be a fair test against 'cp' you should have >> 'dd' write to a file, though the performance should be fairly close. >> Also I can't tell if they are direct/synced writes or not, so I'll >> assume not (sorry for assuming). >> Conrad> quite true, sorry for the ambiquity on my side. of=volume Conrad> refers to a file on the volume. What filesystem are you using? I don't recall seeing that listed, but I've also already deleted the previous messages in the thread. How you mount the filesystem can have a huge impact, along with how you've layed out the filesystem, etc. What is your fstab entry for this filesystem? Conrad> Die direct option in the linux kernel isn't quite what I Conrad> wanted, because apparetntly it keeps only a single BIO Conrad> in-flight at any time. (Which matches my observation) That Conrad> slows things down a lot of course and isn't what 'cp' does. I Conrad> issue a 'sync' command afterwards and calculate the amount of Conrad> bytes written & read between starting "cp" or "dd" and the Conrad> "sync" being finished. >> Your 'dd' numbers look suspiciously high, like buffered writes to RAM. >> Whenever you want to test actual write performance with 'dd', not >> memory or dirty page performance, you need to do oflag=direct: >> dd if=/dev/zero of=/mnt/point/file oflag=direct bs=1M count=1000 Conrad> Actually it is a 24-disk raid array with pretty fast disks. I Conrad> do believe the numbers are correct for sequential(ish) Conrad> read/writes. (each disk delivers ~120MByte/s sequential, so Conrad> the raid should theoretically give 2.8GByte/s. (even though Conrad> the SAS bus cannot go quite as fast I believe) >> Unless you actually want to test writing dirty pages, and I have seen >> a kernel bug where flushing dirty pages was slow, but then you should >> also time a 'sync': >> time dd if=/dev/zero of=/mnt/point/file bs=1M count=1000 && time sync >> Conrad> Mostly I wonder if it is atall possible to get such speeds Conrad> over QDR. Are you in a position where you could perhaps run a Conrad> "cp" followed by "sync" on an infiniband attached storage Conrad> system? Absolutely, Infiniband is fast, and QDR should be insanely fast, and low low latency as well, which is it's primary metric. But can we go back and look at your system again? What OS, what hardware, what filesystem, are you using LVM? How is the array RAID setup, or is it not using RAID on the hardware end? Have you tried using 'fio' the Flexible IO Tester to run your tests instead? THanks, John _______________________________________________ Tech mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
