As to your results, it sounds almost too good to be true. As Bob has pointed out, h/w design targeted hundreds IOPS, and it was hard to believe it can scale 100x. Fantastic.
Regards, Andrey On Thu, Jun 10, 2010 at 6:06 PM, Robert Milkowski <mi...@task.gda.pl> wrote: > On 21/10/2009 03:54, Bob Friesenhahn wrote: > >> >> I would be interested to know how many IOPS an OS like Solaris is able to >> push through a single device interface. The normal driver stack is likely >> limited as to how many IOPS it can sustain for a given LUN since the driver >> stack is optimized for high latency devices like disk drives. If you are >> creating a driver stack, the design decisions you make when requests will be >> satisfied in about 12ms would be much different than if requests are >> satisfied in 50us. Limitations of existing software stacks are likely >> reasons why Sun is designing hardware with more device interfaces and more >> independent devices. >> > > > Open Solaris 2009.06, 1KB READ I/O: > > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t0d0p0& > # iostat -xnzCM 1|egrep "device|c[0123]$" > [...] > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 17497.3 0.0 17.1 0.0 0.0 0.8 0.0 0.0 0 82 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 17498.8 0.0 17.1 0.0 0.0 0.8 0.0 0.0 0 82 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 17277.6 0.0 16.9 0.0 0.0 0.8 0.0 0.0 0 82 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 17441.3 0.0 17.0 0.0 0.0 0.8 0.0 0.0 0 82 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 17333.9 0.0 16.9 0.0 0.0 0.8 0.0 0.0 0 82 c0 > > > Now lets see how it looks like for a single SAS connection but dd to 11x > SSDs: > > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t0d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t1d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t2d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t4d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t5d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t6d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t7d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t8d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t9d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t10d0p0& > # dd of=/dev/null bs=1k if=/dev/rdsk/c0t11d0p0& > > # iostat -xnzCM 1|egrep "device|c[0123]$" > [...] > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 104243.3 0.0 101.8 0.0 0.2 9.7 0.0 0.1 0 968 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 104249.2 0.0 101.8 0.0 0.2 9.7 0.0 0.1 0 968 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 104208.1 0.0 101.8 0.0 0.2 9.7 0.0 0.1 0 967 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 104245.8 0.0 101.8 0.0 0.2 9.7 0.0 0.1 0 966 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 104221.9 0.0 101.8 0.0 0.2 9.7 0.0 0.1 0 968 c0 > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 104212.2 0.0 101.8 0.0 0.2 9.7 0.0 0.1 0 967 c0 > > > It looks like a single CPU core still hasn't been saturated and the > bottleneck is in the device rather then OS/CPU. So the MPT driver in Solaris > 2009.06 can do at least 100,000 IOPS to a single SAS port. > > It also scales well - I did run above dd's over 4x SAS ports at the same > time and it scaled linearly by achieving well over 400k IOPS. > > > hw used: x4270, 2x Intel X5570 2.93GHz, 4x SAS SG-PCIE8SAS-E-Z (fw. > 1.27.3.0), connected to F5100. > > > -- > Robert Milkowski > http://milek.blogspot.com > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
