hi Frank, there is an interesting pattern here (at least, to my untrained eyes) - your %b starts off quite low:
Frank Penczek wrote: .... > --- > dd'ing to NFS mount: > [EMAIL PROTECTED]://tmp> dd if=./file.tmp of=/home/fpz/file.tmp > 200000+0 records in > 200000+0 records out > 102400000 bytes (102 MB) copied, 11.3959 seconds, 9.0 MB/s > > # iostat -xnz 1 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 2.8 17.3 149.4 127.6 0.0 1.3 0.0 66.0 0 12 c2t8d0 > 2.8 17.3 149.4 127.6 0.0 1.3 0.0 65.9 0 13 c2t9d0 > 2.8 17.3 149.3 127.6 0.0 1.3 0.0 66.1 0 13 c2t10d0 > 2.8 17.3 149.3 127.6 0.0 1.3 0.0 66.4 0 13 c2t11d0 > 2.8 17.3 149.5 127.6 0.0 1.3 0.0 66.5 0 13 c2t12d0 > 0.3 1.0 5.4 133.9 0.0 0.0 0.1 27.2 0 1 c1t1d0 > 0.5 0.3 26.8 16.5 0.0 0.0 0.1 11.1 0 0 c1t0d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 1.0 0.0 8.0 0.0 0.0 0.0 8.9 0 1 c1t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 10.0 0.0 7.0 0.0 0.0 0.0 0.5 0 0 c2t8d0 > 0.0 10.0 0.0 7.5 0.0 0.0 0.0 0.5 0 1 c2t9d0 > 0.0 10.0 0.0 6.0 0.0 0.0 0.0 0.7 0 1 c2t10d0 > 0.0 10.0 0.0 7.0 0.0 0.0 0.0 0.3 0 0 c2t11d0 > 0.0 10.0 0.0 7.5 0.0 0.0 0.0 0.3 0 0 c2t12d0 then it jumps - roughly, quadrupling > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 67.6 0.0 1298.6 0.0 9.8 0.2 145.2 1 71 c2t8d0 > 0.0 64.8 0.0 1139.4 0.0 9.2 0.0 141.8 0 69 c2t9d0 > 0.0 59.2 0.0 898.9 0.0 8.6 0.0 144.9 0 68 c2t10d0 > 0.0 67.6 0.0 1379.4 0.0 9.5 0.0 140.0 0 68 c2t11d0 > 0.0 70.4 0.0 1257.3 0.0 11.4 0.0 162.1 0 73 c2t12d0 then it maxes out and stays that way > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 43.8 0.0 3068.5 0.0 34.9 0.0 796.0 0 100 c2t8d0 > 0.0 55.6 0.0 3891.9 0.0 34.7 0.0 624.9 0 100 c2t9d0 > 0.0 58.8 0.0 4211.9 0.0 33.4 0.0 568.2 0 100 c2t10d0 > 0.0 49.2 0.0 3388.6 0.0 34.5 0.0 702.3 0 100 c2t11d0 > 0.0 57.7 0.0 3805.3 0.0 34.3 0.0 594.0 0 100 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 60.0 0.0 4279.6 0.0 35.0 0.0 583.2 0 100 c2t8d0 > 0.0 48.0 0.0 3423.7 0.0 35.0 0.0 729.1 0 100 c2t9d0 > 0.0 41.0 0.0 2910.3 0.0 35.0 0.0 853.6 0 100 c2t10d0 > 0.0 50.0 0.0 3552.2 0.0 35.0 0.0 699.9 0 100 c2t11d0 > 0.0 48.0 0.0 3423.7 0.0 35.0 0.0 729.1 0 100 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 48.0 0.0 3424.6 0.0 35.0 0.0 728.9 0 100 c2t8d0 > 0.0 60.0 0.0 4280.8 0.0 35.0 0.0 583.1 0 100 c2t9d0 > 0.0 55.0 0.0 3938.2 0.0 35.0 0.0 636.1 0 100 c2t10d0 > 0.0 56.0 0.0 4024.3 0.0 35.0 0.0 624.7 0 100 c2t11d0 > 0.0 48.0 0.0 3424.6 0.0 35.0 0.0 728.9 0 100 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 52.0 0.0 3723.5 0.0 35.0 0.0 672.9 0 100 c2t8d0 > 0.0 43.0 0.0 3081.5 0.0 35.0 0.0 813.8 0 100 c2t9d0 > 0.0 46.0 0.0 3296.0 0.0 35.0 0.0 760.7 0 100 c2t10d0 > 0.0 48.0 0.0 3424.0 0.0 35.0 0.0 729.0 0 100 c2t11d0 > 0.0 62.0 0.0 4408.1 0.0 35.0 0.0 564.4 0 100 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 60.0 0.0 4279.8 0.0 35.0 0.0 583.2 0 100 c2t8d0 > 0.0 57.0 0.0 4065.8 0.0 35.0 0.0 613.9 0 100 c2t9d0 > 0.0 59.0 0.0 4194.3 0.0 35.0 0.0 593.1 0 100 c2t10d0 > 0.0 56.0 0.0 4023.3 0.0 35.0 0.0 624.9 0 100 c2t11d0 > 0.0 48.0 0.0 3424.3 0.0 35.0 0.0 729.1 0 100 c2t12d0 drops back a fraction > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 65.7 0.0 1385.0 0.0 14.5 0.0 220.8 0 90 c2t8d0 > 0.9 68.4 39.8 1623.6 0.0 13.0 0.0 187.8 0 87 c2t9d0 > 0.9 74.9 39.3 2054.6 0.0 16.7 0.0 219.6 0 94 c2t10d0 > 0.9 70.3 39.3 1662.9 0.0 15.4 0.0 216.1 0 95 c2t11d0 > 0.0 68.4 0.0 1736.0 0.0 14.9 0.0 217.9 0 87 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 65.3 0.0 3287.1 0.0 29.2 0.0 447.8 0 99 c2t8d0 > 0.0 55.5 0.0 2642.4 0.0 28.2 0.0 508.9 0 99 c2t9d0 > 0.0 47.9 0.0 2130.0 0.0 26.7 0.0 558.2 0 100 c2t10d0 > 0.0 66.4 0.0 3336.1 0.0 29.3 0.0 441.2 0 100 c2t11d0 > 0.0 65.3 0.0 3103.3 0.0 29.7 0.0 454.7 0 99 c2t12d0 > 0.0 1.1 0.0 2.2 0.0 0.0 0.0 10.0 0 1 c1t1d0 but quickly reverts to 100%: > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 44.0 0.0 3125.2 0.0 35.0 0.0 795.1 0 100 c2t8d0 > 0.0 50.0 0.0 3553.8 0.0 35.0 0.0 699.7 0 100 c2t9d0 > 0.0 55.0 0.0 3895.8 0.0 35.0 0.0 636.1 0 100 c2t10d0 > 0.0 44.0 0.0 3081.7 0.0 35.0 0.0 795.1 0 100 c2t11d0 > 0.0 48.0 0.0 3424.7 0.0 35.0 0.0 728.8 0 100 c2t12d0 > 0.0 1.0 0.0 8.0 0.0 0.0 0.0 8.7 0 1 c1t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 52.0 0.0 3724.6 0.0 35.0 0.0 672.8 0 100 c2t8d0 > 0.0 46.0 0.0 3253.1 0.0 35.0 0.0 760.6 0 100 c2t9d0 > 0.0 38.0 0.0 2697.0 0.0 35.0 0.0 920.7 0 100 c2t10d0 > 0.0 51.0 0.0 3638.6 0.0 35.0 0.0 686.0 0 100 c2t11d0 > 0.0 48.0 0.0 3424.6 0.0 35.0 0.0 728.9 0 100 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 44.0 0.0 2915.0 0.0 22.9 0.0 521.2 0 100 c2t8d0 > 0.0 47.0 0.0 3382.1 0.0 24.1 0.0 512.0 0 100 c2t9d0 > 0.0 56.0 0.0 4024.2 0.0 25.7 0.0 459.3 0 100 c2t10d0 > 0.0 41.0 0.0 2954.1 0.0 22.7 0.0 552.4 0 100 c2t11d0 > 0.0 46.0 0.0 3083.6 0.0 22.8 0.0 494.7 0 98 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 103.0 0.0 207.5 0.0 5.4 0.0 52.4 0 89 c2t8d0 > 0.0 101.0 0.0 381.4 0.0 4.8 0.0 47.2 0 89 c2t9d0 > 0.0 102.0 0.0 432.9 0.0 4.0 0.0 39.5 0 79 c2t10d0 > 0.0 112.0 0.0 257.5 0.0 5.9 0.0 52.4 0 95 c2t11d0 > 0.0 111.0 0.0 206.5 0.0 6.1 0.0 54.8 0 92 c2t12d0 and then tails off > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 102.0 0.0 213.0 0.0 4.7 0.0 46.3 0 78 c2t8d0 > 0.0 106.0 0.0 214.5 0.0 5.0 0.0 47.6 0 82 c2t9d0 > 0.0 95.0 0.0 214.5 0.0 4.3 0.0 45.5 0 71 c2t10d0 > 0.0 97.0 0.0 214.0 0.0 4.7 0.0 48.9 0 80 c2t11d0 > 0.0 99.0 0.0 216.5 0.0 5.2 0.0 52.7 0 90 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 67.0 0.0 104.0 0.0 3.2 0.0 47.3 0 55 c2t8d0 > 0.0 68.0 0.0 106.5 0.0 3.4 0.0 49.6 0 58 c2t9d0 > 0.0 66.0 0.0 101.5 0.0 3.2 0.0 48.6 0 60 c2t10d0 > 0.0 64.0 0.0 103.0 0.0 3.1 0.0 48.0 0 57 c2t11d0 > 0.0 69.0 0.0 103.5 0.0 3.1 0.0 45.4 0 62 c2t12d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 1.0 0.0 1.5 0.0 0.0 0.0 10.2 0 1 c1t0d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 1.0 0.0 0.5 0.0 0.0 0.0 10.3 0 1 c1t1d0 All of which, to me, look like you're filling a buffer or two. I don't recall the config of your zpool, but if the devices are disks that are direct or san-attached, I would be wondering about their outstanding queue depths. I think it's time to break out some D to find out where in the stack the bottleneck(s) really are. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
