|
Hi Roch; thanks for the discussion yesterday. I said I would plot the
iostat data, here it is. The strange thing is the seeming half duplex operation of ZFS. Here is a 45 minute plot of %zpool iostat 1 The x-axis is data gathered every second. Note that when the write cycle seems to start at 0,300 and 600 seconds that it kills all read ops to the disk. Remember our Canary data? We receive 110,000 50-100kb gzipped files per day. This means we on average are receiving one file every second. We unzip this file, untar it into its original 10 files and then read one of those 1000 line files to parse it. We are therefore writing and reading all the time. I can see data being updated every second in our directories, i assume that ZFS is caching this data. During the 600-800 second range below the system became very unresponsive (1 second ls command turned into 45 second command, pwd takes 15 seconds). A couple of questions is : 1) is it normal for all read operations to go to zero for 5 minutes why only writes are being done? 2) is this something i can tune? 3) is it expected that this would slow down the user experience?  The previous graph shows read/write operations. Here is the same data showing the bytes/second.  A question from this graph is why is read transfers around 20mb/second and writes are around 2mb? Is there a reason there is a 10x difference. For people inside of Sun contact me and i will give you the demiroot login to itsm-mpk-2.sfbay (this is a T2000 32x1000mhz, 8 gig machine running Solaris 10). We are using only the internal 2.5" disk drives to store the 3.5 million files, 3 billion lines of ASCII. We use about 30gigs to store this in a sliding one month window. The internal drive that we are on is at 100% busy all the time. You can see the write cycle below followed by the big read cycle. itsm-mpk-2% iostat -xn 1 999 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 206.2 199.8 10494.9 1286.4 14.1 13.4 34.7 33.1 76 93 c1t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0 0 c0t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 itsm-mpk-2:vold(pid329) extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 21.4 303.3 1263.9 3874.3 19.0 16.0 58.5 49.2 100 100 c1t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 itsm-mpk-2:vold(pid329) itsm-mpk-2% iostat -xn 1 999 | grep c1t0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 21.4 303.3 1263.9 3874.3 19.0 16.0 58.5 49.2 100 100 c1t0d0 1.0 502.0 8.0 1601.1 21.0 16.0 41.7 31.8 100 100 c1t0d0 1.0 442.0 56.0 1479.4 21.6 16.0 48.8 36.1 100 100 c1t0d0 2.0 521.0 128.0 1886.6 21.5 16.0 41.0 30.6 100 100 c1t0d0 0.0 697.0 0.0 2237.0 21.0 16.0 30.1 22.9 100 100 c1t0d0 1.0 736.0 64.0 2469.1 21.0 16.0 28.4 21.7 100 100 c1t0d0 1.0 696.0 64.0 2367.9 21.0 16.0 30.1 22.9 100 100 c1t0d0 0.0 642.0 0.0 2094.1 21.0 16.0 32.7 24.9 100 100 c1t0d0 2.0 661.9 128.0 2302.1 21.0 16.0 31.6 24.1 100 100 c1t0d0 1.0 674.1 64.0 2226.9 21.5 16.0 31.9 23.7 100 100 c1t0d0 0.0 684.1 0.0 2317.7 20.4 16.0 29.9 23.4 100 100 c1t0d0 0.0 677.0 0.0 2289.1 20.0 16.0 29.5 23.6 100 100 c1t0d0 0.0 679.7 0.0 2290.4 20.0 16.0 29.4 23.5 100 100 c1t0d0 0.0 679.5 0.0 2284.0 20.0 16.0 29.4 23.5 100 100 c1t0d0 0.0 675.5 0.0 2340.7 20.0 16.0 29.6 23.7 100 100 c1t0d0 0.0 714.0 0.0 2320.6 20.2 16.0 28.3 22.4 100 100 c1t0d0 0.0 658.9 0.0 2275.2 20.6 16.0 31.2 24.3 100 100 c1t0d0 0.0 696.0 0.0 2376.6 20.1 16.0 28.9 23.0 100 100 c1t0d0 0.0 642.0 0.0 2129.9 21.0 16.0 32.7 24.9 100 100 c1t0d0 0.0 642.0 0.0 2218.6 20.9 16.0 32.5 24.9 100 100 c1t0d0 0.0 658.0 0.0 2839.4 20.0 16.0 30.4 24.3 100 100 c1t0d0 1.0 613.0 8.0 1829.5 20.4 16.0 33.2 26.0 100 100 c1t0d0 0.0 639.0 0.0 1871.5 21.0 16.0 32.8 25.0 100 100 c1t0d0 1.0 645.0 56.0 2096.9 21.0 16.0 32.4 24.7 100 100 c1t0d0 10.0 598.0 208.0 2017.1 21.6 16.0 35.6 26.3 100 100 c1t0d0 1.0 636.0 8.0 2093.0 21.0 16.0 32.9 25.1 100 100 c1t0d0 2.0 657.0 16.0 2217.6 20.6 16.0 31.3 24.3 100 100 c1t0d0 0.0 652.0 0.0 2375.1 21.0 16.0 32.2 24.5 100 100 c1t0d0 0.0 596.0 0.0 2166.5 20.8 16.0 34.9 26.8 100 100 c1t0d0 0.0 645.0 0.0 2347.9 20.0 16.0 31.0 24.8 100 100 c1t0d0 0.0 718.9 0.0 2202.3 20.0 16.0 27.8 22.2 100 100 c1t0d0 0.0 690.0 0.0 2521.1 20.0 16.0 28.9 23.2 100 100 c1t0d0 0.0 703.0 0.0 2602.4 20.0 16.0 28.4 22.7 100 100 c1t0d0 0.0 737.0 0.0 2101.6 20.0 16.0 27.1 21.7 100 100 c1t0d0 1.0 671.9 8.0 2186.1 20.3 16.0 30.1 23.8 100 100 c1t0d0 0.0 639.1 0.0 2207.4 20.9 16.0 32.7 25.0 100 100 c1t0d0 223.2 213.2 13817.3 476.8 20.1 16.0 46.0 36.6 100 100 c1t0d0 103.4 448.6 6171.3 865.6 22.0 16.0 39.8 29.0 100 100 c1t0d0 23.0 677.0 1045.5 1169.0 22.5 16.0 32.2 22.8 100 100 c1t0d0 10.0 660.1 359.5 1151.1 21.2 16.0 31.7 23.9 100 100 c1t0d0 39.0 675.0 2340.5 987.0 21.2 16.0 29.8 22.4 100 100 c1t0d0 11.0 676.0 616.0 923.5 21.1 16.0 30.7 23.3 100 100 c1t0d0 4.0 705.0 160.0 1015.0 21.0 16.0 29.6 22.5 100 100 c1t0d0 4.0 727.0 168.0 1071.5 21.0 16.0 28.7 21.9 100 100 c1t0d0 297.0 79.0 18518.2 247.5 19.9 16.0 52.9 42.5 100 100 c1t0d0 25.0 613.0 1431.0 2088.0 22.4 16.0 35.0 25.1 100 100 c1t0d0 114.0 428.0 6756.1 1703.9 21.2 16.0 39.0 29.5 100 100 c1t0d0 207.9 217.9 12167.2 614.7 20.7 16.0 48.6 37.5 100 100 c1t0d0 363.2 11.0 21245.1 92.5 8.4 16.0 22.6 42.7 93 100 c1t0d0 351.0 7.0 21443.2 98.0 3.3 14.6 9.1 40.7 54 100 c1t0d0 332.7 5.0 20660.7 77.9 3.4 14.2 10.0 42.2 45 100 c1t0d0 354.3 3.0 22784.8 58.6 10.7 15.5 29.9 43.5 85 100 c1t0d0 349.0 4.0 21999.3 58.5 13.2 16.0 37.3 45.3 98 100 c1t0d0 353.0 3.0 22510.4 58.5 8.3 15.5 23.2 43.5 82 100 c1t0d0 344.9 0.0 21540.7 0.0 7.1 15.0 20.6 43.4 63 100 c1t0d0 386.0 0.0 22447.0 0.0 9.4 15.0 24.3 38.9 73 100 c1t0d0 373.1 0.0 20763.3 0.0 14.4 15.7 38.5 42.1 89 100 c1t0d0 364.9 0.0 23145.3 0.0 5.0 14.7 13.8 40.3 54 100 c1t0d0 363.8 0.0 22783.5 0.0 6.0 14.7 16.5 40.4 60 100 c1t0d0 357.3 0.0 22591.9 0.0 10.6 15.7 29.8 43.9 87 100 c1t0d0 369.6 0.0 22441.0 0.0 13.1 15.9 35.5 43.0 94 100 c1t0d0 344.4 1.0 22314.4 8.0 3.3 14.2 9.6 41.1 47 100 c1t0d0 344.0 0.0 22015.9 0.0 6.1 14.4 17.7 41.7 60 100 c1t0d0 372.0 0.0 22818.4 0.0 10.8 15.7 28.9 42.2 82 100 c1t0d0 376.0 2.0 21307.7 12.0 17.0 16.0 44.9 42.3 100 100 c1t0d0 372.1 0.0 23749.3 0.0 6.2 15.0 16.7 40.3 68 100 c1t0d0 347.9 0.0 22507.0 0.0 5.2 14.8 14.8 42.7 59 100 c1t0d0 357.0 0.0 22572.0 0.0 9.6 15.6 26.9 43.7 82 100 c1t0d0 365.0 0.0 21989.4 0.0 14.4 15.9 39.4 43.6 93 100 c1t0d0 355.0 0.0 22558.1 0.0 4.9 14.6 13.7 41.2 51 100 c1t0d0 website for the canary is located at http://canary.sfbay thanks sean --
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. |
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
