I dig deeper into it and might find some useful information.
I attached an X25 SSD for ZIL to see if it helps. but no luck.
I run IOstate -xnz for more details and got interesting result as below.(maybe
too long)
some explaination:
1. c2d0 is SSD for ZIL
2. c0t3d0, c0t20d0, c0t21d0, c0t22d0 is source pool.
3. you can see the first result is the same as zpool iostate -v, and I've ran
-xnz several times, the first result were always good.
4. then, 30s later comes up the second one, and then 3rd, 4th. you can see my
source pool disks are extremely slow and busy. One by One. asvc_t is high, %b
is high. that's abnormal.
5. I think it might because those 4 drives are the old WD green disks with
"defect" of disk head park/unpark and TLER. other 12 2TB disks are the latest
one which already fixed.
6. Let me try to test on 1TB disk and see if any issue.
r...@sh1slcs001:/volumes# iostat -xnz 30
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.4 0.0 3.0 0.3 0.0 0.0 0.0 0.2 0 0 c2d0
1.4 8.9 57.8 40.3 0.0 0.1 2.5 5.8 1 5 c3d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 986.5 0 0 fd0
2.8 154.8 141.5 16517.7 0.0 2.4 0.1 15.0 2 25 c0t2d0
408.4 0.3 29140.4 0.9 0.0 4.1 0.1 10.1 2 55 c0t3d0
2.8 154.9 136.7 16519.6 0.0 2.4 0.1 14.9 2 25 c0t5d0
2.8 155.0 139.4 16520.9 0.0 2.4 0.1 15.0 2 26 c0t6d0
0.0 0.0 0.5 0.0 0.0 0.0 0.0 0.1 0 0 c0t7d0
0.0 0.0 0.5 0.0 0.0 0.0 0.0 3.4 0 0 c0t8d0
0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.1 0 0 c0t9d0
2.8 154.9 139.2 16523.9 0.0 2.4 0.1 15.2 2 26 c0t10d0
2.8 155.0 136.1 16525.4 0.0 2.5 0.1 15.7 2 26 c0t11d0
409.9 0.3 29152.1 0.9 0.0 3.9 0.1 9.4 2 52 c0t20d0
409.5 0.3 29152.0 0.9 0.0 3.9 0.1 9.6 2 52 c0t21d0
409.2 0.3 29155.7 0.9 0.0 4.1 0.1 10.0 2 55 c0t22d0
2.8 155.0 138.6 16527.6 0.0 2.4 0.1 15.2 2 26 c0t23d0
2.9 154.9 138.0 16528.8 0.0 2.4 0.1 15.1 2 26 c0t24d0
2.9 155.0 136.1 16530.2 0.0 2.5 0.1 15.8 2 27 c0t25d0
2.8 155.1 139.7 16531.6 0.0 2.4 0.1 15.5 2 26 c0t26d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.3 0.0 1.2 0.0 0.0 0.0 0.0 0.1 0 0 c2d0
0.1 17.7 0.1 51.7 0.0 0.1 0.2 4.1 0 7 c3d0
0.1 2.1 0.0 79.8 0.0 0.0 0.1 4.0 0 0 c0t2d0
0.2 0.0 7.1 0.0 0.1 2.3 278.5 11365.1 1 46 c0t3d0
0.1 2.2 0.0 79.9 0.0 0.0 0.1 3.7 0 0 c0t5d0
0.1 2.3 0.0 80.0 0.0 0.0 0.1 9.2 0 0 c0t6d0
0.1 2.5 0.0 80.1 0.0 0.0 0.1 3.8 0 0 c0t10d0
0.1 2.4 0.0 80.0 0.0 0.0 0.1 9.5 0 0 c0t11d0
1.9 0.0 133.0 0.0 0.1 2.8 60.2 1520.6 2 51 c0t20d0
1.6 0.0 110.9 0.0 0.0 0.0 0.0 7.6 0 0 c0t21d0
1.6 0.0 109.5 0.0 0.0 0.0 0.0 11.5 0 0 c0t22d0
0.1 2.4 0.0 80.0 0.0 0.0 0.1 8.6 0 0 c0t23d0
0.1 2.6 0.0 80.0 0.0 0.0 0.1 2.9 0 0 c0t24d0
0.1 2.4 0.0 79.9 0.0 0.0 0.1 6.2 0 0 c0t25d0
0.1 2.3 0.0 79.9 0.0 0.0 0.1 7.8 0 0 c0t26d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 0.0 0.6 0.0 0.0 0.0 0.0 0.1 0 0 c2d0
0.0 21.3 0.0 82.3 0.0 0.1 0.4 4.0 0 8 c3d0
0.0 2.1 0.0 68.2 0.0 0.0 0.2 3.8 0 0 c0t2d0
0.7 0.0 39.1 0.0 0.0 0.6 64.0 884.1 1 10 c0t3d0
0.0 2.2 0.0 68.2 0.0 0.0 0.1 3.5 0 0 c0t5d0
0.0 2.3 0.0 68.3 0.0 0.0 0.1 3.5 0 0 c0t6d0
0.0 2.3 0.0 68.3 0.0 0.0 0.1 3.2 0 0 c0t10d0
0.0 2.2 0.0 68.3 0.0 0.0 0.1 3.7 0 0 c0t11d0
2.0 0.0 133.7 0.0 0.0 0.0 0.0 9.4 0 0 c0t20d0
2.1 0.0 135.8 0.0 0.1 5.2 67.8 2498.1 3 88 c0t21d0
2.8 0.0 134.4 0.0 0.0 0.0 0.0 3.2 0 0 c0t22d0
0.0 2.1 0.0 68.2 0.0 0.0 0.1 3.6 0 0 c0t23d0
0.0 2.4 0.0 68.4 0.0 0.0 0.1 3.1 0 0 c0t24d0
0.0 2.4 0.0 68.4 0.0 0.0 0.1 3.4 0 0 c0t25d0
0.0 2.2 0.0 68.3 0.0 0.0 0.1 3.3 0 0 c0t26d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 0.0 0.6 0.0 0.0 0.0 0.0 0.1 0 0 c2d0
0.1 16.5 0.1 44.5 0.0 0.1 0.2 4.8 0 7 c3d0
0.0 2.3 0.0 73.8 0.0 0.0 0.1 3.8 0 0 c0t2d0
3.5 0.0 246.8 0.0 0.0 0.8 6.3 229.8 1 20 c0t3d0
0.0 2.0 0.0 73.7 0.0 0.1 0.1 48.3 0 2 c0t5d0
0.0 2.2 0.0 73.8 0.0 0.0 0.1 5.4 0 0 c0t6d0
0.0 2.4 0.0 74.0 0.0 0.0 0.1 5.4 0 0 c0t10d0
0.0 2.5 0.0 74.0 0.0 0.0 0.1 4.7 0 0 c0t11d0
3.1 0.0 136.6 0.0 0.0 0.0 0.0 1.4 0 0 c0t20d0
0.7 0.0 29.2 0.0 0.0 0.6 0.0 911.0 0 12 c0t21d0
1.9 0.0 138.7 0.0 0.1 4.7 73.0 2428.6 2 66 c0t22d0
0.0 2.2 0.0 73.9 0.0 0.0 0.1 5.4 0 0 c0t23d0
0.0 2.3 0.0 73.9 0.0 0.1 0.1 27.7 0 2 c0t24d0
0.0 2.4 0.0 74.0 0.0 0.0 0.1 4.7 0 0 c0t25d0
0.0 2.3 0.0 73.9 0.0 0.0 0.1 7.3 0 0 c0t26d0
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
