Hello Matthew,
Wednesday, March 14, 2007, 9:00:28 AM, you wrote:
MA> Robert Milkowski wrote:
>> Hello zfs-discuss,
>>
>> Subject says it all.
>>
>>
>> I first checked - no IO activity at all to the pool named thumper-2.
>> So I started replacing one drive with 'zpool replace thumper-2 c7t7d0
>> c4t1d0'.
>>
>> Now the question is why am I seeing writes to other disks than c7t7d0?
MA> Are you *sure* that nothing else is going on? Not even atime updates?
MA> Do 'zfs umount -a' and see if there's still writes to other disks.
MA> There may be some small amount of writes to update some metadata with
MA> the resilvering status.
MA> I just did this yesterday on a raidz2 pool and didn't see writes to the
MA> other disks. Maybe the code has changed since you tried?
There're data in thumper-8 but no activity at all is happening to
local disk on a server. I did check it with iostat and zpool iostat
for dozen of seconds - no IOs at all.
bash-3.00# zpool replace thumper-8 c7t7d0 c4t1d0
bash-3.00# zpool status
pool: misc
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
misc ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t0d0s4 ONLINE 0 0 0
c5t4d0s4 ONLINE 0 0 0
errors: No known data errors
pool: thumper-8
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0.12% done, 9h1m to go
config:
NAME STATE READ WRITE CKSUM
thumper-8 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c6t4d0 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
c7t5d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c4t6d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c7t6d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
spare ONLINE 0 0 0
c7t7d0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
spares
c4t1d0 INUSE currently in use
c4t2d0 AVAIL
errors: No known data errors
bash-3.00#
bash-3.00# iostat -xnz 1
[stripped out first output]
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
58.2 0.0 195.2 0.0 0.0 0.1 0.3 1.6 2 10 c4t0d0
60.2 0.0 218.7 0.0 0.0 0.1 0.6 1.7 4 10 c6t0d0
72.2 0.0 239.8 0.0 0.0 0.1 0.2 1.4 1 10 c0t0d0
0.0 335.1 0.0 1114.8 0.0 0.1 0.1 0.2 2 5 c4t1d0
62.2 0.0 221.2 0.0 0.0 0.1 0.0 1.5 0 9 c6t1d0
66.2 0.0 207.2 0.0 0.0 0.1 0.0 1.4 0 9 c0t1d0
66.2 0.0 214.2 0.0 0.0 0.1 0.1 1.5 0 10 c5t1d0
16.1 0.0 45.7 0.0 0.0 0.0 0.0 1.3 0 2 c6t2d0
45.2 0.0 233.3 0.0 0.0 0.1 0.2 2.2 1 10 c4t3d0
70.2 0.0 231.3 0.0 0.0 0.1 0.4 2.0 2 14 c0t2d0
18.1 0.0 46.7 0.0 0.0 0.0 0.0 2.5 0 5 c5t2d0
34.1 0.0 182.6 0.0 0.0 0.1 0.3 2.7 1 9 c6t3d0
39.1 0.0 222.8 0.0 0.0 0.1 0.7 2.1 3 8 c5t3d0
21.1 0.0 48.7 0.0 0.0 0.0 0.0 1.9 0 4 c0t3d0
24.1 0.0 72.2 0.0 0.0 0.1 0.0 2.6 0 6 c4t4d0
15.1 0.0 52.7 0.0 0.0 0.0 0.0 2.9 0 4 c6t4d0
23.1 0.0 64.2 0.0 0.0 0.1 0.3 2.2 1 5 c0t4d0
47.2 0.0 249.3 0.0 0.0 0.1 0.0 1.5 0 7 c4t5d0
39.1 0.0 207.7 0.0 0.0 0.1 0.4 1.4 1 5 c6t5d0
66.2 0.0 206.7 0.0 0.0 0.1 0.4 1.5 2 10 c1t0d0
38.1 0.0 199.7 0.0 0.0 0.1 0.2 1.6 1 6 c0t5d0
49.2 223.8 1126.8 123.4 1.6 0.3 5.9 1.3 30 35 c4t6d0
39.1 0.0 222.8 0.0 0.0 0.1 0.5 1.9 1 7 c5t5d0
65.2 0.0 221.2 0.0 0.0 0.1 0.4 1.8 2 12 c1t1d0
45.2 202.7 1127.8 108.9 1.5 0.4 5.9 1.5 35 37 c6t6d0
42.1 0.0 216.7 0.0 0.0 0.1 0.2 1.8 1 8 c0t6d0
44.1 210.7 1141.9 114.9 1.5 0.4 6.0 1.4 35 37 c4t7d0
39.1 217.7 1126.8 124.9 1.8 0.4 7.0 1.5 35 38 c5t6d0
15.1 0.0 45.2 0.0 0.0 0.0 0.7 3.0 1 4 c1t2d0
40.1 182.6 947.7 97.8 1.5 0.3 6.7 1.5 32 34 c6t7d0
58.2 204.7 1136.3 115.9 1.3 0.3 5.0 1.2 28 32 c0t7d0
37.1 178.6 1067.1 97.8 2.0 0.4 9.3 1.8 37 38 c5t7d0
17.1 0.0 46.2 0.0 0.0 0.0 0.0 1.4 0 2 c1t3d0
25.1 0.0 57.7 0.0 0.0 0.0 0.0 1.3 0 3 c1t4d0
44.1 0.0 217.7 0.0 0.0 0.0 0.0 1.0 0 4 c1t5d0
43.1 217.7 1024.5 118.4 2.7 0.4 10.2 1.7 43 44 c1t6d0
59.2 209.7 1145.4 114.4 1.2 0.3 4.3 1.0 26 28 c1t7d0
65.2 0.0 215.2 0.0 0.0 0.1 0.2 2.0 2 13 c7t0d0
61.2 0.0 219.2 0.0 0.0 0.1 0.4 1.3 2 8 c7t1d0
15.1 0.0 52.7 0.0 0.0 0.0 0.0 1.4 0 2 c7t2d0
37.1 0.0 191.6 0.0 0.0 0.1 0.3 2.1 1 8 c7t3d0
13.0 0.0 29.1 0.0 0.0 0.0 0.0 1.3 0 2 c7t4d0
42.1 0.0 224.3 0.0 0.0 0.1 0.1 1.3 0 5 c7t5d0
41.1 191.6 1079.7 102.8 1.7 0.3 7.1 1.5 30 34 c7t6d0
0.0 335.1 0.0 1114.8 0.0 0.1 0.1 0.1 2 5 c7t7d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
52.0 0.0 109.0 0.0 0.0 0.1 0.2 1.2 1 6 c4t0d0
38.0 0.0 80.0 0.0 0.0 0.0 0.1 1.0 1 4 c6t0d0
41.0 0.0 111.5 0.0 0.0 0.1 0.4 1.4 1 6 c0t0d0
0.0 336.0 0.0 1579.0 0.0 0.1 0.1 0.2 2 5 c4t1d0
41.0 0.0 111.0 0.0 0.0 0.0 0.1 1.1 0 4 c6t1d0
40.0 0.0 103.5 0.0 0.0 0.1 0.7 1.4 2 6 c0t1d0
43.0 0.0 112.0 0.0 0.0 0.1 0.5 2.1 2 9 c5t1d0
32.0 0.0 53.5 0.0 0.0 0.0 0.0 1.0 0 3 c6t2d0
44.0 0.0 277.5 0.0 0.0 0.0 0.3 0.9 1 4 c4t3d0
41.0 0.0 111.5 0.0 0.0 0.0 0.5 1.2 1 5 c0t2d0
33.0 0.0 54.0 0.0 0.0 0.0 0.0 1.4 0 5 c5t2d0
50.0 0.0 287.5 0.0 0.0 0.0 0.5 1.0 2 5 c6t3d0
43.0 0.0 262.0 0.0 0.0 0.1 0.4 1.8 1 8 c5t3d0
34.0 0.0 47.0 0.0 0.0 0.0 0.0 1.0 0 3 c0t3d0
36.0 0.0 63.0 0.0 0.0 0.1 0.0 1.6 0 6 c4t4d0
35.0 0.0 63.0 0.0 0.0 0.0 0.0 1.1 0 4 c6t4d0
39.0 0.0 49.5 0.0 0.0 0.1 0.0 1.7 0 6 c0t4d0
46.0 0.0 255.5 0.0 0.0 0.1 0.4 1.6 1 7 c4t5d0
44.0 0.0 269.5 0.0 0.0 0.1 0.7 2.4 1 11 c6t5d0
49.0 0.0 116.0 0.0 0.0 0.1 0.2 1.4 1 7 c1t0d0
48.0 0.0 271.5 0.0 0.0 0.1 0.5 2.5 2 12 c0t5d0
44.0 212.0 1509.0 114.0 1.5 0.3 5.7 1.0 25 27 c4t6d0
43.0 0.0 246.5 0.0 0.0 0.1 0.7 1.6 1 7 c5t5d0
43.0 0.0 112.5 0.0 0.0 0.1 0.2 1.5 1 6 c1t1d0
42.0 209.0 1516.5 113.0 1.6 0.3 6.4 1.0 23 26 c6t6d0
42.0 0.0 276.0 0.0 0.1 0.1 1.3 1.8 3 7 c0t6d0
40.0 210.0 1501.0 112.0 2.3 0.3 9.0 1.2 28 30 c4t7d0
47.0 214.0 1516.5 113.5 1.7 0.3 6.6 1.1 27 29 c5t6d0
33.0 0.0 54.0 0.0 0.0 0.0 0.0 1.1 0 4 c1t2d0
51.0 207.0 1407.0 117.0 1.5 0.3 5.7 1.1 28 30 c6t7d0
43.0 214.0 1515.0 116.5 2.6 0.3 10.3 1.3 32 34 c0t7d0
42.0 224.0 1525.0 125.0 2.2 0.3 8.4 1.2 29 31 c5t7d0
37.0 0.0 71.0 0.0 0.0 0.0 0.0 1.2 0 4 c1t3d0
33.0 0.0 69.0 0.0 0.0 0.0 0.0 1.2 0 4 c1t4d0
49.0 0.0 272.0 0.0 0.0 0.1 0.3 1.4 1 7 c1t5d0
46.0 205.0 1356.5 114.5 2.6 0.3 10.5 1.3 31 33 c1t6d0
40.0 210.0 1501.0 112.0 2.5 0.3 9.8 1.3 30 31 c1t7d0
34.0 0.0 84.5 0.0 0.0 0.0 0.1 1.0 0 3 c7t0d0
40.0 0.0 103.5 0.0 0.0 0.1 0.5 1.7 2 7 c7t1d0
37.0 0.0 48.5 0.0 0.0 0.0 0.0 1.2 0 5 c7t2d0
50.0 0.0 272.5 0.0 0.1 0.1 1.3 1.7 5 8 c7t3d0
31.0 0.0 53.5 0.0 0.0 0.1 0.0 2.0 0 6 c7t4d0
38.0 0.0 244.5 0.0 0.0 0.1 0.7 1.6 1 6 c7t5d0
48.0 212.0 1571.0 118.0 2.9 0.3 11.1 1.2 30 32 c7t6d0
0.0 339.0 0.0 1579.0 0.0 0.1 0.1 0.2 2 5 c7t7d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
82.0 0.0 289.0 0.0 0.0 0.1 0.4 1.7 3 14 c4t0d0
85.0 0.0 313.0 0.0 0.0 0.1 0.2 1.3 2 11 c6t0d0
80.0 0.0 273.0 0.0 0.0 0.1 0.1 1.6 1 13 c0t0d0
0.0 438.0 0.0 1692.5 0.0 0.1 0.1 0.1 3 6 c4t1d0
83.0 0.0 290.0 0.0 0.0 0.1 0.1 1.5 1 13 c6t1d0
77.0 0.0 280.0 0.0 0.0 0.1 0.2 1.4 1 11 c0t1d0
80.0 0.0 265.5 0.0 0.0 0.1 0.1 1.3 1 10 c5t1d0
18.0 0.0 62.0 0.0 0.0 0.1 0.0 3.0 0 5 c6t2d0
28.0 0.0 171.5 0.0 0.0 0.1 1.1 2.6 2 7 c4t3d0
83.0 0.0 290.0 0.0 0.0 0.1 0.0 1.1 0 9 c0t2d0
17.0 0.0 68.5 0.0 0.0 0.0 0.0 2.8 0 5 c5t2d0
27.0 0.0 178.5 0.0 0.0 0.1 1.2 3.0 3 8 c6t3d0
28.0 0.0 179.0 0.0 0.0 0.1 0.3 2.8 1 8 c5t3d0
16.0 0.0 45.5 0.0 0.0 0.0 0.5 2.4 1 4 c0t3d0
19.0 0.0 77.0 0.0 0.0 0.0 0.0 1.6 0 3 c4t4d0
21.0 0.0 78.0 0.0 0.0 0.0 0.6 2.3 1 5 c6t4d0
22.0 0.0 63.5 0.0 0.0 0.0 0.0 1.9 0 4 c0t4d0
30.0 0.0 157.5 0.0 0.0 0.1 0.0 3.0 0 9 c4t5d0
31.0 0.0 165.5 0.0 0.0 0.1 1.2 2.3 2 7 c6t5d0
83.0 0.0 275.5 0.0 0.0 0.1 0.3 1.5 2 12 c1t0d0
31.0 0.0 158.0 0.0 0.0 0.1 0.6 2.2 2 7 c0t5d0
70.0 231.0 1712.5 131.0 2.4 0.5 8.0 1.7 48 52 c4t6d0
25.0 0.0 147.5 0.0 0.0 0.0 0.2 1.9 1 5 c5t5d0
81.0 0.0 282.0 0.0 0.0 0.1 0.0 1.2 0 10 c1t1d0
67.0 230.0 1735.5 127.0 1.8 0.5 6.2 1.8 46 52 c6t6d0
25.0 0.0 147.5 0.0 0.0 0.1 0.4 2.4 0 6 c0t6d0
74.0 237.0 1718.5 133.0 1.8 0.5 5.7 1.5 41 47 c4t7d0
70.0 241.0 1713.0 131.5 2.2 0.5 7.0 1.6 42 48 c5t6d0
15.0 0.0 52.5 0.0 0.0 0.1 0.0 3.5 0 5 c1t2d0
68.0 225.0 1523.0 128.5 2.6 0.5 9.0 1.7 43 49 c6t7d0
66.0 230.0 1724.0 132.0 2.4 0.6 8.0 2.0 52 58 c0t7d0
78.0 230.0 1717.5 130.5 1.5 0.5 4.8 1.5 39 46 c5t7d0
16.0 0.0 45.5 0.0 0.0 0.1 0.0 4.2 0 7 c1t3d0
24.0 0.0 79.5 0.0 0.0 0.1 0.0 3.2 0 8 c1t4d0
34.0 0.0 167.0 0.0 0.0 0.1 0.1 2.4 0 8 c1t5d0
69.0 233.0 1566.0 134.5 1.9 0.5 6.4 1.7 46 50 c1t6d0
70.0 236.0 1717.5 131.0 1.7 0.4 5.5 1.5 39 45 c1t7d0
84.0 0.0 283.0 0.0 0.0 0.1 0.0 1.5 0 12 c7t0d0
86.0 0.0 291.5 0.0 0.0 0.1 0.2 1.3 1 11 c7t1d0
16.0 0.0 45.5 0.0 0.0 0.0 0.0 1.8 0 3 c7t2d0
27.0 0.0 171.0 0.0 0.0 0.1 0.8 2.2 2 6 c7t3d0
17.0 0.0 53.5 0.0 0.0 0.0 0.0 2.3 0 4 c7t4d0
29.0 0.0 164.5 0.0 0.0 0.1 0.8 3.0 2 9 c7t5d0
68.0 225.0 1718.0 130.5 2.5 0.5 8.6 1.8 46 53 c7t6d0
0.0 434.0 0.0 1692.5 0.1 0.1 0.1 0.1 3 6 c7t7d0
^C
bash-3.00#
As you can see there are writes to other disks.
bash-3.00# zfs get -r atime thumper-8
NAME PROPERTY VALUE SOURCE
thumper-8 atime off local
thumper-8/poczta atime off inherited from
thumper-8
thumper-8/poczta/d6601 atime off inherited
from thumper-8
thumper-8/poczta/d6602 atime off inherited
from thumper-8
thumper-8/poczta/d6603 atime off inherited
from thumper-8
thumper-8/poczta/d6604 atime off inherited
from thumper-8
thumper-8/poczta/d6605 atime off inherited
from thumper-8
thumper-8/poczta/[EMAIL PROTECTED] atime - -
thumper-8/poczta/[EMAIL PROTECTED] atime - -
thumper-8/poczta/d6606 atime off inherited
from thumper-8
thumper-8/poczta/d6607 atime off inherited
from thumper-8
thumper-8/poczta/d6608 atime off inherited
from thumper-8
thumper-8/poczta/d6609 atime off inherited
from thumper-8
thumper-8/poczta/d6610 atime off inherited
from thumper-8
bash-3.00#
So atime is off for all file systems in the pool.
I did 'zfs umount -a' and checked with 'df -h' that all zfs file
systems are unmounted but still I can see many writes to other disks.
System is:
bash-3.00# uname -a
SunOS thumper-8.srv 5.10 Generic_125101-02 i86pc i386 i86pc
Also 'zpool iostat 1' looks strange:
bash-3.00# zpool iostat 1
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 109 62 656K 1.20M
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 1.06K 489 19.0M 705K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 944 431 29.3M 616K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 754 305 22.5M 410K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 904 401 13.8M 614K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 911 525 10.3M 759K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 898 348 30.6M 511K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 993 385 24.2M 563K
---------- ----- ----- ----- ----- ----- -----
misc 78.5K 440G 0 0 0 0
thumper-8 1.40T 18.6T 1.24K 622 23.1M 837K
---------- ----- ----- ----- ----- ----- -----
^C
bash-3.00#
And also:
bash-3.00# iostat -xnz 1|egrep "c7t7d0|c4t1d0"
0.0 0.3 0.0 1.3 0.0 0.0 2.9 0.2 0 0 c4t1d0
5.4 8.9 45.3 41.0 0.0 0.0 0.9 1.4 0 2 c7t7d0
0.0 784.2 0.0 1000.6 0.1 0.1 0.1 0.1 4 11 c4t1d0
0.0 780.2 0.0 1000.1 0.1 0.1 0.1 0.2 5 12 c7t7d0
0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 6 14 c4t1d0
0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 7 15 c7t7d0
0.0 811.1 0.0 1175.2 0.1 0.1 0.1 0.1 4 12 c4t1d0
0.0 815.1 0.0 1175.1 0.1 0.1 0.1 0.1 5 12 c7t7d0
0.0 849.3 0.0 3577.6 0.3 0.2 0.4 0.2 9 15 c4t1d0
0.0 846.3 0.0 3577.6 0.2 0.1 0.3 0.2 8 15 c7t7d0
0.0 1029.0 0.0 1630.5 0.8 0.3 0.8 0.2 19 26 c4t1d0
0.0 1027.0 0.0 1630.5 0.9 0.2 0.9 0.2 16 23 c7t7d0
0.0 1009.1 0.0 532.5 0.2 0.2 0.2 0.2 8 17 c4t1d0
0.0 1024.0 0.0 532.5 0.1 0.2 0.1 0.2 7 17 c7t7d0
0.0 1132.7 0.0 2599.6 0.2 0.2 0.1 0.2 8 18 c4t1d0
0.0 1137.6 0.0 2599.8 0.2 0.2 0.1 0.2 9 17 c7t7d0
0.0 1266.3 0.0 1730.5 0.4 0.2 0.3 0.2 10 19 c4t1d0
0.0 1285.5 0.0 1730.3 0.2 0.2 0.2 0.2 9 20 c7t7d0
0.0 963.8 0.0 5626.8 1.3 0.2 1.3 0.2 14 21 c4t1d0
0.0 963.8 0.0 5626.9 0.7 0.2 0.8 0.2 13 21 c7t7d0
0.0 1023.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c4t1d0
0.0 1025.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c7t7d0
^C
bash-3.00#
Data should be copied onto c4t1d0 disk - as you can see data are being
written also to c7t7d0 which is being replaced.
Lets see how it looks on all controllers of x4500:
bash-3.00# iostat -xnzC 1|egrep "device| c[0-9]$"
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
43.4 69.9 364.5 317.4 0.1 0.2 0.9 1.4 0 16 c0
43.5 70.0 366.4 317.5 0.1 0.2 1.0 1.4 0 16 c1
32.6 52.8 275.9 240.2 0.1 0.1 1.0 1.4 0 12 c4
32.6 54.1 276.1 241.1 0.1 0.1 1.0 1.4 0 12 c5
43.4 70.0 366.4 317.5 0.1 0.2 1.0 1.4 0 16 c6
43.4 70.1 364.5 319.4 0.1 0.2 1.0 1.4 0 16 c7
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
1012.2 1023.3 9568.7 898.6 28.8 3.7 14.2 1.8 5 371 c0
621.7 2167.6 5802.9 1812.1 30.8 3.0 11.0 1.1 4 301 c1
611.7 3059.6 5863.9 2700.2 58.3 4.4 15.9 1.2 7 438 c4
610.7 2091.4 5851.8 1879.7 36.4 3.3 13.5 1.2 5 331 c5
788.9 1998.3 7714.5 1799.6 50.1 4.2 18.0 1.5 6 423 c6
804.9 2076.4 7675.4 1832.1 36.6 4.0 12.7 1.4 6 401 c7
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
869.0 1347.0 7780.3 1227.0 21.3 3.7 9.6 1.7 5 366 c0
522.0 2496.2 4626.9 2421.7 32.7 3.0 10.8 1.0 4 299 c1
511.0 3692.4 4513.4 3642.9 45.3 3.7 10.8 0.9 5 371 c4
512.1 2381.2 4516.5 2381.7 33.8 2.9 11.7 1.0 4 294 c5
704.1 2200.2 6146.2 2436.3 44.5 3.9 15.3 1.3 6 388 c6
706.1 2180.3 6184.7 2413.8 41.5 3.9 14.4 1.3 5 388 c7
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
220.0 113.0 1356.5 56.5 0.5 0.9 1.4 2.6 0 85 c0
182.0 240.0 1570.0 121.0 0.4 0.7 1.0 1.7 0 72 c1
121.0 383.0 1521.5 677.5 0.3 0.5 0.6 0.9 0 48 c4
149.0 238.0 1549.5 119.5 0.3 0.5 0.8 1.4 0 54 c5
183.0 224.0 1585.0 112.5 0.3 0.6 0.7 1.5 0 63 c6
171.0 259.0 1166.5 614.0 0.4 0.7 0.9 1.5 0 66 c7
^C
bash-3.00#
So there're reads and writes basically to all disks.
Again, I'm 100% sure no other activity is on the server.
>> Also why in case of replacing a disk we do not just copy disk-to-disk?
>> It would be MUCH faster here. Probably 'coz we're traversing
>> meta-data? But perhaps it could be done in a clever way so we endup
>> just copying from one disk to another. Checking parity or checksum
>> here it's not necessary - scrub is for it. What we want in most cases
>> is to replace drive as fast as possible.
MA> In some cases it would be faster but others not. For example, if the
MA> pool is not very full, it would be slower. Also if the disk you're
MA> replacing is not available, a straight disk-to-disk copy would not be
MA> possible.
bash-3.00# iostat -xnz 1|egrep "c7t7d0|c4t1d0"
0.0 0.3 0.0 1.3 0.0 0.0 2.9 0.2 0 0 c4t1d0
5.4 8.9 45.3 41.0 0.0 0.0 0.9 1.4 0 2 c7t7d0
0.0 784.2 0.0 1000.6 0.1 0.1 0.1 0.1 4 11 c4t1d0
0.0 780.2 0.0 1000.1 0.1 0.1 0.1 0.2 5 12 c7t7d0
0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 6 14 c4t1d0
0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 7 15 c7t7d0
0.0 811.1 0.0 1175.2 0.1 0.1 0.1 0.1 4 12 c4t1d0
0.0 815.1 0.0 1175.1 0.1 0.1 0.1 0.1 5 12 c7t7d0
0.0 849.3 0.0 3577.6 0.3 0.2 0.4 0.2 9 15 c4t1d0
0.0 846.3 0.0 3577.6 0.2 0.1 0.3 0.2 8 15 c7t7d0
0.0 1029.0 0.0 1630.5 0.8 0.3 0.8 0.2 19 26 c4t1d0
0.0 1027.0 0.0 1630.5 0.9 0.2 0.9 0.2 16 23 c7t7d0
0.0 1009.1 0.0 532.5 0.2 0.2 0.2 0.2 8 17 c4t1d0
0.0 1024.0 0.0 532.5 0.1 0.2 0.1 0.2 7 17 c7t7d0
0.0 1132.7 0.0 2599.6 0.2 0.2 0.1 0.2 8 18 c4t1d0
0.0 1137.6 0.0 2599.8 0.2 0.2 0.1 0.2 9 17 c7t7d0
0.0 1266.3 0.0 1730.5 0.4 0.2 0.3 0.2 10 19 c4t1d0
0.0 1285.5 0.0 1730.3 0.2 0.2 0.2 0.2 9 20 c7t7d0
0.0 963.8 0.0 5626.8 1.3 0.2 1.3 0.2 14 21 c4t1d0
0.0 963.8 0.0 5626.9 0.7 0.2 0.8 0.2 13 21 c7t7d0
0.0 1023.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c4t1d0
0.0 1025.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c7t7d0
^C
bash-3.00#
Looks like data are being written with about 1-5MB/s on c4t1d0 - which
is very slow. Now the system is not doing anything else other than
replacing disk. When I put actual production workload I get 2-3 to
resynchronize single disk with such a config.
>> On another thumper I have a failing drive (port resets, etc.) so I
>> issued over a week ago drive replacement. Well it still hasn't
>> completed even 4% in a week! The pool config is the same. It's just
>> waaaay to slow and in a long term risky.
MA> Are you taking snapshots? They cause scrubbing / resilvering to restart
MA> (this is bug 6343667).
At first I did and I haven't noticed.
All above data are without taking/deleteing any snapshots or doing
anything else.
--
Best regards,
Robert mailto:[EMAIL PROTECTED]
http://milek.blogspot.com
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
