Carsten Aulbert wrote:
> Put some stress on the system with bonnie and other tools and try to
> find slow disks and see if this could be the main problem but also look
> into more vdevs and then possible move to raidz to somehow compensate
> for lost disk space. Since we have 4 cold spares on the shelf plus a SMS
> warnings on disk failures (that is if fma catches them) the risk
> involved should be tolerable.
First result with bonnie during the "writing intelligently..." phase I
see this in a 2 minute average:
zpool iostats:
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
atlashome 1.70T 19.2T 225 1.49K 342K 107M
raidz2 550G 6.28T 74 409 114K 32.6M
c0t0d0 - - 0 314 32.3K 2.51M
c1t0d0 - - 0 315 31.8K 2.52M
c4t0d0 - - 0 313 31.3K 2.52M
c6t0d0 - - 0 315 32.3K 2.51M
c7t0d0 - - 0 326 32.8K 2.50M
c0t1d0 - - 0 309 33.9K 2.52M
c1t1d0 - - 0 313 33.4K 2.51M
c4t1d0 - - 0 314 33.4K 2.52M
c5t1d0 - - 0 308 32.8K 2.52M
c6t1d0 - - 0 314 31.3K 2.51M
c7t1d0 - - 0 311 31.8K 2.52M
c0t2d0 - - 0 309 31.8K 2.52M
c1t2d0 - - 0 313 31.8K 2.51M
c4t2d0 - - 0 315 31.8K 2.52M
c5t2d0 - - 0 307 32.8K 2.52M
raidz2 567G 6.26T 64 529 96.5K 36.3M
c6t2d0 - - 1 368 74.2K 2.79M
c7t2d0 - - 1 366 74.2K 2.80M
c0t3d0 - - 1 364 75.8K 2.80M
c1t3d0 - - 1 365 75.2K 2.80M
c4t3d0 - - 1 368 76.8K 2.80M
c5t3d0 - - 1 362 76.3K 2.80M
c6t3d0 - - 1 366 77.9K 2.80M
c7t3d0 - - 1 365 76.8K 2.80M
c0t4d0 - - 1 361 76.8K 2.80M
c1t4d0 - - 1 363 75.8K 2.80M
c4t4d0 - - 1 366 76.3K 2.80M
c6t4d0 - - 1 364 78.4K 2.80M
c7t4d0 - - 1 370 78.9K 2.79M
c0t5d0 - - 1 365 77.3K 2.80M
c1t5d0 - - 1 364 74.7K 2.80M
raidz2 620G 6.64T 86 582 131K 37.9M
c4t5d0 - - 18 382 1.16M 2.74M
c5t5d0 - - 10 380 674K 2.74M
c6t5d0 - - 18 378 1.15M 2.73M
c7t5d0 - - 9 384 628K 2.74M
c0t6d0 - - 18 377 1.16M 2.74M
c1t6d0 - - 10 383 680K 2.75M
c4t6d0 - - 19 379 1.21M 2.73M
c5t6d0 - - 10 383 691K 2.75M
c6t6d0 - - 19 379 1.21M 2.73M
c7t6d0 - - 10 383 676K 2.72M
c0t7d0 - - 18 374 1.19M 2.75M
c1t7d0 - - 10 381 676K 2.74M
c4t7d0 - - 19 380 1.22M 2.74M
c5t7d0 - - 10 382 696K 2.74M
c6t7d0 - - 18 381 1.17M 2.74M
c7t7d0 - - 9 386 631K 2.75M
---------- ----- ----- ----- ----- ----- -----
iostat -Mnx 120:
extended device statistics
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d0
0.0 1.4 0.0 0.0 0.0 0.0 1.5 0.4 0 0 c5t0d0
0.6 351.5 0.0 2.6 0.4 0.1 1.2 0.2 3 8 c7t0d0
0.6 336.3 0.0 2.6 0.1 0.1 0.4 0.2 3 7 c0t0d0
0.6 340.8 0.0 2.6 0.2 0.1 0.6 0.2 3 7 c1t0d0
0.6 330.6 0.0 2.6 0.1 0.1 0.3 0.2 3 7 c5t1d0
0.6 336.7 0.0 2.6 0.1 0.1 0.3 0.2 3 7 c4t0d0
0.6 331.8 0.0 2.6 0.1 0.1 0.3 0.2 3 7 c0t1d0
0.6 339.0 0.0 2.6 0.4 0.1 1.1 0.2 3 7 c7t1d0
0.6 335.4 0.0 2.6 0.1 0.1 0.4 0.2 3 7 c1t1d0
0.6 329.2 0.0 2.6 0.1 0.1 0.3 0.2 3 7 c5t2d0
0.6 343.7 0.0 2.6 0.3 0.1 0.7 0.2 3 7 c4t1d0
0.6 331.8 0.0 2.6 0.1 0.1 0.3 0.2 2 7 c0t2d0
1.2 396.3 0.1 2.9 0.3 0.1 0.7 0.2 4 8 c7t2d0
0.6 336.7 0.0 2.6 0.1 0.1 0.4 0.2 3 7 c1t2d0
0.6 341.9 0.0 2.6 0.2 0.1 0.7 0.2 3 7 c4t2d0
1.3 390.7 0.1 2.9 0.3 0.1 0.8 0.2 4 9 c5t3d0
1.3 396.7 0.1 2.9 0.3 0.1 0.8 0.2 4 9 c7t3d0
1.3 393.6 0.1 2.9 0.2 0.1 0.6 0.2 4 9 c0t3d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c5t4d0
1.3 396.2 0.1 2.9 0.2 0.1 0.5 0.2 4 8 c1t3d0
1.3 399.2 0.1 2.9 0.3 0.1 0.8 0.2 4 9 c4t3d0
1.3 401.8 0.1 2.9 0.3 0.1 0.8 0.2 4 9 c7t4d0
1.3 388.5 0.1 2.9 0.2 0.1 0.5 0.2 4 8 c0t4d0
1.3 391.8 0.1 2.9 0.2 0.1 0.5 0.2 4 9 c1t4d0
1.3 395.1 0.1 2.9 0.2 0.1 0.6 0.2 4 8 c4t4d0
9.9 409.7 0.6 2.9 0.8 0.2 1.9 0.4 10 18 c7t5d0
1.3 395.0 0.1 2.9 0.3 0.1 0.6 0.2 4 9 c0t5d0
10.6 405.3 0.7 2.9 0.8 0.2 2.0 0.4 11 18 c5t5d0
1.3 392.8 0.1 2.9 0.2 0.1 0.5 0.2 4 8 c1t5d0
10.7 407.6 0.7 2.9 0.9 0.2 2.1 0.4 11 19 c7t6d0
18.6 407.5 1.2 2.9 1.0 0.2 2.4 0.6 15 24 c4t5d0
10.9 407.8 0.7 2.9 0.8 0.2 2.0 0.4 11 19 c5t6d0
0.6 337.6 0.0 2.6 0.2 0.1 0.5 0.2 3 7 c6t0d0
10.7 408.8 0.7 2.9 0.8 0.2 1.9 0.4 11 19 c1t6d0
10.0 411.6 0.6 2.9 0.8 0.2 1.8 0.4 11 18 c7t7d0
19.3 403.1 1.2 2.9 1.1 0.3 2.6 0.6 16 26 c4t6d0
0.6 336.2 0.0 2.6 0.1 0.1 0.4 0.2 3 7 c6t1d0
11.0 407.7 0.7 2.9 0.8 0.2 1.9 0.4 11 19 c5t7d0
10.6 406.6 0.7 2.9 0.8 0.2 2.0 0.4 11 19 c1t7d0
18.5 401.7 1.2 2.9 1.0 0.2 2.5 0.6 15 25 c0t6d0
19.4 404.8 1.2 2.9 1.0 0.3 2.5 0.6 15 25 c4t7d0
1.2 397.6 0.1 2.9 0.3 0.1 0.9 0.2 4 9 c6t2d0
19.0 398.7 1.2 2.9 1.0 0.3 2.5 0.6 15 25 c0t7d0
1.3 396.1 0.1 2.9 0.2 0.1 0.5 0.2 4 8 c6t3d0
1.3 392.8 0.1 2.9 0.2 0.1 0.4 0.2 4 8 c6t4d0
18.4 403.3 1.2 2.9 1.1 0.2 2.5 0.6 15 24 c6t5d0
19.3 402.7 1.2 2.9 1.1 0.3 2.5 0.6 15 25 c6t6d0
18.8 406.1 1.2 2.9 1.0 0.2 2.4 0.6 15 25 c6t7d0
Any experts here to say if that's just because bonnie via NFSv3 is a
very special test - if it is I can start something else, suggestions? -
or if some disks are really too busy and slowing down the pool.
Thanks for more insight
Carsten
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
