While looking over iostats from various programs, I see that
my OS HDD is busy writing, about 2Mb/sec stream all the time
(at least while the "dcpool" import/recovery attempts are
underway, but also now during a mere zdb walk).
According to "iostat" this load stands out greatly:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
25.0 0.0 100.0 0.0 0.0 0.3 0.0 11.6 0 29 c7t0d0
10.0 0.0 40.0 0.0 0.0 0.1 0.0 8.4 0 8 c7t1d0
2.0 0.0 8.0 0.0 0.0 0.0 0.0 13.6 0 3 c7t2d0
32.0 0.0 188.0 0.0 0.0 0.3 0.0 9.8 0 31 c7t3d0
14.0 0.0 116.0 0.0 0.0 0.1 0.0 10.3 0 14 c7t4d0
2.0 0.0 8.0 0.0 0.0 0.0 0.0 19.0 0 4 c7t5d0
0.0 327.0 0.0 2947.6 0.2 0.1 0.5 0.2 5 5 c4t1d0
59.0 0.0 125.5 0.0 0.0 0.7 0.0 12.4 0 73
c0t600144F09844CF0000004D8376AE0002d0
"zpool iostat" confirmed it is rpool and not some other
partition:
---------- ----- ----- ----- ----- ----- -----
rpool 17.1G 2.77G 0 271 1.08K 2.20M
c4t1d0s0 17.1G 2.77G 0 271 1.08K 2.20M
---------- ----- ----- ----- ----- ----- -----
For a while I thought it might be some swapping IO (despite
the fact that vmstat and top show no swap-area usage at the
moment, and no PI/PO operations in vmstat). I disabled swap
in rpool volumes, but the 2Mb/s stream is still there, both
with TXG sync times bumped to 30 sec and reduced to 1 sec.
So far I did not find a DTraceToolkit-0.99 utility which
would show me what that would be:
# /export/home/jim/DTraceToolkit-0.99/rwsnoop | egrep -v '/proc|/dev|<unkn'
UID PID CMD D BYTES FILE
0 1251 freeram-watc W 78
/var/log/freeram-watchdog.log.1307796483
0 1251 freeram-watc W 78
/var/log/freeram-watchdog.log.1307796483
0 1251 freeram-watc W 78
/var/log/freeram-watchdog.log.1307796483
0 394 nscd R 13492 /etc/security/prof_attr
0 1251 freeram-watc W 78
/var/log/freeram-watchdog.log.1307796483
0 1251 freeram-watc W 78
/var/log/freeram-watchdog.log.1307796483
0 677 utmpd R 4 /var/adm/wtmpx
0 1251 freeram-watc W 156
/var/log/freeram-watchdog.log.1307796483
These lines appear about once per second, so I gather that
file-based IO is not heavy. There are many "<unknown>"
entries which seem to be associated with "grep" and other
pipes I have running, but they shouldn't go through disk?
"iosnoop" also shows little detail, except also pointing
that all writes on the system at this moment go to rpool:
0 6 W 32081558 16384 zpool-rpool <none>
0 6 W 32092916 16384 zpool-rpool <none>
0 6 W 32116495 16384 zpool-rpool <none>
0 6 W 32117832 16384 zpool-rpool <none>
0 6 W 32125173 16384 zpool-rpool <none>
0 0 R 3025883400 4096 sched <none>
0 6 W 32156861 16384 zpool-rpool <none>
0 6 W 32206221 8192 zpool-rpool <none>
0 6 W 16701011 16384 zpool-rpool <none>
0 6 W 16702547 4096 zpool-rpool <none>
0 6 W 5281714 16384 zpool-rpool <none>
0 6 W 5462106 4096 zpool-rpool <none>
0 6 W 5251672 16384 zpool-rpool <none>
0 6 W 5253790 4096 zpool-rpool <none>
0 6 W 5257100 16384 zpool-rpool <none>
0 6 W 5408779 16384 zpool-rpool <none>
0 6 W 5431113 16384 zpool-rpool <none>
0 6 W 5433800 16384 zpool-rpool <none>
0 6 W 5438181 16384 zpool-rpool <none>
0 6 W 5447201 16384 zpool-rpool <none>
0 6 W 5462114 4096 zpool-rpool <none>
0 6 W 5503260 2048 zpool-rpool <none>
0 6 W 5510618 16384 zpool-rpool <none>
0 6 W 16633532 2048 zpool-rpool <none>
0 6 W 16640398 16384 zpool-rpool <none>
0 6 W 16648096 16384 zpool-rpool <none>
0 6 W 16650717 16384 zpool-rpool <none>
0 6 W 16651864 16384 zpool-rpool <none>
0 6 W 16658841 16384 zpool-rpool <none>
0 6 W 16658883 16384 zpool-rpool <none>
0 6 W 16662945 16384 zpool-rpool <none>
I have little idea which dataset it could be landing on,
or which process/task generates such a stream?
As I wrote above, the system is busy trying to import
or zdbwalk "dcpool" which resides in a volume on another
separate "pool", system is otherwise idle, and all other
processes which write to files account to a few bytes
per second on average...
I had a hunch this may be related to having dedup(=verify)
on the root pool as well, but disabling it and waiting
about 5 minutes did not change "iostat" substantially.
True, at some seconds the writes went down to about
80 write IOPS ~ 300-500k/sec, but they went back up to
200-300 wIOPS ~ 2-3Mb/sec just afterwards.
So I'm still wondering... what is being written at such
rate and does not deplete my pool's free space? ;)
Thanks for hints,
//Jim
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
