ek> Have you increased the load on this machine? I have seen a
similar
ek> situation (new requests being blocked waiting for the sync
thread to
ek> finish), but that's only been when either 1) the hardware is
broken
ek> and taking too long or 2) the server is way overloaded.
I don't think HW is broken - the same situation on two v440 servers
and T2000 server. iostat doesn't show any problems accessing disks (no
queues, short service times, etc.).
We moved workload from v440 with 8GB of RAM to T2000 with 32GB of RAM
and for many hours it was working just great. We did try to stop nfsd
on T2000 and it exited within a second or two - looked almost like it
was working great. But then next day (today) we've started
experiencing the same problems - long IOs (dozen of seconds) so we
decided to spot nfsd - this time it took over 20 minutes for all
threads to complete. Then we did 'zpool export f3-2' and it took 59
minutes to complete!! See other thread here I've just started
("[zfs-discuss] zpool export consumes whole CPU and takes more than 30
minutes to complete").
Looks like for some reason ZFS isn't able to complete all writes to
disks. More memory just delayed the problem and zil_disable set to 1
mitigates the problem for some time until zfs has filled up all memory
and has to wait for data being written to disk and then nfs operations
starts to take 30-90s, sometimes even much more. Then you've got
problem with stopping nfsd, or exporting a pool (different thing is
why during export entire one cpu is consumed by zfs which is the
limiting factor).
The same on S10U3 and snv_54.
So dtracing metaslab_ff_alloc() would be a good way to know if you're
hitting:
6495013 Loops and recursion in metaslab_ff_alloc can kill
performance, even on a pool with lots of free data
A dscript mentioned in the bug report (i believe the non-public part)
is:
"
#!/usr/sbin/dtrace -s
#pragma D option quiet
BEGIN
{
self->in_metaslab = 0;
}
fbt::metaslab_ff_alloc:entry
/self->in_metaslab == 0/
{
self->in_metaslab = 1;
self->loopcount = 0;
}
fbt::avl_walk:entry
/self->in_metaslab/
{
self->loopcount++;
}
fbt::metaslab_ff_alloc:return
/self->in_metaslab/
{
self->in_metaslab = 0;
@loops["Loops count"] = quantize(self->loopcount);
self->loopcount = 0;
}
"
Now, note, this dscript isn't perfect as it doesn't take into
recursion, but feel free to tweak it if/as you like. If you're
seeing lots of avl_walk() calls per metaslab_ff_alloc() call then its
the above bug.
eric
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
