eric kustarz wrote:
So i'm guessing there's lots of files being created over NFS in one
particular dataset?
We should figure out how many creates/second you are doing over NFS (i
should have put a timeout on the script). Here's a real simple one
(from your snoop it looked like you're only doing NFSv3, so i'm not
tracking NFSv4):
"
#!/usr/sbin/dtrace -s
rfs3_create:entry,
zfs_create:entry
{
@creates[probefunc] = count();
}
tick-60s
{
exit(0);
}
"
Eric, I love you.
Running this bit of DTrace reveled more than 4,000 files being created
in almost any given 60 second window. And I've only got one system that
would fit that sort of mass file creation: our Joyent Connector products
Courier IMAP server which uses Maildir. As a test I simply shutdown
Courier and unmounted the mail NFS share for good measure and sure
enough the problem vanished and could not be reproduced. 10 minutes
later I re-enabled Courier and our problem came back.
Clearly ZFS file creation is just amazingly heavy even with ZIL
disabled. If creating 4,000 files in a minute squashes 4 2.6Ghz Opteron
cores we're in big trouble in the longer term. In the meantime I'm
going to find a new home for our IMAP Mail so that the other things
served from that NFS server at least aren't effected.
You asked for the zpool and zfs info, which I don't want to share
because its confidential (if you want it privately I'll do so, but not
on a public list), but I will say that its a single massive Zpool in
which we're using less than 2% of the capacity. But in thinking about
this problem, even if we used 2 or more pools, the CPU consumption still
would have choked the system, right? This leaves me really nervous
about what we'll do when its not an internal mail server thats creating
all those files but a customer.
Oddly enough, this might be a very good reason to use iSCSI instead of
NFS on the Thumper.
Eric, I owe you a couple cases of beer for sure. I can't tell you how
much I appreciate your help. Thanks to everyone else who chimed in with
ideas and suggestions, all of you guys are the best!
benr.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
