Hello eric,
Monday, February 12, 2007, 7:08:20 PM, you wrote:
ek> On Feb 12, 2007, at 7:52 AM, Robert Milkowski wrote:
>> Hello Roch,
>>
>> Monday, February 12, 2007, 3:54:30 PM, you wrote:
>>
>> RP> Duh!.
>>
>> RP> Long sync (which delays the next sync) are also possible on
>> RP> a write intensive workloads. Throttling heavy writters, I
>> RP> think, is the key to fixing this.
>>
>> Well, then maybe it's not the cause to our problems.
>> Nevertheless 60-90s for unlink() is just plain wrong especially when
>> you've got <10ms IOs to array, almost zero writes, plenty of CPU free,
>> etc.
>>
>> Definitely something is wrong here.
ek> Looks like spa_sync() via the txg_sync_thread thread is taking way
ek> too long, which is causing new (NFS) requests to be delayed (such as
ek> unlink).
ek> Is this just a NFS server, or is there local activity as well?
No local activity, no other applications - just NFS server.
ek> A complete threadlist would be interesting, as would memory usage.
System is not paging at all and free memory reported by vmstat stays
at about 1GB.
All processes in a sysytem:
bash-3.00# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 0 0 0 Feb 09 ? 0:44 sched
root 1 0 0 Feb 09 ? 0:12 /sbin/init -v
root 2 0 0 Feb 09 ? 0:00 pageout
root 3 0 0 Feb 09 ? 42:09 fsflush
root 245 1 0 Feb 09 ? 0:00 /usr/sbin/mdmonitord
root 7 1 0 Feb 09 ? 0:18 /lib/svc/bin/svc.startd
root 9 1 0 Feb 09 ? 0:49 /lib/svc/bin/svc.configd
daemon 296 1 0 Feb 09 ? 14:20 /usr/lib/nfs/lockd
daemon 283 1 0 Feb 09 ? 0:00 /usr/lib/nfs/statd
root 403 1 0 Feb 09 ? 0:00 /usr/sbin/nsrexecd
root 286 7 0 Feb 09 ? 0:00 /usr/lib/saf/sac -t 300
daemon 282 1 0 Feb 09 ? 0:02 /usr/lib/nfs/nfsmapid
root 159 1 0 Feb 09 ? 0:46 /usr/lib/picl/picld
root 161 1 0 Feb 09 ? 0:30 /usr/sbin/nscd
root 24699 380 0 16:41:19 ? 0:00 /usr/lib/ssh/sshd
daemon 278 1 0 Feb 09 ? 0:17 /usr/sbin/rpcbind
daemon 173 1 0 Feb 09 ? 0:01 /usr/lib/crypto/kcfd
root 156 1 0 Feb 09 ? 0:00 /usr/lib/sysevent/syseventd
root 295 1 0 Feb 09 ? 0:01 /usr/lib/utmpd
root 233 1 0 Feb 09 ? 0:42 /usr/lib/inet/xntpd
root 274 1 0 Feb 09 ? 0:03 /usr/sbin/cron
root 328 1 0 Feb 09 ? 0:01 /usr/sbin/syslogd
root 404 403 0 Feb 09 ? 0:01 /usr/sbin/nsrexecd
root 187 1 0 Feb 09 ? 3:14 /usr/lib/inet/in.mpathd -a
root 179 1 0 Feb 09 ? 0:00 devfsadmd
root 298 1 0 Feb 09 ? 0:07 /usr/lib/inet/inetd start
root 299 7 0 Feb 09 console 0:00 /usr/lib/saf/ttymon -g -d
/dev/console -l console -T sun -m ldterm,ttcompat -h
root 301 286 0 Feb 09 ? 0:00 /usr/lib/saf/ttymon
root 378 1 0 Feb 09 ? 0:07 /usr/lib/fm/fmd/fmd
root 380 1 0 Feb 09 ? 0:00 /usr/lib/ssh/sshd
root 588 1 0 Feb 09 ? 23:46
/usr/jdk/jdk1.5.0_09/bin/java -Xms4M -Xmx64M -classpath /usr/share/lib/jdmk/jdm
root 801 1 0 Feb 09 ? 0:01 /usr/lib/nfs/mountd
root 24702 24699 0 16:41:20 ? 0:00 /usr/lib/ssh/sshd
root 19793 1 0 Feb 10 ? 4:36 /usr/sfw/sbin/snmpd
root 8598 380 0 14:49:02 ? 0:00 /usr/lib/ssh/sshd
root 22686 22680 0 Feb 12 pts/1 0:00 bash
root 22678 22671 0 Feb 12 ? 0:02 /usr/lib/ssh/sshd
root 24704 24702 0 16:41:20 pts/2 0:00 -bash
root 8638 8636 0 14:49:05 pts/3 0:00 -sh
root 8647 8645 0 14:49:11 pts/3 0:00 ps -ef
root 8645 8638 0 14:49:06 pts/3 0:00 bash
root 22680 22678 0 Feb 12 pts/1 0:00 -sh
root 8636 8598 0 14:49:05 ? 0:00 /usr/lib/ssh/sshd
root 22671 380 0 Feb 12 ? 0:00 /usr/lib/ssh/sshd
daemon 28297 1 0 22:17:32 ? 173:25 /usr/lib/nfs/nfsd
bash-3.00#
ek> Have you increased the load on this machine? I have seen a similar
ek> situation (new requests being blocked waiting for the sync thread to
ek> finish), but that's only been when either 1) the hardware is broken
ek> and taking too long or 2) the server is way overloaded.
I don't think HW is broken - the same situation on two v440 servers
and T2000 server. iostat doesn't show any problems accessing disks (no
queues, short service times, etc.).
We moved workload from v440 with 8GB of RAM to T2000 with 32GB of RAM
and for many hours it was working just great. We did try to stop nfsd
on T2000 and it exited within a second or two - looked almost like it
was working great. But then next day (today) we've started
experiencing the same problems - long IOs (dozen of seconds) so we
decided to spot nfsd - this time it took over 20 minutes for all
threads to complete. Then we did 'zpool export f3-2' and it took 59
minutes to complete!! See other thread here I've just started
("[zfs-discuss] zpool export consumes whole CPU and takes more than 30
minutes to complete").
Looks like for some reason ZFS isn't able to complete all writes to
disks. More memory just delayed the problem and zil_disable set to 1
mitigates the problem for some time until zfs has filled up all memory
and has to wait for data being written to disk and then nfs operations
starts to take 30-90s, sometimes even much more. Then you've got
problem with stopping nfsd, or exporting a pool (different thing is
why during export entire one cpu is consumed by zfs which is the
limiting factor).
The same on S10U3 and snv_54.
Something is really broken here.
Maybe the hardware...
--
Best regards,
Robert mailto:[EMAIL PROTECTED]
http://milek.blogspot.com
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
