Red herring...
Actually, I had compression=gzip-9 enabled on that filesystem, which is
apparently too much for the old xeon's in that server (it's a Dell
1850). The CPU was sitting at 100% kernel time while it tried to
compress + sync.
Switching to compression=off or compression=on (lzjb) makes the problem
go away.
Interestingly, creating a second processor set also alleviates many of
the symptoms - certainly the slow ping goes away. Assigning the ssh +
shell session I had on the machine while running these to the second set
restores responsiveness to that too, it appears that all the compression
happens in set 0.
Regards,
Tristan.
Tristan Ball wrote:
Is the system otherwise responsive during the zfs sync cycles?
I ask because I think I'm seeing a similar thing - except that it's
not only other writers that block , it seems like other interrupts are
blocked. Pinging my zfs server in 1s intervals results in large delays
while the system syncs, followed by normal response times while the
system buffers more input...
Thanks,
Tristan.
Bob Friesenhahn wrote:
It has been quite some time (about a year) since I did testing of
batch processing with my software (GraphicsMagick). In between time,
ZFS added write-throttling. I am using Solaris 10 with kernel
141415-03.
Quite a while back I complained that ZFS was periodically stalling
the writing process (which UFS did not do). The ZFS write-throttling
feature was supposed to avoid that. In my testing today I am still
seeing ZFS stall the writing process periodically. When the process
is stalled, there is a burst of disk activity, a burst of context
switching, and total CPU use drops to almost zero. Zpool iostat says
that read bandwidth is 15.8M and write bandwidth is 15.8M over a 60
second averaging interval. Since my drive array is good for writing
over 250MB/second, this is a very small write load and the array is
loafing.
My program uses the simple read->process->write approach. Each file
written (about 8MB/file) is written contiguously and written just
once. Data is read and written in 128K blocks. For this application
there is no value obtained by caching the file just written. From
what I am seeing, reading occurs as needed, but writes are being
batched up until the next ZFS synchronization cycle. During the ZFS
synchronization cycle it seems that processes are blocked from
writing. Since my system has a lot of memory and the ARC is capped at
10GB, quite a lot of data can be queued up to be written. The ARC is
currently running at its limit of 10GB.
If I tell my software to invoke fsync() before closing each written
file, then the stall goes away, but the program then needs to block
so there is less beneficial use of the CPU.
If this application stall annoys me, I am sure that it would really
annoy a user with mission-critical work which needs to get done on a
uniform basis.
If I run this little script then the application runs more smoothly
but I see evidence of many shorter stalls:
while true
do
sleep 3
sync
done
Is there a solution in the works for this problem?
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us,
http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
