Is the system otherwise responsive during the zfs sync cycles?
I ask because I think I'm seeing a similar thing - except that it's not
only other writers that block , it seems like other interrupts are
blocked. Pinging my zfs server in 1s intervals results in large delays
while the system syncs, followed by normal response times while the
system buffers more input...
Thanks,
Tristan.
Bob Friesenhahn wrote:
It has been quite some time (about a year) since I did testing of
batch processing with my software (GraphicsMagick). In between time,
ZFS added write-throttling. I am using Solaris 10 with kernel 141415-03.
Quite a while back I complained that ZFS was periodically stalling the
writing process (which UFS did not do). The ZFS write-throttling
feature was supposed to avoid that. In my testing today I am still
seeing ZFS stall the writing process periodically. When the process
is stalled, there is a burst of disk activity, a burst of context
switching, and total CPU use drops to almost zero. Zpool iostat says
that read bandwidth is 15.8M and write bandwidth is 15.8M over a 60
second averaging interval. Since my drive array is good for writing
over 250MB/second, this is a very small write load and the array is
loafing.
My program uses the simple read->process->write approach. Each file
written (about 8MB/file) is written contiguously and written just
once. Data is read and written in 128K blocks. For this application
there is no value obtained by caching the file just written. From
what I am seeing, reading occurs as needed, but writes are being
batched up until the next ZFS synchronization cycle. During the ZFS
synchronization cycle it seems that processes are blocked from
writing. Since my system has a lot of memory and the ARC is capped at
10GB, quite a lot of data can be queued up to be written. The ARC is
currently running at its limit of 10GB.
If I tell my software to invoke fsync() before closing each written
file, then the stall goes away, but the program then needs to block so
there is less beneficial use of the CPU.
If this application stall annoys me, I am sure that it would really
annoy a user with mission-critical work which needs to get done on a
uniform basis.
If I run this little script then the application runs more smoothly
but I see evidence of many shorter stalls:
while true
do
sleep 3
sync
done
Is there a solution in the works for this problem?
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us,
http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
