Re: [zfs-discuss] ZFS write I/O stalls
> On Wed, 24 Jun 2009, Richard Elling wrote: > >> > >> "The new code keeps track of the amount of data > accepted in a TXG and the > >> time it takes to sync. It dynamically adjusts that > amount so that each TXG > >> sync takes about 5 seconds (txg_time variable). It > also clamps the limit to > >> no more than 1/8th of physical memory." > > > > hmmm... methinks there is a chance that the 1/8th > rule might not work so well > > for machines with lots of RAM and slow I/O. I'm > also reasonably sure that > > that sort of machine is not what Sun would > typically build for performance > > lab > > testing, as a rule. Hopefully Roch will comment > when it is morning in > > Europe. > > Slow I/O is relative. If I install more memory does > that make my I/O > even slower? > > I did some more testing. I put the input data on a > different drive > and sent application output to the ZFS pool. I no > longer noticed any > stalls in the execution even though the large ZFS > flushes are taking > place. This proves that my application is seeing > stalled reads rather > than stalled writes. There is a bug in the database about reads blocked by writes which may be related: http://bugs.opensolaris.org/view_bug.do?bug_id=6471212 The symptom is sometimes reducing queue depth makes read perform better. > > Bob > -- > Bob Friesenhahn > bfrie...@simple.dallas.tx.us, > http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, >http://www.GraphicsMagick.org/ > > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS write I/O stalls
> On Wed, 24 Jun 2009, Lejun Zhu wrote: > > > There is a bug in the database about reads blocked > by writes which may be related: > > > > > http://bugs.opensolaris.org/view_bug.do?bug_id=6471212 > > > > The symptom is sometimes reducing queue depth makes > read perform better. > > I have been banging away at this issue without > resolution. Based on > Roch Bourbonnais's blog description of the ZFS write > throttle code, it > seems that I am facing a perfect storm. Both the > storage write > bandwidth (800+ MB/second) and the memory size of my > system (20 GB) > result in the algorithm batching up 2.5 GB of user > data to write. With ZFS write throttle, the number 2.5GB is tunable. From what I've read in the code, it is possible to e.g. set zfs:zfs_write_limit_override = 0x800 (bytes) to make it write 128M instead. > Since I am using mirrors, this results in 5 GB of > data being written > at full speed to the array on a very precise schedule > since my > application is processing fixed-sized files with a > fixed algorithm. > The huge writes lead to at least 3 seconds of read > starvation, > resulting in a stalled application and a square-wave > of system CPU > utilization. I could attempt to modify my > application to read ahead > by 3 seconds but that would require gigabytes of > memory, lots of > complexity, and would not be efficient. > > Richard Elling thinks that my array is pokey, but > based on write speed > and memory size, ZFS is always going to be batching > up data to fill > the write channel for 5 seconds so it does not really > matter how fast > that write channel is. If I had 32GB of RAM and 2X > the write speed, > the situation would be identical. > > Hopefully someone at Sun is indeed working this read > starvation issue > and it will be resolved soon. > > Bob > -- > Bob Friesenhahn > bfrie...@simple.dallas.tx.us, > http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, >http://www.GraphicsMagick.org/ > > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?
If cpu seems to be idle, the tool latencytop probably can give you some clue. It's developed for OpenSolaris but Solaris 10 should work too (with glib 2.14 installed). You can get a copy of v0.1 at http://opensolaris.org/os/project/latencytop/ To use latencytop, open a terminal and start "latencytop -s -k 2". The tool will show a window with activities that are being blocked in the system. Then you can launch your application to reproduce the performance problem in another terminal, switch back to latencytop window, and use "<" and ">" to find your process. The list will tell you which function is causing the delay. After a couple minutes you may press "q" to exit from latencytop. When it ends, a log file /var/log/latencytop.log will be created. It includes the stack trace of waiting for IO, semaphore etc. when latencytop was running. If you post the log here, I can probably extract a list of worst delays in ZFS source code, and other experts may comment. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Migrating 10TB of data from NTFS is there a simple way?
> Ok so this is my solution, pls be advised I am a > total linux nube so I am learning as I go along. I > installed opensolaris and setup rpool as my base > install on a single 1TB drive. I attached one of my > NTFS drives to the system then used a utility called > prtparts to get the name of the NTFS drive attached > and then mounted it succesfully. > I then started transfering data accross till the > drive was empty (this is currently in progress) Once > thats done I will add the empty NTFS drive to my ZFS > pool and repeat the operation with my other drives. > > This leaves me with the issue of redundancy which is > sorely lacking, ideally I would like to do the same > think directly into a zraid pool, but I understand > from what I have read that you cant add single drives > to a zraid and I want all my drives in a single pool > as only to loose the space for the pool redundancy > once. > > I havent worked out if I can transform my zpool int a > zraid after I have copied all my data. > > Once again thx for the great support. And maybe > someone can direct me to an area in a forum that > explains y I cant use sudo... Hope this helps http://forums.opensolaris.com/thread.jspa?threadID=583&tstart=-1 -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
