0.1 seconds per I/O is because of the queue time. No matter how slow it is, the data read out will be stored in cache, right? and after it is cached, the next time read it should be faster. The problem is we periodically read the same control file, some time is slow, some time is fast. I am hoping solaris could use all the free memory effectively, so that it can cache all the data unless there is not enough free memory.
Thanks, On Mon, Apr 12, 2010 at 11:20 AM, Richard Elling <richard.ell...@gmail.com>wrote: > more below... > > On Apr 11, 2010, at 6:14 PM, Qihua Wu wrote: > > > Thank Richard, > > > > We used netapp as standby database, the write performance looks like good > as the recovery can catch up(might because the write anywhere feature is > optimized to write). But read is very slow, not matter which datafile. But > we mainly care the read performance about one file: the control file, > because we need to read it periodically to know the status of the database. > For other files, they are only needed to be read when we need to copy the > files back to primary database if primary database failed, but in that case > we can stop the recover to reduce the disk contention. > > > > I know there are long queue so the read from disk should be slow, what I > can't understand is that why solaris doesn't cache the data in file system > cache? As show in the post, the free memory is about 25G, if solaris uses > all of them to cache the data, then we don't have so many IO request going > to the netapp filer as it can be got direct from local file system cache. > > The cache is not your problem. Your problem is that it takes, on average, > more than 0.1 seconds per I/O. Is the NetApp on the other side of the > world? > > > > > TEST7-stdby-phxdbnfs11$> uname -a > > SunOS phxdbnfs11 5.10 Generic_141414-07 sun4v sparc > SUNW,SPARC-Enterprise-T5120 > > > > TEST7-stdby-phxdbnfs11$> cat /etc/vfstab|grep phxdbfiler > > phxdbfiler03-A1:/vol/ora11 - /export/home/oracle nfs 2 yes > rw,bg,vers=3,proto=tcp,hard, > > intr,rsize=65535,wsize=65535 > > These parameters are ok. > > > phxdbfiler03-A1:/vol/com11 - /com nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler03-A1:/vol/test01sys - /oracle/TEST1/sys nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler04-A1:/vol/test01arc - /oracle/TEST1/archive nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler03-A1:/vol/test01data01 - /oracle/TEST1/data01 nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler03-A1:/vol/test02sys - /oracle/TEST2/sys nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler04-A1:/vol/test02arc - /oracle/TEST2/archive nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler03-A1:/vol/test02data01 - /oracle/TEST2/data01 nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler03-A1:/vol/test07sys - /oracle/TEST7/sys nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler04-A1:/vol/test07arc - /oracle/TEST7/archive nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > phxdbfiler03-A1:/vol/test07data01 - /oracle/TEST7/data01 nfs 2 yes > rw,bg,vers=3,proto=tcp,hard,intr,rsize=65535,wsize=65535 > > > > > > > > TEST7-stdby-phxdbnfs11$> sar -d 1 10|grep nfs > > SunOS phxdbnfs11 5.10 Generic_141414-07 sun4v 04/11/2010 > > 18:08:30 nfs23 0 0.0 0 0 0.0 0.0 > > nfs24 100 1398976.5 234 14946 5978533.5 > 102.4 > > Something is seriously wrong here. This data says that an I/O to the > NetApp responds in 102.4 milliseconds. Until you find the solution > to this problem, your NetApp is almost as slow as a floppy disk. > -- richard > > > > nfs25 0 0.0 1 16 0.0 0.4 > > nfs26 0 0.0 1 16 0.0 0.5 > > nfs27 0 0.0 1 16 0.0 0.8 > > nfs28 100 51371.2 344 22044 149073.0 69.5 > > nfs29 100 32211.3 289 18555 111289.8 82.8 > > nfs30 0 0.0 1 16 0.0 0.7 > > nfs31 0 0.0 0 0 0.0 0.0 > > nfs32 0 0.0 1 16 0.0 0.7 > > nfs33 0 0.0 1 16 0.0 0.6 > > 18:08:32 nfs23 0 0.0 0 0 0.0 0.0 > > nfs24 100 1398762.2 112 7185 12459108.5 > 213.6 > > nfs25 0 0.0 0 0 0.0 0.0 > > nfs26 0 0.0 0 0 0.0 0.0 > > nfs27 0 0.0 0 0 0.0 0.0 > > nfs28 100 51190.7 108 6781 475578.0 222.9 > > nfs29 100 32122.8 137 8667 235000.2 175.6 > > nfs30 0 0.0 0 0 0.0 0.0 > > nfs31 0 0.0 0 0 0.0 0.0 > > nfs32 0 0.0 0 0 0.0 0.0 > > nfs33 0 0.0 0 0 0.0 0.0 > > 18:08:33 nfs23 0 0.0 0 0 0.0 0.0 > > nfs24 100 1398617.3 312 19981 4486553.7 > 76.8 > > nfs25 0 0.0 0 0 0.0 0.0 > > nfs26 0 0.0 0 0 0.0 0.0 > > nfs27 0 0.0 0 0 0.0 0.0 > > nfs28 100 51027.9 129 8268 394811.7 185.6 > > nfs29 100 31962.5 170 10649 187460.1 140.7 > > nfs30 0 0.0 0 0 0.0 0.0 > > nfs31 0 0.0 0 0 0.0 0.0 > > nfs32 0 0.0 0 0 0.0 0.0 > > nfs33 0 0.0 0 0 0.0 0.0 > > 18:08:34 nfs23 0 0.0 0 0 0.0 0.0 > > nfs24 100 1398497.6 191 12217 7325920.0 > 125.6 > > nfs25 0 0.0 1 16 0.0 0.5 > > nfs26 0 0.0 1 16 0.0 0.5 > > nfs27 0 0.0 1 16 0.0 0.5 > > nfs28 100 50931.4 266 17008 191558.5 90.2 > > nfs29 100 31875.2 194 12457 164435.0 123.7 > > nfs30 0 0.0 1 16 0.0 0.6 > > nfs31 0 0.0 0 0 0.0 0.0 > > nfs32 0 0.0 1 16 0.0 0.9 > > nfs33 0 0.0 1 16 0.0 1.3 > > > > Thanks, > > > > > > On Sun, Apr 11, 2010 at 11:30 PM, Richard Elling < > richard.ell...@gmail.com> wrote: > > On Apr 9, 2010, at 8:18 PM, Daniel, Wu wrote: > > > We have a NFS data volume, reading data from it is so slow(10 minutes > to copy 512M). But since we have 25G free memory, why solaris 10 doesn't > cache the data in the file system cache? > > > > It does. > > > > > If the free memory is nearly 0, and the read is bad, that means we > reach the limit of the system. But since we have so much free memory, the > system is not optimized, right? What do I need to look into? > > > > iostat will show the latency and queues for I/O to the storage device. > > -- richard > > > > > $> meminfo > > > RAM _____Total 65408.0 Mb > > > RAM Unusable 1425.5 Mb > > > RAM Kernel 3862.5 Mb > > > RAM Locked 33728.0 Mb > > > RAM Used 1174.2 Mb > > > RAM Avail 25217.9 Mb > > > > > > Disk _____Total 49150.9 Mb > > > Disk Alloc 0.0 Mb > > > Disk Free 49150.9 Mb > > > > > > Swap _____Total 102453.1 Mb > > > Swap Alloc 36413.6 Mb > > > Swap Unalloc 186.4 Mb > > > Swap Avail 65853.2 Mb > > > Swap (MinFree) 8027.9 Mb > > > -- > > > This message posted from opensolaris.org > > > _______________________________________________ > > > perf-discuss mailing list > > > perf-discuss@opensolaris.org > > > > ZFS storage and performance consulting at http://www.RichardElling.com > > ZFS training on deduplication, NexentaStor, and NAS performance > > Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com > > > > > > > > > > > > > > ZFS storage and performance consulting at http://www.RichardElling.com > ZFS training on deduplication, NexentaStor, and NAS performance > Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com > > > > > >
_______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
