Re: [perf-discuss] poor application performance - very high cross-calls

2009-07-05 Thread Andrew C. Henle
> Yes, temporarily moving to UFS would eliminate the > xcalls. > > - Steve > The smaller UFS blocksize will probably help, too. The call stacks indicate the application in question is using fread() to read data. The fread() call will read a full disk block at a time, which IIRC comes from th

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-30 Thread Steve Sistare
Yes, temporarily moving to UFS would eliminate the xcalls. - Steve On 06/29/09 23:51, Matthew Flanagan wrote: [...] It appears that fwd is mmapping the files. So as a temporary measure would moving > the logs to UFS improve things? I'll be applying latest recommended patches but another po

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Matthew Flanagan
Hi Phil, Comments inline. > > > > truss shows writes like this: > > > > /1: write(22, "\0 4\001 :919C\v J H90 b".., 96) > = 96 > : write(23, " %97 *AF", 4) >= 4 > 97 *AF", 4)= 4 > > > > and reads like this: > > /1: read(77, "\0 D\003

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Mike Gerdts
On Mon, Jun 29, 2009 at 2:54 AM, Matthew Flanagan wrote: [snip] > 6. What is it reading & writing? This first column is the file descriptor > number, the second the number of times it was seen: > > # dtrace -n 'syscall::read:entry /pid == 2153/ { @[arg0] = count(); }' > dtrace: description 'sysc

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Steve Sistare
The high xcall rate consuming high %sys is likely the majority of the problem. You need the fix for: 6699438 zfs induces crosscall storm under heavy mapped sequential read This is the case that Phil recalled working on. It was recently fixed, and will be in S10U8 RR. If you need a patch earlier

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Phil Harman
Comments inline. Matthew Flanagan wrote: Hi Phil Matthew, In addition to Roland's comment about patching (which is particularly important due to the rate of change in ZFS) ... 1) Your dd test doesn't really tell us much. Do you know what size writes fwd is doing? I doubt it is 128k. You

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Matthew Flanagan
Hi Phil > Matthew, > > In addition to Roland's comment about patching (which > is particularly > important due to the rate of change in ZFS) ... > > 1) Your dd test doesn't really tell us much. Do you > know what size > writes fwd is doing? I doubt it is 128k. You could > use truss/dtrace to >

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Matthew Flanagan
> stion): Did you ever apply the Recommended > Patch Cluster ? AFAIK the latest kernel patch is > 137137-09 (see > http://sunsolve.sun.com/search/document.do?assetkey=1- > 21-137137-09) > which should fix some issues related to high number > of crosscalls etc. I'm planning to apply the recommended

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Phil Harman
Matthew, In addition to Roland's comment about patching (which is particularly important due to the rate of change in ZFS) ... 1) Your dd test doesn't really tell us much. Do you know what size writes fwd is doing? I doubt it is 128k. You could use truss/dtrace to find out and then use dd ag

Re: [perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Roland Mainz
Matthew Flanagan wrote: > I'm trying to track down the source of a performance problem with an > application. The application in question is from a firewall vendor and the > component of it that is suffering is the log handling daemon. The log daemon > is single threaded and receives logs from f

[perf-discuss] poor application performance - very high cross-calls

2009-06-29 Thread Matthew Flanagan
Hi, I'm trying to track down the source of a performance problem with an application. The application in question is from a firewall vendor and the component of it that is suffering is the log handling daemon. The log daemon is single threaded and receives logs from firewalls and also services