Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-16 Thread Zeljko Vrba
On Wed, Apr 16, 2008 at 11:02:51AM -0700, Bart Smaalders wrote: > > What's the size of the relevant TLBs? With text, stack and heap > mappings for all > threads, this result isn't terribly surprising. > Huh, how do I find the size of TLBs? :/ Anyway, stack and heap mappings should not affect t

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-16 Thread Bart Smaalders
Zeljko Vrba wrote: > On Fri, Apr 11, 2008 at 11:23:10AM -0700, David Lutz wrote: >> Take a look at your cache miss rates as you cross the 2^11 boundary. >> My guess is that you will see something start to go through the roof. >> > cputrack has too much overhead when having a bunch of LWPs. I did r

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-14 Thread Zeljko Vrba
On Mon, Apr 14, 2008 at 07:21:06AM -0500, Mike Gerdts wrote: > > Solaris 10 u5 (due soon) begins to offer support for shared contexts. > I have read the release notes, and this applies only to SPARC systems and dynamic intimate shared memory (DISM). __

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-14 Thread Mike Gerdts
On Mon, Apr 14, 2008 at 3:11 AM, Zeljko Vrba <[EMAIL PROTECTED]> wrote: > All threads are executing the same code which is rather small -- so I see > no reason for this linear increase in the # of ITLB misses with the number > of threads. OK, more threads = more user<>kernel transitions. Does

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-14 Thread Zeljko Vrba
On Fri, Apr 11, 2008 at 11:23:10AM -0700, David Lutz wrote: > > Take a look at your cache miss rates as you cross the 2^11 boundary. > My guess is that you will see something start to go through the roof. > cputrack has too much overhead when having a bunch of LWPs. I did run cpustat though, in

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-11 Thread Eric Saxe
Zeljko Vrba wrote: > On Thu, Apr 10, 2008 at 01:02:34PM -0700, Alexander Kolbasov wrote: > >> It may be useful to observe prstat -mL which will report micro-state >> accounting data for each thread. >> >> > prstat -mL takes a *lot* of time with 16k threads. Nevertheless, I have some > fu

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-11 Thread David Lutz
Message - From: Zeljko Vrba <[EMAIL PROTECTED]> Date: Thursday, April 10, 2008 11:00 am Subject: Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions To: Mike Gerdts <[EMAIL PROTECTED]> Cc: perf-discuss@opensolaris.org > On Thu, Apr 10, 2008 at 11:57:2

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-11 Thread Zeljko Vrba
On Fri, Apr 11, 2008 at 12:16:21PM +0200, [EMAIL PROTECTED] wrote: > Hi, > I think you already said this, but are you using pthread_cond_signal, or > pthread_cond_broadcast to wake up threads? > I use pthread_cond_signal since there's ever only at most a single thread to be woken up.

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-11 Thread [EMAIL PROTECTED]
Hi, I think you already said this, but are you using pthread_cond_signal, or pthread_cond_broadcast to wake up threads? max Zeljko Vrba wrote: > On Thu, Apr 10, 2008 at 01:02:34PM -0700, Alexander Kolbasov wrote: > >> It may be useful to observe prstat -mL which will report micro-state >> acc

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-11 Thread Zeljko Vrba
On Thu, Apr 10, 2008 at 01:02:34PM -0700, Alexander Kolbasov wrote: > > It may be useful to observe prstat -mL which will report micro-state > accounting data for each thread. > prstat -mL takes a *lot* of time with 16k threads. Nevertheless, I have some further data: reruning the dtrace gave

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Alexander Kolbasov
> On Thu, Apr 10, 2008 at 12:08:59PM -0400, Jonathan Adams wrote: > > > > Hrm; try dropping the ', ustack()' from the script; that way, you're > > only considering kernel stacks, which should raise the amount of signal. > > > OK, this is again for 16k threads: > > - 835 hits in unix`cpu_halt

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Zeljko Vrba
On Thu, Apr 10, 2008 at 11:57:27AM -0500, Mike Gerdts wrote: > > In the "Opteron's Data Cache & Load/Store Units" notice, for instance, > that "Dual L1 Tags" has 2x1024 (presumably = 2048) entries. That > seems suspiciously close to the knee in your observed performance. > Hm, might be, but it s

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Zeljko Vrba
On Thu, Apr 10, 2008 at 12:08:59PM -0400, Jonathan Adams wrote: > > Hrm; try dropping the ', ustack()' from the script; that way, you're > only considering kernel stacks, which should raise the amount of signal. > OK, this is again for 16k threads: - 835 hits in unix`cpu_halt - 63 in unix`_r

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Mike Gerdts
On Thu, Apr 10, 2008 at 8:58 AM, Zeljko Vrba <[EMAIL PROTECTED]> wrote: > The benchmark is compiled in 64-bit mode and executed on Solaris 10, > dual-core AMD64 (1 socket with two cores) and 2GB of RAM. Now the > results: for M=2^1 .. M=2^11 (2 .. 2048) threads, the running time > (wall-clock time)

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Jonathan Adams
On Apr 10, 2008, at 11:41 AM, Zeljko Vrba wrote: > On Thu, Apr 10, 2008 at 10:48:06AM -0400, Jonathan Adams wrote: >> >> >> Do you have administrator access to this system? With dtrace(1M), >> you >> can drill down on this, and get more data on what's going on. See: >> > Yes, I do; backtraces

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Zeljko Vrba
Ugh, I'm to blame on the title -- the graph from which I drew the conclusion has logarithmic x-axis and linear y-axis, so the increase looks exponential, whereas it is in fact linear. However, the measurements are still accurate - after a certain threshold (~ 2048 threads), the scheduling is no

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread David . Valin
What version/rev of Solaris are you running? Thanks Dave V Jonathan Adams wrote: > On Apr 10, 2008, at 9:58 AM, Zeljko Vrba wrote: > >> I have a 1:M producer-consumer problem (1 producer, M consumers) >> that I use for benchmarking. Every (producer, consumer_i) pair has >> a dedicated me

Re: [perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Jonathan Adams
On Apr 10, 2008, at 9:58 AM, Zeljko Vrba wrote: > I have a 1:M producer-consumer problem (1 producer, M consumers) > that I use for benchmarking. Every (producer, consumer_i) pair has > a dedicated message queue implemented with a linked list protected > with one pthread_mutex_t and two pth

[perf-discuss] Thread scheduler: exponentially slow in extreme conditions

2008-04-10 Thread Zeljko Vrba
I have a 1:M producer-consumer problem (1 producer, M consumers) that I use for benchmarking. Every (producer, consumer_i) pair has a dedicated message queue implemented with a linked list protected with one pthread_mutex_t and two pthread_cond_t with default attributes. A 256MB block of RAM i