Thanks for retesting. In hindsight my analysis has a fatal flaw. I used the
reap deficit at the end of the run of 162468 threads to estimate hash table
residency, but the residency should instead track the steady state rate of
thread creation/destruction of approx 5000 threads (assuming that thread_reap
can keep up). The hash lists are much smaller at that level.
Your log files show "smv threads created" per second, but they do not
show "smv threads destroyed" per second. Try measuring the latter.
Possibly there is a delay in destroying the threads, either at the
application layer or in the kernel. If the threads have not reached
deathrow, then the thread_reaper cannot reap them! A thread is
added to deathrow in the kernel function reapq_add().
If that does not explain things, then profile the kernel to account for the
time spent in thread_reap and descendents.
- Steve
On 10/23/09 09:25, Thomas Blank wrote:
Hi Steve,
thanks for your detailed explanations and the suggestion with default_stksize.
All your explanations sound comprehensible but I'm sorry, setting a non-default
stacksize does not help.
The thread_reaper() rate of 2500 threads/second is limiting your throughput and
sounds low.
Indeed, the thread_reaper is not limited to 2500 threads/s. Most of the time it
keeps up with the amount of created threads. There are only a few samples where
the amount of reaped threads is much lower than the threads created. I can
clarify this with some logfiles.
I did two tests, both with segkp set to 8gb and stacks size set to 32kb:
[r...@itotcsol104 ~]# echo default_stksize/D|mdb -k
default_stksize:
default_stksize:32768
The machine was rebooted directly before the first test. Filesystem is UFS
mounted with forcedirectio, IO is generated with filebench (8kb writes).
First test runs 360 secs with a total of 1644951 ops. No hang occurs as sekpg
is not filled (sekgp-mem_inuse1.log). I stopped the dtrace script directly
after filebench has finished (thread_reap1.log). Please note, filebench had to
allocate the files first, therefore the amount of threads created and IOs
performed by filebench's test differ. Find the output of iostat attached.
I ran a second test with a duration of 1200 secs. Take a look at the logfiles
(name with *2.log). At 10:46:40 the segkp is filled and the IO hangs (see
iostat-logfile). IO returns at 10:47:54. During this time the reaper again
handles ~2300 threads/s, same value as after filebench has finished. filebench
did 3776514 ops and equals number of threads reaped and created.
During the run of filebench, you can see that number of reaped threads is
nearly equal to created threads. Sporadically, thread_reaper slows down
extremly:
2009 Oct 23 10:47:59
threads reaped 16464
smv threads created 16462
2009 Oct 23 10:48:00
threads reaped 1374 <<<<
smv threads created 11341
2009 Oct 23 10:48:01
threads reaped 15360
smv threads created 16633
From these points in time the difference is coming from: The reaper
sporadically can't keep up with the massive creation of threads.
I'm wondering why he is slowing down sometimes? Why is he not able to keep up
the high rates?
Thanks in advance, Thomas
------------------------------------------------------------------------
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
