I have a 1:M producer-consumer problem (1 producer, M consumers) that I use for benchmarking. Every (producer, consumer_i) pair has a dedicated message queue implemented with a linked list protected with one pthread_mutex_t and two pthread_cond_t with default attributes.
A 256MB block of RAM is divided into M = 2^0 .. 2^14 chunks, and for each chunk a new thread is created; the producer has also a dedicated thread (yes, I create up to 16385 threads). For each chunk, a simple message consisting of (pointer,length) pair is sent 128 times to each of the workers and AES-encrypted. When the worker is finished with a chunk, it sends the reply to the producer. If the message queue is empty, the threads block on their respective condition variables associated with the queues. The producer first sends all 128 chunks to the workers (which are signalled if necessary), then waits for 128 replies from each of them. This is done serially, and not round-robin (i.e. first 128 messages are sent to worker 0, then to worker 1, etc..; similarly when reading replies). Each communication also includes allocation and freeing of a message; for this the umem allocator is used with a cache. The threads' stack sizes are set to 16kB. The benchmark is compiled in 64-bit mode and executed on Solaris 10, dual-core AMD64 (1 socket with two cores) and 2GB of RAM. Now the results: for M=2^1 .. M=2^11 (2 .. 2048) threads, the running time (wall-clock time) is fairly constant around ~10 seconds. Beyond this number, as M doubles, the running time also roughly doubles: (2^12 threads, 13s), (2^13 threads, 20s), (2^14 threads, 35s). Running iostat and vmstat in parallel confirms that no swapping occurs. 33% of time is reported to be spent in system, (with 9% of CPU time idle?!), with ~150k/sec systemcalls and ~120k/sec context switches. Can anybody offer some insight on why this sudden degradation in performance occurs? This message posted from opensolaris.org _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
