Memory Growth during multi-threaded connections

[Haney John-CJH043]

Title: Message

I'm seeing the CPU usage increase when running a multi-threaded application doing many connections.  This appears to be a function of the number of connections which have been established AND the rate at which new connections are being established.  The CPU graph is very clearly linear and increasing.

 

To better analyze this problem, I've removed polling and messaging from my application (basically anything besides the SSL connections) and still see this increase.

 

The test we are running most often establishes 32 connections per second (1920 per minute) on solaris.

 

I've captured sar data while the app is running and noticed that there is significant memory growth on the box (not sure if this is directly related to the increased CPU or not).

 

19:07:27 sml_mem   alloc  fail  lg_mem   alloc  fail  ovsz_alloc  fail

...

19:11:07 164382976 81841911     0 1489608704 912652856     0    38682624     0
19:11:12 164382976 82292739     0 1489608704 914741880     0    38682624     0
19:11:17 164382976 82543687     0 1489608704 915770680     0    38682624     0
19:11:22 164382976 82891719     0 1489608704 917329592     0    39010304     0
19:11:27 164382976 83311059     0 1489608704 919285496     0    39010304     0
19:11:32 164382976 83595487     0 1489608704 920461816     0    39010304     0
19:11:37 164382976 83942223     0 1489608704 922017080     0    39010304     0

2187.825 small memory allocs per connection

9754.4 large memory allocs per connection

 

The following are the context and SSL settings:

 

SSL_CTX_set_timeout(ctx, 43200); /* 12 hours */
SSL_CTX_set_session_cache_mode(ctx, SSL_SESS_CACHE_OFF);
SSL_CTX_set_mode(ctx, SSL_MODE_AUTO_RETRY);
SSL_set_verify(ssl, SSL_VERIFY_NONE, my_empty_cb);  /* my_empty_cb returns 1 */

 

 

Thanks in advance,

- John Haney