I've made some good progress on tuning Kafka for our needs, reaching about 40,000 events per second in our environment. One thing that I noticed in all of the Kafka server logs is an inordinate amount (I think?) of Closing socket connection INFO statements.
I realize they are INFO, so I assume they are expected. Can someone point me to a resource or provide a couple sentence overview on the socket strategy here? I'm curious why 6 producer clients, each using 2 threads, are opening and closing sockets so much. That's an expensive operation, and surely it affects performance since these connections are closing anywhere from 2 seconds to 30 seconds (seemingly randomly). Thanks! Jason [2013-05-21 14:31:07,726] INFO Closing socket connection to /10.10.1.133. (kafka.network.Processor) [2013-05-21 14:31:08,583] INFO Closing socket connection to /10.10.1.132. (kafka.network.Processor) [2013-05-21 14:31:08,930] INFO Closing socket connection to /10.10.1.130. (kafka.network.Processor) [2013-05-21 14:31:09,599] INFO Closing socket connection to /10.10.1.133. (kafka.network.Processor) [2013-05-21 14:31:10,310] INFO Closing socket connection to /10.10.1.67. (kafka.network.Processor) [2013-05-21 14:31:10,436] INFO Closing socket connection to /10.10.1.131. (kafka.network.Processor) [2013-05-21 14:31:11,165] INFO Closing socket connection to /10.10.1.134. (kafka.network.Processor) [2013-05-21 14:31:13,260] INFO Closing socket connection to /10.10.1.67. (kafka.network.Processor) [2013-05-21 14:31:13,591] INFO Closing socket connection to /10.10.1.134. (kafka.network.Processor) [2013-05-21 14:31:20,988] INFO Closing socket connection to /10.10.1.130. (kafka.network.Processor) [2013-05-21 14:31:22,361] INFO Closing socket connection to /10.10.1.132. (kafka.network.Processor) [2013-05-21 14:31:22,459] INFO Closing socket connection to /10.10.1.133. (kafka.network.Processor) [2013-05-21 14:31:23,915] INFO Closing socket connection to /10.10.1.131. (kafka.network.Processor) [2013-05-21 14:31:24,068] INFO Closing socket connection to /10.10.1.132. (kafka.network.Processor) [2013-05-21 14:31:24,947] INFO Closing socket connection to /10.10.1.130. (kafka.network.Processor) [2013-05-21 14:31:26,168] INFO Closing socket connection to /10.10.1.131. (kafka.network.Processor) [2013-05-21 14:31:26,534] INFO Closing socket connection to /10.10.1.133. (kafka.network.Processor) [2013-05-21 14:31:26,536] INFO Closing socket connection to /10.10.1.67. (kafka.network.Processor) [2013-05-21 14:31:28,251] INFO Closing socket connection to /10.10.1.134. (kafka.network.Processor) [2013-05-21 14:31:30,734] INFO Closing socket connection to /10.10.1.67. (kafka.network.Processor) [2013-05-21 14:31:31,322] INFO Closing socket connection to /10.10.1.134. (kafka.network.Processor) [2013-05-21 14:31:40,833] INFO Closing socket connection to /10.10.1.130. (kafka.network.Processor) [2013-05-21 14:31:41,107] INFO Closing socket connection to /10.10.1.133. (kafka.network.Processor) [2013-05-21 14:31:43,123] INFO Closing socket connection to /10.10.1.132. (kafka.network.Processor) [2013-05-21 14:31:43,169] INFO Closing socket connection to /10.10.1.131. (kafka.network.Processor) [2013-05-21 14:31:43,221] INFO Closing socket connection to /10.10.1.132. (kafka.network.Processor) [2013-05-21 14:31:45,833] INFO Closing socket connection to /10.10.1.130. (kafka.network.Processor) [2013-05-21 14:31:47,143] INFO Closing socket connection to /10.10.1.133. (kafka.network.Processor) [2013-05-21 14:31:48,199] INFO Closing socket connection to /10.10.1.131. (kafka.network.Processor) This electronic message contains information which may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify us by e-mail at (postmas...@rapid7.com) immediately.
