As described in https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network the Kafka protocol is message-oriented, thus largely connectionless (TCP is used for reliability, not for connectedness). The overhead is just the TCP handshake - if the TCP handshake is your largest performance problem, then I have to congratulate you :-).
On Tue, Mar 22, 2016 at 1:02 AM, BYEONG-GI KIM <bg...@bluedigm.com> wrote: > I have not measured that value exactly, but I just thought the case when we > use db; the frequent connection/disconnection causes overhead so that > connection pool is generally used to solve the problem. > > Does the kafka bring less overhead comparing with that case as much as what > the connection pool doesn't need to be considered? > > Best regards > > bgkim > > 2016-03-22 11:40 GMT+09:00 Christian Posta <christian.po...@gmail.com>: > > > Can you explain more? Have you measured the overhead of opening the > > connections? > > > > If I'm not mistaken, Kafka manages the connections under the covers to > each > > of the brokers that have topics (leader partitions) from which you're > > consuming. The connection(s) to each partition leader will stay around as > > long as you're consuming from them (ie, calling poll()) method. If you > > aren't polling the broker for messages within the session timeout period, > > the heartbeat mechanism will actually kick the consumer out and do a > > rebalance. I'd be curious to understand more about your usecase to see > > how/when you're experiencing overhead of the connection handshaking. > > > > On Mon, Mar 21, 2016 at 7:33 PM, BYEONG-GI KIM <bg...@bluedigm.com> > wrote: > > > > > Hello. I have a question that the latest kafka, 0.9.0.1, provides any > > APIs > > > for managing connection pool of kafka on both consumer and producer > > sides. > > > > > > I think the overhead which happens while establishing connection from > > > consumer/producer to kafka broker(s) seems a little heavy. > > > > > > Thanks in advance! > > > > > > Best regards > > > > > > bgkim > > > > > > > > > > > -- > > *Christian Posta* > > twitter: @christianposta > > http://www.christianposta.com/blog > > http://fabric8.io > > > -- *Cees de Groot* PRINCIPAL SOFTWARE ENGINEER [image: PagerDuty logo] <http://pagerduty.com/> pagerduty.com c...@pagerduty.com <m...@pagerduty.com> +1(416)435-4085 [image: Twitter] <http://twitter.com/pagerduty>[image: FaceBook] <https://www.facebook.com/PagerDuty>[image: Google+] <https://plus.google.com/114339089137644062989>[image: LinkedIn] <https://www.linkedin.com/company/pagerduty>[image: Blog] <https://blog.pagerduty.com/>
