Can you explain more? Have you measured the overhead of opening the connections?
If I'm not mistaken, Kafka manages the connections under the covers to each of the brokers that have topics (leader partitions) from which you're consuming. The connection(s) to each partition leader will stay around as long as you're consuming from them (ie, calling poll()) method. If you aren't polling the broker for messages within the session timeout period, the heartbeat mechanism will actually kick the consumer out and do a rebalance. I'd be curious to understand more about your usecase to see how/when you're experiencing overhead of the connection handshaking. On Mon, Mar 21, 2016 at 7:33 PM, BYEONG-GI KIM <bg...@bluedigm.com> wrote: > Hello. I have a question that the latest kafka, 0.9.0.1, provides any APIs > for managing connection pool of kafka on both consumer and producer sides. > > I think the overhead which happens while establishing connection from > consumer/producer to kafka broker(s) seems a little heavy. > > Thanks in advance! > > Best regards > > bgkim > -- *Christian Posta* twitter: @christianposta http://www.christianposta.com/blog http://fabric8.io
