Hi all, I'm developing an AMQP - Kafka bridge and I'm facing with a great limitation of current 0.9.1 Kafka client APIs.
The consumer.poll() method is synchronous and as we know it's needed in order to send the heartbeat even if no records are available. It means that poll() needs to be called frequently before session ending. For that reason a thread per consumer is suggested because if we use a thread pool (let me say with 20 threads) but 1000 consumer ... it's possible that one consumer will be served too late without sending the heartbeat. I'm saying that because the bridging between AMQP and Kafka works at AMQP link level so for each attached link to a specific topic there is the corresponding Kafka consumer and the client could be thousands. Any thoughts about that ? Paolo. Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience
