Hello, I am using Kafka in plain producer-consumer style. I have a daily exchange of about 1 million messages avg 50kb size per message. I am still using 0.10.2.1
Whilst I am relying on Kafka's stability under a 9-node configuration - I would like to build some internal process which can keep my messages in case of "Unreachable" broker/host issue. I will be able to resend those messages provided that I can query them without moving the pointer. >From my last encounter, I remember that using topic/partition/offset details fetched using get() future call on new producer API, I can get details and use them in seek() to retrieve details. However, seek() will move the head as I understand from javadocs here here. <https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html> To explain my problem better, lets look at the following: 1) The producer failed to send some messages - no future received. This is due to broker outage or a disaster. 2) These pending messages are logged in DB or in some file. 3) The disaster is resolved - all systems are back up. 4) New messages are being generated and sent/received successfully. 5) But my message @ step 1 is still pending. For me, as long as I can query those messages and resend them it's all good. Currently, I simply store them in a DB upon failure, retrieve them when everything's back up, and resend them. I was wondering if someone can confirm whether v.11 will have something to ease this process, or should I submit a KIP for this? Regards,
