TLDR: I use one Callback per job I send to Kafka and include that sort of information by reference in the Callback instance.
Our system is currently moving data from beanstalkd to Kafka due to historical reasons so we use the callback to either delete or release the message depending on success. The org.apache.kafka.clients.producer.Callback I give to the send method is an instance of a class that stores all the additional information I need to process the callback. Remember that the async call operates in the Kafka producer thread so they must be fast to avoid constraining the throughput. My call back ends up putting information about the call to beanstalk into another executor service for later processing. Christian On 08/26/2014 12:35 PM, Ryan Persaud wrote: > Hello, > > I'm looking to insert log lines from log files into kafka, but I'm concerned > with handling asynchronous send() failures. Specifically, if some of the log > lines fail to send, I want to be notified of the failure so that I can > attempt to resend them. > > Based on previous threads on the mailing list > (http://comments.gmane.org/gmane.comp.apache.kafka.user/1322), I know that > the trunk version of kafka supports callbacks for dealing with failures. > However, the callback function is not passed any metadata that can be used by > the producer end to reference the original message. Including the key of the > message in the RecordMetadata seems like it would be really useful for > recovery purposes. Is anyone using the callback functionality to trigger > resends of failed messages? If so, how are they tying the callbacks to > messages? Is anyone using other methods for handling async errors/resending > today? I can’t imagine that I am the only one trying to do this. I asked > this question on the IRC channel today, and it sparked some discussion, but I > wanted to hear from a wider audience. > > Thanks for the information, > -Ryan > >
signature.asc
Description: OpenPGP digital signature
