Thanks for answers. Does the callback get call on failure only or for success as well ? Also, how do I do this on Kafka 0.8.0 ? Is there any plan for adding buffering on disk for next version ? Also, when application restart, kafka producer will have to transmit the messages (buffered messages on disk) ?
I can not upgrade to Kafka 0.8.1.1 right now. how can I achieve this on Kafka version 0.8.0. Also, when retransmitting the messages I wanted to transfer dedicated Kafka back-up or reserved partition for topic (near-real time topic). How would you handle retransmission of messages ? Any idea about implementation will be very helpful. Thanks, Bhavesh On Mon, May 5, 2014 at 9:29 PM, Jun Rao <jun...@gmail.com> wrote: > At LinkedIn, we only buffer messages in memory in the producer. We try as > much as we can to make the Kafka cluster always available (with > replication). The "on failure" feature you mentioned can be implemented by > providing a callback. You can see the example in the javadoc. > > *send > < > http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord > , > kafka.clients.producer.Callback)>*(ProducerRecord< > http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html > > > record, Callback< > http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html> > callback) > > Thanks, > > Jun > > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry > <mistry.p.bhav...@gmail.com>wrote: > > > We are using Kafka for operation metrics and we do not want to loose any > > data at all if there is issue with Network or all brokers needs to be > > rebooted or operation reason down time while all producers are still > > producing data on front end side. We use async to publish messages and > we > > are using Kafka version 0.8.0. > > > > > > > > Has any one implemented buffering on local disk (on producer side) and > > transmit messages when network connection is restored? How do I get > > handle to list of messages async thread could not transfer after x reties > > ? I know new producer API has callback interface, but is per message not > > per producer instance ? > > > > > > > > Is this final new Producer API ? > > > > > http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html > > > > > > is there a plan to add method like. onFailure(List<Messages> messages, > > Exception exception ) ? > > > > > > > > > > Basically, I have to address the QOS on producer side, and be able to > > buffer on disk and retransmit all message to partitions that are reserved > > for messages that happened in past. > > > > > > > > How does Linked-in handle QOS on producer side ? > > > > > > Is there any plan to add this QOS feature on Producer Side with > Strategies > > to > > store and retransmit the message ? If we do get the list of messages is > > call back, will it be compressed data ? > > > > > > > > I would appreciate Kafka Developers and others feedback on how to > implement > > QOS. > > > > > > Thanks, > > > > > > Bhavesh > > >
