Re: Scala: Kafka Consumer (kafka-clients 0.9.0.1)

2016-08-16 Thread Ajay Sharma
BTW we used 0.8.x On 8/16/16, 9:51 AM, "Ajay Sharma" wrote: >Amir, >We had similar requirement to consume every message reliably; the approach >I picked was to push any message with unsuccessful consumption to a >secondary topic for later entertainment; in our case the message/events >were non-d

Re: Scala: Kafka Consumer (kafka-clients 0.9.0.1)

2016-08-16 Thread Ajay Sharma
Amir, We had similar requirement to consume every message reliably; the approach I picked was to push any message with unsuccessful consumption to a secondary topic for later entertainment; in our case the message/events were non-dependant so we use to make second attempt for consumption and on any

Re: Scala: Kafka Consumer (kafka-clients 0.9.0.1)

2016-08-16 Thread Amir Zuker
This is not the case in my situation. I am using an older version where it doesn't have the subscriber model, and processing messages can occur concurrently in multiple threads. If I use the KafkaConsumer with multiple threads and partitions - kafkaConsumer.run(5) //5 thread count - How can I comm

Re: Scala: Kafka Consumer (kafka-clients 0.9.0.1)

2016-08-16 Thread Sudev A C
Hi, Message object consists of partition, topic, offset and message. https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html You many use this to get current offsets for topic-partition combination. Thanks Sudev On Tue, Aug 16, 2016 at 2:20 PM, Amir Z

Scala: Kafka Consumer (kafka-clients 0.9.0.1)

2016-08-16 Thread Amir Zuker
Hi everyone, I have a question regarding the 'KafkaConsumer' and its API in regards to committing offsets. (kafka-clients 0.9.0.1) *The scenario -* I am working with auto commit set to disabled because I want to implement a retry mechanism and eventually transfer the message to another topic that