If you want the offset of individual kafka messages , you can use this
consumer form Spark Packages ..
http://spark-packages.org/package/dibbhatt/kafka-spark-consumer
Regards,
Dibyendu
On Tue, Jul 28, 2015 at 6:18 PM, Shushant Arora <shushantaror...@gmail.com>
wrote:
> Hi
>
> I am processing kafka messages using spark streaming 1.3.
>
> I am using mapPartitions function to process kafka message.
> How can I access offset no of individual message getting being processed.
>
>
> JavaPairInputDStream<byte[], byte[]> directKafkaStream
> =KafkaUtils.createDirectStream(..);
>
> directKafkaStream.mapPartitions(new
> FlatMapFunction<Iterator<Tuple2<byte[],byte[]>>, String>() {
> public Iterable<String> call(Iterator<Tuple2<byte[], byte[]>> t)
> throws Exception {
>
> while(t.hasNext()){
> Tuple2<byte[], byte[]> tuple = t.next();
> byte[] key = tuple._1();
> byte[] msg = tuple._2();
> ///how to get kafka partition no and offset of this message
> }
> }
> });
>
>
>
>
>
