Hi
I am processing kafka messages using spark streaming 1.3.
I am using mapPartitions function to process kafka message.
How can I access offset no of individual message getting being processed.
JavaPairInputDStream<byte[], byte[]> directKafkaStream
=KafkaUtils.createDirectStream(..);
directKafkaStream.mapPartitions(new
FlatMapFunction<Iterator<Tuple2<byte[],byte[]>>, String>() {
public Iterable<String> call(Iterator<Tuple2<byte[], byte[]>> t)
throws Exception {
while(t.hasNext()){
Tuple2<byte[], byte[]> tuple = t.next();
byte[] key = tuple._1();
byte[] msg = tuple._2();
///how to get kafka partition no and offset of this message
}
}
});