>>
forcing people on kafka 8.x to upgrade their brokers is questionable.
<<
I agree and i was more thinking maybe there is a way to support both for a
period of time (of course means some more code to maintain :-)).
thanks
Mario
From: Cody Koeninger <c...@koeninger.org>
To: Mario Ds Briggs/India/IBM@IBMIN
Cc: "dev@spark.apache.org" <dev@spark.apache.org>
Date: 04/12/2015 12:15 am
Subject: Re: Spark Streaming Kafka - DirectKafkaInputDStream: Using the
new Kafka Consumer API
Honestly my feeling on any new API is to wait for a point release before
taking it seriously :)
Auth and encryption seem like the only compelling reason to move, but
forcing people on kafka 8.x to upgrade their brokers is questionable.
On Thu, Dec 3, 2015 at 11:30 AM, Mario Ds Briggs <mario.bri...@in.ibm.com>
wrote:
Hi,
Wanted to pick Cody's mind on what he thinks about
DirectKafkaInputDStream/KafkaRDD internally using the new Kafka consumer
API. I know the latter is documented as beta-quality, but yet wanted to
know if he sees any blockers as to why shouldn't go there shortly. On my
side the consideration is that kafka 0.9.0.0 introduced Authentication
and Encryption (beta again) between clients & brokers, but this is
available only newer Consumer API's and not in the older
Low-level/High-level API's.
From briefly studying the implementation of
DirectKafkaInputDStream/KafkaRDD and new Consumer API, my thinking is
that it is possible to support the exact current implementation you have
using the new API's.
One area that isnt so straightforward was the ctor of KafkaRDD fixes the
offsetRange (I did read about the deterministic feature you were after)
and i couldnt find a direct method in the new Consumer API to get the
current 'latest' offset - however one can do a consumer.seekToEnd() and
then call a consumer.position().
Of course one other benefit is that the new Consumer API's abstracts away
having to deal with finding the leader for a partition, so can get rid of
that code
Would be great to get your thoughts.
thanks in advance
Mario
