Or can I generally create new RDD from transformation and enrich its partitions with some metadata so that I would copy OffsetRanges in my new RDD in DStream?
On Mon, Aug 17, 2015 at 1:08 PM, Petr Novak <oss.mli...@gmail.com> wrote: > Hi all, > I need to transform KafkaRDD into a new stream of deserialized case > classes. I want to use the new stream to save it to file and to perform > additional transformations on it. > > To save it I want to use offsets in filenames, hence I need OffsetRanges > in transformed RDD. But KafkaRDD is private, hence I don't know how to do > it. > > Alternatively I could deserialize directly in messageHandler before > KafkaRDD but it seems it is 1:1 transformation while I need to drop bad > messages (KafkaRDD => RDD it would be flatMap). > > Is there a way how to do it using messageHandler, is there another > approach? > > Many thanks for any help. > Petr >
