Re: Streaming Receiverless Kafka API + Offset Management

2015-11-16 Thread Nick Evans
wse/SPARK-10963 > > I think adding any ZK specific behavior to spark is a bad idea, since ZK > may no longer be the preferred storage location for Kafka offsets within > the next year. > > > > On Mon, Nov 16, 2015 at 9:53 AM, Nick Evans wrote: > >> I really like the S

Streaming Receiverless Kafka API + Offset Management

2015-11-16 Thread Nick Evans
I really like the Streaming receiverless API for Kafka streaming jobs, but I'm finding the manual offset management adds a fair bit of complexity. I'm sure that others feel the same way, so I'm proposing that we add the ability to have consumer offsets managed via an easy-to-use API. This would be