Re: [Spark Streaming] kafka consumer announce

2014-08-29 Thread Evgeniy Shishkin
TD, can you please comment on this code? I am really interested in including this code in Spark. But i am bothering about some point about persistence: 1. When we extend Receiver and call store, is it blocking call? Does it return only when spark stores rdd as requested (i.e. replicated or on

Re: [Spark Streaming] kafka consumer announce

2014-08-21 Thread Evgeniy Shishkin
>> On 21 Aug 2014, at 20:25, Tim Smith wrote: >> >> Thanks. Discovering kafka metadata from zookeeper instead of brokers >> is nicer. Saving metadata and offsets to HBase, is that optional or >> mandatory? >> Can it be made optional (default to zookeeper)? >> For now we implemented and somewhat