Re: Reliable SQS Receiver for Spark Streaming

Thanks Akhil! I just looked it up in the code as well. Receiver.store(ArrayBuffer[T], ...) ReceiverSupervisorImpl.pushArrayBuffer(ArrayBuffer[T], ...) ReceiverSupervisorImpl.pushAndReportBlock(...) WriteAheadLogBasedBlockHandler.storeBlock(...) This implementation stor

Re: Reliable SQS Receiver for Spark Streaming

Yes, if you have enabled WAL and checkpointing then after the store, you can simply delete the SQS Messages from your receiver. Thanks Best Regards On Sat, Jun 13, 2015 at 6:14 AM, Michal Čizmazia wrote: > I would like to have a Spark Streaming SQS Receiver which deletes SQS > messages only aft

Reliable SQS Receiver for Spark Streaming

-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reliable-SQS-Receiver-for-Spark-Streaming-tp23302.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reliable SQS Receiver for Spark Streaming

I would like to have a Spark Streaming SQS Receiver which deletes SQS messages only after they were successfully stored on S3. For this a Custom Receiver can be implemented with the semantics of the Reliable Receiver. The store(multiple-records) call blocks until the given records have been store