Hi TD, Thanks a lot for your reply :) I am already looking into creating a new DStream for SQS messages. It would be very helpful if you can provide with some guidance regarding the same.
The main motive of integrating SQS with spark streaming is to make my Jobs run in high availability. As of now I am having a downloader, which downloads file pointed by SQS messages and the my spark streaming job comes in action to process them. I am planning to move whole architecture into high availability (spark streaming job can easily be shifted to high availability), only piece left is integrate SQS with spark streaming such that it can automatically recover master node failure. Also, I want to make a single pipeline, start from getting SQS message to the processing of corresponding file. I couldn't think of any other approach to make my SQS downloader run in high availability mode. The only thing I have to get, is create a Dstream which reads sqs messages from the corresponding queue. Please let me know if there is any other work around. Thanks -- Lalit ----- Lalit Yadav la...@sigmoidanalytics.com -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-at-least-once-guarantee-tp10902p11525.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
