Also, 3. with spark 2.x structured streaming , if we want to switch across different modes like from micro-batching to continuous streaming mode, how it can be done while using Beam?
These are some of the initial questions which I am not able to understand currently. Regards, Chandan On Tue, May 15, 2018 at 5:45 PM, chandan prakash <chandanbaran...@gmail.com> wrote: > Hi Everyone, > I have just started exploring and understanding Apache Beam for new > project in my firm. > In particular, we have to take decision whether to implement our product > over spark streaming (as spark batch is already in our eco system) or > should we use Beam over spark runner to have future liberty of changing > underline runner. > > Couple of questions, after going through beam docs and examples, I have is: > > > 1. Are there any limitations in terms of implementations, > functionalities or performance if we want to run streaming on Beam with > Spark runner vs streaming on Spark-Streaming directly ? > > 2. Spark features like checkpointing, kafka offset management, how are > they supported in Apache Beam? Do we need to do some extra work for them? > > > Any answer or link to like wise discussion will be really appreciable. > Thanks in advance. > > Regards, > -- > Chandan Prakash > > -- Chandan Prakash
