Hello All, I just want to make sure this is a right use case for Kafka --> Spark Streaming
Few words about my use case : When the user watches a video, I get the position events from the user that indicates how much they have completed viewing and at a certain point, I mark that Video as complete and that triggers a lot of other events. I need a way to notify the app about the creation of the completion event. Appreciate any suggestions. Thanks On Mon, Nov 28, 2016 at 2:35 PM, shyla deshpande <deshpandesh...@gmail.com> wrote: > In this case, persisting to Cassandra is for future analytics and > Visualization. > > I want to notify that the app of the event, so it makes the app > interactive. > > Thanks > > On Mon, Nov 28, 2016 at 2:24 PM, vincent gromakowski < > vincent.gromakow...@gmail.com> wrote: > >> Sorry I don't understand... >> Is it a cassandra acknowledge to actors that you want ? Why do you want >> to ack after writing to cassandra ? Your pipeline kafka=>spark=>cassandra >> is supposed to be exactly once, so you don't need to wait for cassandra >> ack, you can just write to kafka from actors and then notify the user ? >> >> 2016-11-28 23:15 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: >> >>> Thanks Vincent for the input. Not sure I understand your suggestion. >>> Please clarify. >>> >>> Few words about my use case : >>> When the user watches a video, I get the position events from the user >>> that indicates how much they have completed viewing and at a certain point, >>> I mark that Video as complete and persist that info to cassandra. >>> >>> How do I notify the user that it was marked complete? >>> >>> Are you suggesting I write the completed events to kafka(different >>> topic) and the akka consumer could read from this? There could be many >>> completed events from different users in this topic. So the akka consumer >>> should pretty much do what a spark streaming does to process this without >>> the knowledge of the kafka offset. >>> >>> So not sure what you mean by kafka offsets will do the job, how will >>> the akka consumer know the kafka offset? >>> >>> On Mon, Nov 28, 2016 at 12:52 PM, vincent gromakowski < >>> vincent.gromakow...@gmail.com> wrote: >>> >>>> You don't need actors to do kafka=>spark processing=>kafka >>>> Why do you need to notify the akka producer ? If you need to get back >>>> the processed message in your producer, then implement an akka consummer in >>>> your akka app and kafka offsets will do the job >>>> >>>> 2016-11-28 21:46 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: >>>> >>>>> Thanks Daniel for the response. >>>>> >>>>> I am planning to use Spark streaming to do Event Processing. I will >>>>> have akka actors sending messages to kafka. I process them using Spark >>>>> streaming and as a result a new events will be generated. How do I notify >>>>> the akka actor(Message producer) that a new event has been generated? >>>>> >>>>> >>>>> >>>>> On Mon, Nov 28, 2016 at 9:51 AM, Daniel van der Ende < >>>>> daniel.vandere...@gmail.com> wrote: >>>>> >>>>>> Well, I would say it depends on what you're trying to achieve. Right >>>>>> now I don't know why you are considering using Akka. Could you please >>>>>> explain your use case a bit? >>>>>> >>>>>> In general, there is no single correct answer to your current >>>>>> question as it's quite broad. >>>>>> >>>>>> Daniel >>>>>> >>>>>> On Mon, Nov 28, 2016 at 9:11 AM, shyla deshpande < >>>>>> deshpandesh...@gmail.com> wrote: >>>>>> >>>>>>> My data pipeline is Kafka --> Spark Streaming --> Cassandra. >>>>>>> >>>>>>> Can someone please explain me when would I need to wrap akka around >>>>>>> the spark streaming app. My knowledge of akka and the actor system is >>>>>>> poor. >>>>>>> Please help! >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Daniel >>>>>> >>>>> >>>>> >>>> >>> >> >
