Sorry I don't understand... Is it a cassandra acknowledge to actors that you want ? Why do you want to ack after writing to cassandra ? Your pipeline kafka=>spark=>cassandra is supposed to be exactly once, so you don't need to wait for cassandra ack, you can just write to kafka from actors and then notify the user ?
2016-11-28 23:15 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: > Thanks Vincent for the input. Not sure I understand your suggestion. > Please clarify. > > Few words about my use case : > When the user watches a video, I get the position events from the user > that indicates how much they have completed viewing and at a certain point, > I mark that Video as complete and persist that info to cassandra. > > How do I notify the user that it was marked complete? > > Are you suggesting I write the completed events to kafka(different topic) > and the akka consumer could read from this? There could be many completed > events from different users in this topic. So the akka consumer should > pretty much do what a spark streaming does to process this without the > knowledge of the kafka offset. > > So not sure what you mean by kafka offsets will do the job, how will the > akka consumer know the kafka offset? > > On Mon, Nov 28, 2016 at 12:52 PM, vincent gromakowski < > vincent.gromakow...@gmail.com> wrote: > >> You don't need actors to do kafka=>spark processing=>kafka >> Why do you need to notify the akka producer ? If you need to get back the >> processed message in your producer, then implement an akka consummer in >> your akka app and kafka offsets will do the job >> >> 2016-11-28 21:46 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: >> >>> Thanks Daniel for the response. >>> >>> I am planning to use Spark streaming to do Event Processing. I will have >>> akka actors sending messages to kafka. I process them using Spark streaming >>> and as a result a new events will be generated. How do I notify the akka >>> actor(Message producer) that a new event has been generated? >>> >>> >>> >>> On Mon, Nov 28, 2016 at 9:51 AM, Daniel van der Ende < >>> daniel.vandere...@gmail.com> wrote: >>> >>>> Well, I would say it depends on what you're trying to achieve. Right >>>> now I don't know why you are considering using Akka. Could you please >>>> explain your use case a bit? >>>> >>>> In general, there is no single correct answer to your current question >>>> as it's quite broad. >>>> >>>> Daniel >>>> >>>> On Mon, Nov 28, 2016 at 9:11 AM, shyla deshpande < >>>> deshpandesh...@gmail.com> wrote: >>>> >>>>> My data pipeline is Kafka --> Spark Streaming --> Cassandra. >>>>> >>>>> Can someone please explain me when would I need to wrap akka around >>>>> the spark streaming app. My knowledge of akka and the actor system is >>>>> poor. >>>>> Please help! >>>>> >>>>> Thanks >>>>> >>>> >>>> >>>> >>>> -- >>>> Daniel >>>> >>> >>> >> >
