Thanks everybody! This is super useful!
Louisia >From Jagadish Venkatraman <jagadish1...@gmail.com> Subject Re: Send a msg to all partitions Date Sat, 19 Mar 2016 02:12:55 GMT The incoming message envelope that you receive has methods to look up what system or stream partition it is from. So you can choose to react differently if this message is from a broadcast stream. Let us know if you need more help! On Friday, March 18, 2016, Jagadish Venkatraman <jagadish1...@gmail.com> wrote: > Broad cast stream is the Samza feature designed for this exact > requirement. Your thinking is on the right lines. > > Please take a look at task.broadcast.inputs from the Samza configuration > reference page. > > On Friday, March 18, 2016, Louisia Famalda <louisia.fama...@gmail.com > <javascript:_e(%7B%7D,'cvml','louisia.fama...@gmail.com');>> wrote: > >> Hi, >> >> I'm new to Samza and I'm trying to do this: >> From my standalone app, I'm writing some ActionEvents to a Kafka topic >> with >> 5 partitions >> >> From Samza, I want to process those events but I need to send some filters >> information on how to process those events. >> For example, I want to keep all messages that are coming from a specific >> ZipCode. >> >> >> My thinking is to have another Topic: ControlEventStream, which will be a >> bootstrap kafka stream. >> ControlEvent will contains a userId as the key and a list of zipCode as a >> value. >> My Samza task will read from those 2 topics, get the filter directives >> from >> the ControlEventStream and then start processing all events. >> A user of the system, will then be able to send a ControlEvent with a new >> list of ZipCode that he wants to watch for. >> * The number of users using the system is less than 10. >> >> - Am I on the right track by using bootstrap stream + compaction to define >> the list of filters that a specific user is interested in ? >> >> Now, I need the Samza instantiated tasks to be able to receive update so a >> user can Add/remove/update the list of ZipCode that he is looking for. >> All ActionEvents are distributed cross all partitions and process in a >> distributed way. >> >> How do I guarantee that all Samza tasks will all receive ALL messages from >> ControlEventStream, so they all have the same set of ZipCode filters. >> >> Thanks in advance, >> Louisia. >> > > > -- > Sent from my iphone. > -- Sent from my iphone. On Fri, Mar 18, 2016 at 6:30 PM, Louisia Famalda <louisia.fama...@gmail.com> wrote: > Hi, > > I'm new to Samza and I'm trying to do this: > From my standalone app, I'm writing some ActionEvents to a Kafka topic > with 5 partitions > > From Samza, I want to process those events but I need to send some filters > information on how to process those events. > For example, I want to keep all messages that are coming from a specific > ZipCode. > > > My thinking is to have another Topic: ControlEventStream, which will be a > bootstrap kafka stream. > ControlEvent will contains a userId as the key and a list of zipCode as a > value. > My Samza task will read from those 2 topics, get the filter directives > from the ControlEventStream and then start processing all events. > A user of the system, will then be able to send a ControlEvent with a new > list of ZipCode that he wants to watch for. > * The number of users using the system is less than 10. > > - Am I on the right track by using bootstrap stream + compaction to define > the list of filters that a specific user is interested in ? > > Now, I need the Samza instantiated tasks to be able to receive update so > a user can Add/remove/update the list of ZipCode that he is looking for. > All ActionEvents are distributed cross all partitions and process in a > distributed way. > > How do I guarantee that all Samza tasks will all receive ALL messages from > ControlEventStream, so they all have the same set of ZipCode filters. > > Thanks in advance, > Louisia. >
