Hi Gyula, I have another question. So if i cache something on the operator, to keep it up to date, i will always need to add and connect another stream of changes to the operator ?
Is this right for every case ? Cheers On Wed, Aug 19, 2015 at 3:21 PM, Welly Tambunan <if05...@gmail.com> wrote: > Hi Gyula, > > That's really helpful. The docs is improving so much since the last time > (0.9). > > Thanks a lot ! > > Cheers > > On Wed, Aug 19, 2015 at 3:07 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: > >> Hey, >> >> If it is always better to check the events against a more up-to-date >> model (even if the events we are checking arrived before the update) then >> it is fine to keep the model outside of the system. >> >> In this case we need to make sure that we can push the updates to the >> external system consistently. If you are using the PersistentKafkaSource >> for instance it can happen that some messages are replayed in case of >> failure. In this case you need to make sure that you remove duplicate >> updates or have idempotent updates. >> >> You can read about the checkpoint mechanism in the Flink website: >> https://ci.apache.org/projects/flink/flink-docs-master/internals/stream_checkpointing.html >> >> Cheers, >> Gyula >> >> On Wed, Aug 19, 2015 at 9:56 AM Welly Tambunan <if05...@gmail.com> wrote: >> >>> Thanks Gyula, >>> >>> Another question i have.. >>> >>> > ... while external model updates would be *tricky *to keep consistent. >>> >>> Is that still the case if the Operator treat the external model as >>> read-only ? We create another stream that will update the external model >>> separately. >>> >>> Could you please elaborate more about this one ? >>> >>> Cheers >>> >>> On Wed, Aug 19, 2015 at 2:52 PM, Gyula Fóra <gyula.f...@gmail.com> >>> wrote: >>> >>>> In that case I would apply a map to wrap in some common type, like a n >>>> Either<t1,t2> before the union. >>>> >>>> And then in the coflatmap you can unwrap it. >>>> On Wed, Aug 19, 2015 at 9:50 AM Welly Tambunan <if05...@gmail.com> >>>> wrote: >>>> >>>>> Hi Gyula, >>>>> >>>>> Thanks. >>>>> >>>>> However update1 and update2 have a different type. Based on my >>>>> understanding, i don't think we can use union. How can we handle this one >>>>> ? >>>>> >>>>> We like to create our event strongly type to get the domain language >>>>> captured. >>>>> >>>>> >>>>> Cheers >>>>> >>>>> On Wed, Aug 19, 2015 at 2:37 PM, Gyula Fóra <gyula.f...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hey, >>>>>> >>>>>> One input of your co-flatmap would be model updates and the other >>>>>> input would be events to check against the model if I understand >>>>>> correctly. >>>>>> >>>>>> This means that if your model updates come from more than one stream >>>>>> you need to union them into a single stream before connecting them with >>>>>> the >>>>>> event stream and applying the coatmap. >>>>>> >>>>>> DataStream updates1 = .... >>>>>> DataStream updates2 = .... >>>>>> DataStream events = .... >>>>>> >>>>>> events.connect(updates1.union(updates2).broadcast()).flatMap(...) >>>>>> >>>>>> Does this answer your question? >>>>>> >>>>>> Gyula >>>>>> >>>>>> >>>>>> On Wednesday, August 19, 2015, Welly Tambunan <if05...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Gyula, >>>>>>> >>>>>>> Thanks for your response. >>>>>>> >>>>>>> However the model can received multiple event for update. How can we >>>>>>> do that with co-flatmap as i can see the connect API only received >>>>>>> single >>>>>>> datastream ? >>>>>>> >>>>>>> >>>>>>> > ... while external model updates would be tricky to keep >>>>>>> consistent. >>>>>>> Is that still the case if the Operator treat the external model as >>>>>>> read-only ? We create another stream that will update the external model >>>>>>> separately. >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> On Wed, Aug 19, 2015 at 2:05 PM, Gyula Fóra <gyf...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey! >>>>>>>> >>>>>>>> I think it is safe to say that the best approach in this case is >>>>>>>> creating a co-flatmap that will receive updates on one input. The >>>>>>>> events >>>>>>>> should probably be broadcasted in this case so you can check in >>>>>>>> parallel. >>>>>>>> >>>>>>>> This approach can be used effectively with Flink's checkpoint >>>>>>>> mechanism, while external model updates would be tricky to keep >>>>>>>> consistent. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Gyula >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 19, 2015 at 8:44 AM Welly Tambunan <if05...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> We have a streaming computation that required to validate the data >>>>>>>>> stream against the model provided by the user. >>>>>>>>> >>>>>>>>> Right now what I have done is to load the model into flink >>>>>>>>> operator and then validate against it. However the model can be >>>>>>>>> updated and >>>>>>>>> changed frequently. Fortunately we always publish this event to >>>>>>>>> RabbitMQ. >>>>>>>>> >>>>>>>>> I think we can >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. Create RabbitMq listener for model changed event from >>>>>>>>> inside the operator, then update the model if event arrived. >>>>>>>>> >>>>>>>>> But i think this will create race condition if not handle >>>>>>>>> correctly and it seems odd to keep this >>>>>>>>> >>>>>>>>> 2. We can move the model into external in external memory >>>>>>>>> cache storage and keep the model up to date using flink. So the >>>>>>>>> operator >>>>>>>>> will retrieve that from memory cache >>>>>>>>> >>>>>>>>> 3. Create two stream and using co operator for managing the >>>>>>>>> shared state. >>>>>>>>> >>>>>>>>> >>>>>>>>> What is your suggestion on keeping the state up to date from >>>>>>>>> external event ? Is there some kind of best practice for maintaining >>>>>>>>> model >>>>>>>>> up to date on streaming operator ? >>>>>>>>> >>>>>>>>> Thanks a lot >>>>>>>>> >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Welly Tambunan >>>>>>>>> Triplelands >>>>>>>>> >>>>>>>>> http://weltam.wordpress.com >>>>>>>>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Welly Tambunan >>>>>>> Triplelands >>>>>>> >>>>>>> http://weltam.wordpress.com >>>>>>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Welly Tambunan >>>>> Triplelands >>>>> >>>>> http://weltam.wordpress.com >>>>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>>>> >>>> >>> >>> >>> -- >>> Welly Tambunan >>> Triplelands >>> >>> http://weltam.wordpress.com >>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>> >> > > > -- > Welly Tambunan > Triplelands > > http://weltam.wordpress.com > http://www.triplelands.com <http://www.triplelands.com/blog/> > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>
