Re: Broadcast to all the other operators

2017-11-09 Thread Ladhari Sadok
ding to different use cases. > > Best Regards, > Tony Wei > > > 2017-11-09 17:27 GMT+08:00 Ladhari Sadok : > >> >> -- Forwarded message -- >> From: Ladhari Sadok >> Date: 2017-11-09 10:26 GMT+01:00 >> Subject: Re: Broadcast to all th

Re: Broadcast to all the other operators

2017-11-09 Thread Tony Wei
: Ladhari Sadok > Date: 2017-11-09 10:26 GMT+01:00 > Subject: Re: Broadcast to all the other operators > To: Tony Wei > > > Thanks Tony for your very fast answer , > > Yes it resolves my problem that way, but with flatMap I will get > Tuple2 always in the processing fun

Fwd: Broadcast to all the other operators

2017-11-09 Thread Ladhari Sadok
-- Forwarded message -- From: Ladhari Sadok Date: 2017-11-09 10:26 GMT+01:00 Subject: Re: Broadcast to all the other operators To: Tony Wei Thanks Tony for your very fast answer , Yes it resolves my problem that way, but with flatMap I will get Tuple2 always in the processing

Re: Broadcast to all the other operators

2017-11-09 Thread Tony Wei
Hi Sadok, What I mean is to keep the rules in the operator state. The event in Rule Stream is just the change log about rules. For more specific, you can fetch the rules from Redis in the open step of CoFlatMap and keep them in the operator state, then use Rule Stream to notify the CoFlatMap to 1.

Re: Broadcast to all the other operators

2017-11-08 Thread Ladhari Sadok
Thank you for the answer, I know that solution, but I don't want to stream the rules all time. In my case I have the rules in Redis and at startup of flink they are loaded. I want to broadcast changes just when it occurs. Thanks. Le 9 nov. 2017 7:51 AM, "Tony Wei" a écrit : > Hi Sadok, > > Sin

Re: Broadcast to all the other operators

2017-11-08 Thread Tony Wei
Hi Sadok, Since you want to broadcast Rule Stream to all subtasks, it seems that it is not necessary to use KeyedStream. How about use broadcast partitioner, connect two streams to attach the rule on each record or imply rule on them directly, and do the key operator after that? If you need to do

Broadcast to all the other operators

2017-11-08 Thread Ladhari Sadok
Hello, I'm working on Rules Engine project with Flink 1.3, in this project I want to update some keyed operator state when external event occurred. I have a Datastream of updates (from kafka) I want to broadcast the data contained in this stream to all keyed operator so I can change the state in