Re: Implementing a low level join

Thanks for the advice. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Aug 15, 2019 at 9:59 AM Fabian Hueske wrote: > Hi Felipe, > > No, this is not possible (with reasonable effort). > A che

Re: Implementing a low level join

Hi Felipe, No, this is not possible (with reasonable effort). A checkpoint would be the right time to do the change, but this would be very involved, IMO. As I said, you need a global decision to switch the strategy. This could be communicated with a checkpoint barrier. Then all operator would nee

Re: Implementing a low level join

Hi Fabian, thanks for jumping within this thread. Do you think there is possible to extend any join generic operator in order to make it a little dynamic? I was thinking that after I process a checkpoint I can change the join strategy. and if you do, do you have any toy example of this? Thanks,

Re: Implementing a low level join

Hi, Just to clarify. You cannot dynamically switch the join strategy while a job is running. What Hequn suggested was to have a util method Util.joinDynamically(ds1, ds2) that chooses the join strategy when the program is generated (before it is submitted for execution). The problem is that distr

Re: Implementing a low level join

I see, I am gonna try this. Thanks Hequn *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Aug 15, 2019 at 4:01 AM Hequn Cheng wrote: > Hi Felipe, > > If I understand correctly, you also have t

Re: Implementing a low level join

Hi Felipe, If I understand correctly, you also have to decide whether to broadcast the datastream from the right side before performing the function? One option is you can add a Util method to join dynamically, e.g., Util.joinDynamically(ds1, ds2). In the util method, you can implement your own s

Re: Implementing a low level join

Hi Hequn, I am implementing the broadcast and the regular join. As you said I need different functions. My question is more about if I can have an operator which decides beteween broadcast and regular join dynamically. I suppose I will have to extend the generic TwoInputStreamOperator in Flink. Do

Re: Implementing a low level join

Hi Felipe, > I want to implement a join operator which can use different strategies for joining tuples. Not all kinds of join strategies can be applied to streaming jobs. Take sort-merge join as an example, it's impossible to sort an unbounded data. However, you can perform a window join and use t

Implementing a low level join

Hi all, I want to implement a join operator which can use different strategies for joining tuples. I saw that with CoProcessFunction I am able to implement low-level joins [1]. However, I do know how to decide between different algorithms to join my tuples. On the other hand, to do a broadcast jo