Hi Averell, As far as these two solutions are concerned, I think you can only choose option 2, because as you have stated, the current Flink DataStream API does not support the replacement of one of the input stream types of CoFlatMapFunction. Another choice:
1. Split it into two separate jobs. But in comparison, I still think that Option 2 is better. 2. Since you said that stream_c is slower and has fewer updates, if it is not very large, you can store it in the RDBMS and then join it with stream_a and stream_b respectively (using CoFlatMapFunction as well). I think you should give priority to your option 2. Thanks, vino. Averell <lvhu...@gmail.com> 于2018年8月15日周三 下午1:51写道： > Hi, > > I have stream_A of type "Dog", which needs to be transformed using data > from > stream_C of type "Name_Mapping". As stream_C is a slow one (data is not > being updated frequently), to do the transformation I connect two streams, > do a keyBy, and then use a RichCoFlatMapFunction in which mapping data from > stream_C is saved into a State (flatMap1 generates 1 output, while flatMap2 > is just to update State table, not generating any output). > > Now I have another stream B of type "Cat", which also needs to be > transformed using data from stream_C. After that transformation, > transformed_B will go through a completely different pipeline from > transformed A. > > I can see two approaches for this: > 1. duplicate stream_C and the RichCoFlatMapFunction and apply on stream_B > 2. create a new stream D of type "Animal", transform it with C, then split > the result into two streams using split/select using case class pattern > matching. > > My question is which option should I choose? > With option 1, at least I need to maintain two State tables, let alone the > cost for duplicating stream (I am not sure how expensive this is in term of > resource), and the requirement on duplicating the CoFlatMapFunction (*). > With option 2, there's additional cost coming from unioning, > splitting/selecting, and type-casting at the final streams. > Is there any better option for me? > > Thank you very much for your support. > Regards, > Averell > > (*) I am using Scala, and I tried to create a RichCoFlatMapFunction of type > [Animal, Name_Mapping] but it cannot be used for a stream of [Dog, > Name_Mapping] or [Cat, Name_Mapping]. Thus I needed to duplicate the > Function as well. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
