I have pipeline defined . pipeline does source(kafka) ,transformation,dedup and persisting to DB . [image: image.png]
Before reaching to DB task lots of transformation is applied on the pipeline Therefore want to validate the record with raw json message which is available in kafka with the transformed record. Hence I want to know How to do that in flink. On Tue, Jan 29, 2019 at 6:43 PM miki haiat <miko5...@gmail.com> wrote: > If c1 and c2 are listing to the same topic they will consume the same > data . > so i cant understand this > >> these two streams one(c2) is fast and other(c1) > > > > > > On Tue, Jan 29, 2019 at 2:44 PM Selvaraj chennappan < > selvarajchennap...@gmail.com> wrote: > >> Team, >> >> I have two kafka consumer for same topic and want to join second stream >> to first after couple of subtasks computation in the first stream then >> validate the record . KT - C1 ,C2 >> >> KT - C1 - Transformation(FlatMap) - Dedup - Validate --ifvalidsave it to >> DB >> -C2 - Process -------------------------- >> >> if record is invalid then save it to Error topic . >> >> How do I merge these two streams one(c2) is fast and other(c1) is little >> slow (two levels of computation) ? >> Same record is flowing from C1-Flatmap-FlatMap and other consumer C2 . I >> have to validate that record based on the rules. >> Please find the attached image herewith reference. >> [image: two-stream.png] >> >> -- >> >> >> >> >> >> Regards, >> Selvaraj C >> > -- Regards, Selvaraj C
