Got it, Thanks a lot Fabian. Looking forward to seeing your book. Best, Chengzhi
On Thu, Apr 26, 2018 at 4:02 PM, Fabian Hueske <fhue...@gmail.com> wrote: > You can also merge all three types into an nary-Either type and union all > three inputs together. > However, Flink only supports a binary Either, so you'd have to implement a > custom TypeInformation and TypeSerializer to make that work. > > Best, Fabian > > 2018-04-26 20:44 GMT+02:00 Chengzhi Zhao <w.zhaocheng...@gmail.com>: > >> Thanks Fabian for the explanation. >> >> If I have data with different schemas, it seems the only option I have is >> to use connect to perform joins (inner, outer), is there any operators that >> can put more than two streams together (all different schema)? >> >> Best, >> Chengzhi >> >> On Thu, Apr 26, 2018 at 6:05 AM, Fabian Hueske <fhue...@gmail.com> wrote: >> >>> Hi Chengzhi, >>> >>> Functions in Flink are implemented in a way to preserve the timestamps >>> of elements or assign timestamps which are aligned with the existing >>> watermarks. >>> For example, the result of a time window aggregation has the end >>> timestamp of the window as a timestamp and records emitted by the onTimer() >>> method have the timestamp of the timer as a record timestamp. >>> So unless you fiddle with internal APIs to reset the record timestamps >>> of elements, you don't need to worry about generating new watermarks. >>> >>> Best, Fabian >>> >>> 2018-04-25 20:20 GMT+02:00 Chengzhi Zhao <w.zhaocheng...@gmail.com>: >>> >>>> Hi, everyone, >>>> >>>> I am trying to do some join-like pipeline using flink connect operator >>>> and CoProcessFunction, I have use case that I need to connect 3+ streams. >>>> So I am having something like this: >>>> >>>> A >>>> ===> C >>>> B ==> E >>>> D >>>> >>>> So two streams A and B connect at first with 3 hours late on low >>>> watermark, after data has been emitted (the output C stream), a new stream >>>> D connect to C and emitted E as final output. I was wondering how the >>>> downstream watermark should be defined. Should I give C stream a new >>>> watermark for 3 hours delay again? or when I connect stream D, everything >>>> will be 6 hours late on low watermark. >>>> >>>> I am using BoundedOutOfOrdernessGenerator[1] with maxOutOfOrderness 3 >>>> hours >>>> >>>> Thanks for your tips and help in advance. >>>> >>>> Best, >>>> Chengzhi >>>> >>>> [1]https://ci.apache.org/projects/flink/flink-docs-release-1 >>>> .4/dev/event_timestamps_watermarks.html#with-periodic-watermarks >>>> >>> >>> >> >
