Hi, AFAIK there is no special watermark generation logic for temporal table join operator. Could you share your example's codes then I can help to analyze and debug?
Best, Kurt On Tue, Mar 17, 2020 at 9:53 PM Dominik Wosiński <wos...@gmail.com> wrote: > Hey Guys, > I have observed a weird behavior on using the Temporal Table Join and the > way it pushes the Watermark forward. Generally, I think the question is > *When > is the Watermark pushed forward by the Temporal Table Join?* > > The issue I have noticed is that Watermark seems to be pushed forward even > if elements are not generated, is that the expected behavior? > > I have created a simple test that takes two streams rates & ccyCodes. Now, > I do temporal table Join of rates & ccyCodes in two ways : > 1) SELECT ccyIsoCode, ccyIsoName, rate, rates_ts as ratesLong > > | FROM RatesTable, LATERAL TABLE(ccyTable(rates_rowtime)) > | WHERE ccyIsoCode = ratesCcyIsoCode > > > > 2) SELECT ccyIsoCode, ccyIsoName, rate, rates_rowtime as ratesTs > > | FROM RatesTable, LATERAL TABLE(ccyTable(rates_rowtime)) > | WHERE ccyIsoCode = ratesCcyIsoCode > > So, as You can see the only difference is the fact the in the first one I > am selecting the field that is the timestamp but wasn't marked with > *rowtime* in the second one I am selecting the actual *rowtime.* > > Now, I for the first method I need to reassign timestamps and watermarks, > which I do. Finally, I create time windows of size let's say 7000 > miliseconds and print the results. Now the unexpected behaviour I am facing > is the fact that, say I create four artificial records for rates that are > joined correctly with ccy with timestamps (1000, 5000, 8000, 20000) > > - For the first method there is only one window generated with elements > [1000, 5000] > - But for the second method with *rowtime* there are two different > windows [1000, 5000] and [8000], this basically means that the watermark > for 20000 was generated. > > Is that the expected behavior ?? I was quite surprised that selecting > different field can actually yied different results in terms of > Watermarking. > > Best Regards, > Dom. >
