Hey Guys, I have observed a weird behavior on using the Temporal Table Join and the way it pushes the Watermark forward. Generally, I think the question is *When is the Watermark pushed forward by the Temporal Table Join?*
The issue I have noticed is that Watermark seems to be pushed forward even if elements are not generated, is that the expected behavior? I have created a simple test that takes two streams rates & ccyCodes. Now, I do temporal table Join of rates & ccyCodes in two ways : 1) SELECT ccyIsoCode, ccyIsoName, rate, rates_ts as ratesLong | FROM RatesTable, LATERAL TABLE(ccyTable(rates_rowtime)) | WHERE ccyIsoCode = ratesCcyIsoCode 2) SELECT ccyIsoCode, ccyIsoName, rate, rates_rowtime as ratesTs | FROM RatesTable, LATERAL TABLE(ccyTable(rates_rowtime)) | WHERE ccyIsoCode = ratesCcyIsoCode So, as You can see the only difference is the fact the in the first one I am selecting the field that is the timestamp but wasn't marked with *rowtime* in the second one I am selecting the actual *rowtime.* Now, I for the first method I need to reassign timestamps and watermarks, which I do. Finally, I create time windows of size let's say 7000 miliseconds and print the results. Now the unexpected behaviour I am facing is the fact that, say I create four artificial records for rates that are joined correctly with ccy with timestamps (1000, 5000, 8000, 20000) - For the first method there is only one window generated with elements [1000, 5000] - But for the second method with *rowtime* there are two different windows [1000, 5000] and [8000], this basically means that the watermark for 20000 was generated. Is that the expected behavior ?? I was quite surprised that selecting different field can actually yied different results in terms of Watermarking. Best Regards, Dom.
