Glad it has worked! So sounds like data has been dropped as they are considered late data and `.withAllowedLateness()` make the data emitted.
-Rui On Thu, Aug 27, 2020 at 10:09 AM Minreng Wu <wumr...@gmail.com> wrote: > Hi Rui, > > Thanks for your advice! > > After reading Chapter 2&3 of *Streaming Systems* and some other > materials, eventually I make it work! It indeed turned out to be an issue > of not setting the trigger correctly. Previously, I didn't set the trigger > & watermark so it would use the default settings. After I added > `.withAllowedLateness()`, it can correctly materialize the window output as > expected. Thank you so much for your help! > > Thanks & Regards, > Minreng > > > On Mon, Aug 24, 2020 at 1:58 PM Rui Wang <ruw...@google.com> wrote: > >> Hi, >> >> I checked the query in your SO question and I think the SQL usage is >> correct. >> >> My current guess is that the problem is how does watermark generate and >> advance in KafkaIO. It could be either the watermark didn't pass the end of >> your SQL window for aggregation or the data was lagging behind the >> watermark so they are considered late data. >> >> One way to verify it is you can try to use TestStream as the source to >> evaluate your pipeline and see whether it works well. >> >> -Rui >> >> On Mon, Aug 24, 2020 at 11:06 AM Minreng Wu <wumr...@gmail.com> wrote: >> >>> Hi contributors, >>> >>> Sorry to bother you! I met a problem when I was trying to apply a >>> windowing aggregation Beam SQL query to a Kafka input source. >>> >>> The details of the question are in the following link: >>> https://stackoverflow.com/questions/63566057/how-to-integrate-beam-sql-windowing-query-with-kafkaio. >>> And the version of the Beam Java SDK I used is *2.23.0* >>> >>> Really appreciate your help and advice! Stay safe and happy! >>> >>> Thanks and regards, >>> Minreng >>> >>
