Hi Averell, I think your original solution is the right one, given your requirements. I don't think it is over complicated.
As for the memory concerns, there is no bult-in mechanism for backpressure/alignment based on event time. The community did take that into consideration when discussing the new source interface though[1]. But as Hequn already mentioned if you use the RocksDBStateBackend, the amount of space is limited by the disk space. Moreover you could add some safety timer, that would fire every x minutes and clear the oldest entries. Best, Dawid [1] https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E On 19/04/2019 05:15, Averell wrote: > Thank you Hecheng. > > I just tried to use Table API as your suggestion, and it almost worked (it > worked with two issues here below): > - I only get the output when my event-time watermark goes pass the end > of the tumbling window. But, because I know that there are maximum 2 records > per window (one from each stream), I would like to collect my output record > as soon as I received two input records. With low-level-API, I believe I can > do this with Trigger. Can I achieve a similar result with Table API? > - In the UDAggF document, I saw a recommendation to use Java instead of > Scala. Does this apply to the low-level-API functions as well? > > Thanks and best regards, > Averell > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
signature.asc
Description: OpenPGP digital signature
