Re: Unexpected behavior from interval join in Flink

2019-06-24 Thread Fabian Hueske
Ah, that's great! Thanks for letting us know :-) Am Mo., 24. Juni 2019 um 11:33 Uhr schrieb Wouter Zorgdrager < w.d.zorgdra...@tudelft.nl>: > Hi Fabian, > > Thanks for your reply. I managed to resolve this issue. Actually this > behavior was not so unexpected, I messed up using xStream as a 'base

Re: Unexpected behavior from interval join in Flink

2019-06-24 Thread Wouter Zorgdrager
Hi Fabian, Thanks for your reply. I managed to resolve this issue. Actually this behavior was not so unexpected, I messed up using xStream as a 'base' while I needed to use yStream as a 'base'. I.e. yStream.element - 60 min <= xStream.element <= yStream.element + 30 min. Interchanging both datastr

Re: Unexpected behavior from interval join in Flink

2019-06-24 Thread Fabian Hueske
Hi Wouter, Not sure what is going wrong there, but something that you could try is to use a custom watemark assigner and always return a watermark of 0. When the source finished serving the watermarks, it emits a final Long.MAX_VALUE watermark. Hence the join should consume all events and store th

Re: Unexpected behavior from interval join in Flink

2019-06-21 Thread Wouter Zorgdrager
Anyone some leads on this issue? Have been looking into the IntervalJoinOperator code, but that didn't really help. My intuition is that it is rejected because of lateness, however that still confuses me since I'm sure that both datastreams have monotonic increasing timestamps. Thx, Wouter Op ma

Unexpected behavior from interval join in Flink

2019-06-17 Thread Wouter Zorgdrager
Hi all, I'm experiencing some unexpected behavior using an interval join in Flink. I'm dealing with two data sets, lets call them X and Y. They are finite (10k elements) but I interpret them as a DataStream. The data needs to be joined for enrichment purposes. I use event time and I know (because