Re: Problem with KafkaSource and watermark idleness

2022-08-14 Thread Yan Shen
thinking about how to write a > failing test, and I'm wondering if this has some impact that is easily > observed. (My first thought was "How can something this basic be broken?" > but then I realized that the impact is fairly subtle.) > > David > > On Sat,

Re: Problem with KafkaSource and watermark idleness

2022-08-13 Thread Yan Shen
again) instead of repeatedly. I will try to raise an issue on this unless somebody can point out where I went wrong with this. Thanks. On Wed, Aug 10, 2022 at 1:26 PM Yan Shen wrote: > Hi, > > I am using a org.apache.flink.connector.kafka.source.KafkaSource with a > watermark s

Problem with KafkaSource and watermark idleness

2022-08-09 Thread Yan Shen
Hi, I am using a org.apache.flink.connector.kafka.source.KafkaSource with a watermark strategy like this: WatermarkStrategy.forMonotonousTimestamps().withIdleness(Duration.ofSeconds(10)) I noticed that after a short while all the partitions seem to be marked as idle even though there are message

Re: Data Stream countWindow followed by keyBy does not preserve time order

2021-11-04 Thread Yan Shen
. So feel free > to open a jira. > Best, > Guowei > > > On Wed, Nov 3, 2021 at 7:30 PM Yan Shen wrote: > >> Hi, >> >> It will complicate things a lot if we cannot assume input order of any >> operator after a keyBy. So far I only have the problem w

Re: Data Stream countWindow followed by keyBy does not preserve time order

2021-11-03 Thread Yan Shen
assume the input's order of > an operator simply. > Best, > Guowei > > > On Wed, Nov 3, 2021 at 3:10 PM Yan Shen wrote: > >> Yes, it does not happen in streaming mode. Is this considered a bug or is >> it by design? >> >> Thanks! >> >>

Re: Data Stream countWindow followed by keyBy does not preserve time order

2021-11-03 Thread Yan Shen
, because in the Batch > execution mode FLINK will "sort" the Key (this might be an unstable sort). > So would you like to experiment with the results of running with Streaming > mode and to see what happens? > > Best, > Guowei > > > On Wed, Nov 3, 2021 at 12:16

Data Stream countWindow followed by keyBy does not preserve time order

2021-11-02 Thread Yan Shen
Hi all, Can anyone advise on this? I wrote a simple test of the countWindow method (in Kotlin) as below package aero.airlab.flinkjobs.headingreminder import org.apache.flink.api.common.RuntimeExecutionMode import org.apache.flink.api.common.eventtime.WatermarkStrategy import org.apache.flink.st