Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-30 Thread Till Rohrmann
sensor events") > > The SensorInputPBDeserializationSchema is a basic KafkaRecordDeserializer > that does SensorInputPB.parseFrom(record.value()) and finally > collector.collect(v) > > From here on we're doing a keyed windowed aggregation with .keyBy(...). > window(Even

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Lars Skjærven
: Thursday, April 29, 2021 18:44 To: Lars Skjærven Cc: Becket Qin ; user@flink.apache.org Subject: Re: KafkaSourceBuilder causing invalid negative offset on checkpointing Thanks for the additional information Lars. Could you maybe also share the full stack traces of the errors you see when the

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Till Rohrmann
ssionWindows.withGap(Time.seconds(60))).aggregate(new > SensorEventAggregator) > > L > > > -- > *From:* Till Rohrmann > *Sent:* Thursday, April 29, 2021 09:16 > *To:* Lars Skjærven ; Becket Qin < > becket....@gmail.com> > *Cc:* user@fl

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Lars Skjærven
rom: Till Rohrmann Sent: Thursday, April 29, 2021 09:16 To: Lars Skjærven ; Becket Qin Cc: user@flink.apache.org Subject: Re: KafkaSourceBuilder causing invalid negative offset on checkpointing Hi Lars, The KafkaSourceBuilder constructs the new KafkaSource which has not been fully hardene

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Till Rohrmann
Hi Lars, I think this is a duplicate message. Let's continue the discussion on your original message. Cheers, Till On Wed, Apr 28, 2021 at 8:50 PM Lars Skjærven wrote: > Hello, > I ran into an issue when using the new KafkaSourceBuilder (running Flink > 1.12.2, scala 2.12.13, on ververica plat

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Till Rohrmann
Hi Lars, The KafkaSourceBuilder constructs the new KafkaSource which has not been fully hardened in 1.12.2. In fact, it should not be documented yet. I think you are running into an instability/bug of. The new Kafka source should be hardened a lot more in the 1.13.0 release. Could you tell us exa