Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-30 Thread Till Rohrmann
sensor events") > > The SensorInputPBDeserializationSchema is a basic KafkaRecordDeserializer > that does SensorInputPB.parseFrom(record.value()) and finally > collector.collect(v) > > From here on we're doing a keyed windowed aggregation with .keyBy(...). > window(Even

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Lars Skjærven
: Thursday, April 29, 2021 18:44 To: Lars Skjærven Cc: Becket Qin ; user@flink.apache.org Subject: Re: KafkaSourceBuilder causing invalid negative offset on checkpointing Thanks for the additional information Lars. Could you maybe also share the full stack traces of the errors you see when the

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Till Rohrmann
ssionWindows.withGap(Time.seconds(60))).aggregate(new > SensorEventAggregator) > > L > > > -- > *From:* Till Rohrmann > *Sent:* Thursday, April 29, 2021 09:16 > *To:* Lars Skjærven ; Becket Qin < > becket....@gmail.com> > *Cc:* user@fl

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Lars Skjærven
rom: Till Rohrmann Sent: Thursday, April 29, 2021 09:16 To: Lars Skjærven ; Becket Qin Cc: user@flink.apache.org Subject: Re: KafkaSourceBuilder causing invalid negative offset on checkpointing Hi Lars, The KafkaSourceBuilder constructs the new KafkaSource which has not been fully hardene

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Till Rohrmann
Hi Lars, I think this is a duplicate message. Let's continue the discussion on your original message. Cheers, Till On Wed, Apr 28, 2021 at 8:50 PM Lars Skjærven wrote: > Hello, > I ran into an issue when using the new KafkaSourceBuilder (running Flink > 1.12.2, scala 2.12.13, on ververica plat

Re: KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-29 Thread Till Rohrmann
Hi Lars, The KafkaSourceBuilder constructs the new KafkaSource which has not been fully hardened in 1.12.2. In fact, it should not be documented yet. I think you are running into an instability/bug of. The new Kafka source should be hardened a lot more in the 1.13.0 release. Could you tell us exa

KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-28 Thread Lars Skjærven
Hello, I ran into an issue when using the new KafkaSourceBuilder (running Flink 1.12.2, scala 2.12.13, on ververica platform in a container with java 8). Initially it generated warnings on kafka configuration, but the job was able to consume and produce messages. The configuration 'client.id.prefi

KafkaSourceBuilder causing invalid negative offset on checkpointing

2021-04-28 Thread Lars Skjærven
Hello, I ran into some issues when using the new KafkaSourceBuilder (running Flink 1.12.2, scala 2.12.13, on ververica platform in a container with java 8). Initially it generated warnings on kafka configuration, but the job was able to consume and produce messages. The configuration 'client.i