Re: KafkaSource and Event Time in Message Payload

2022-12-15 Thread Niklas Wilcke
Hi Martijn, thanks for referencing both related FLIPs and providing a recommendation. That was already helpful. I need to find some time to further investigate this topic. So far I agree that it might be the most reasonable approach to just use the Kafka timestamp to carry the event time. Than

Re: KafkaSource and Event Time in Message Payload

2022-12-13 Thread Martijn Visser
Hi Niklas, On your confirmations: a1) The default behaviour is documented at https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#event-time-and-watermarks - Flink uses the timestamp embedded in the Kafka ConsumerRecord. a2) I'm not 100% sure: since Flink 1.15, t

KafkaSource and Event Time in Message Payload

2022-12-08 Thread Niklas Wilcke
Hi Flink Community, I have a few questions regarding the new KafkaSource and event time, which I wasn't able to answer myself via checking the docs, but please point me to the right pages in case I missed something. I'm not entirely whether my knowledge entirely holds for the new KafkaSource, b