Hi, When migrating Stream API based Flink application from 1.9.2 to 1.11.1 the watermark generation has issues with file source alone. It works well with Kafka source.
With 1.9.2 a custom watermark generator implementation of AssignerWithPeriodicWatermarks was to be used. Starting 1.11.1 it is deprecated and to be replaced with WatermarkStrategy (that combines both WatermarkGenerator and TimestampAssigner). With Flink 1.11.1 when using Kafka source both the above options (i.e. old AssignerWithPeriodicWatermarks and new WatermarkStrategy) work perfectly well but with file source none of them works. The watermark assigner never increments the watermarks resulting in stateful operators not clearing their state ever, leading to erroneous results and continuously increasing memory usage. Same code works well with Kafka source. Is this a known issue? If so, any fix planned shortly? A side note (and probably a candidate for separate email, but I will write it here) even checkpoints do not work with File Source since 1.9.2 and it is still the problem with 1.11.1. Just wondering if File source with stream API is not a priority in Flink development? If so we can rethink our sources. Thanks & regards, Arti
