Hi. For local and tests development, I want to flush the events in my system to make sure I'm processing everything. My watermark does not progress to finish all of the data.
What's the best practice for local development or tests? If I use idle sources for 1 Kafka partition, this appears broken. I'm guessing there is logic to prevent removing an idle partition if it's the only partition. Is there a version of this I can enable for local development that supports 1 partition? I see this tech talk. Are there other talks to watch? https://www.youtube.com/watch?v=bQmz7JOmE_4&feature=youtu.be Do I need to write my own watermark generator? Or change my test data to have a way of generating watermarks? I've tried a few variants of the following source code. The watermark doesn't progress in the operator right after creating the source. SingleOutputStreamOperator<T> viewInput = env.addSource(...) .uid("source-view") .assignTimestampsAndWatermarks( WatermarkStrategy.<T>forBoundedOutOfOrderness(Duration.ofSeconds(1)).withIdleness(Duration.ofMinutes(1)));
