Problem with KafkaSource and watermark idleness

2022-08-09 Thread Yan Shen
Hi, I am using a org.apache.flink.connector.kafka.source.KafkaSource with a watermark strategy like this: WatermarkStrategy.forMonotonousTimestamps().withIdleness(Duration.ofSeconds(10)) I noticed that after a short while all the partitions seem to be marked as idle even though there are message

Re: Issues with Flink scheduler?

2022-08-09 Thread Weihua Hu
Hi, Hemanga Could not acquire the minimum required resources - This log just shows that there are not enough task managers to schedule your job. Referring to your description, maybe there was some problem with creating the task manager. Maybe you can check the status of the task manager pod wh

Re: Pojo state schema evolution not working correctly

2022-08-09 Thread tao xiao
Not exactly reproduciable. I cannot reproduce the problem in a test environment with the same setting. This is only reproducible in PRD. I guess it has something to do with the state itself. Where you added the debugging log ? In RocksDBListState#get() ? > I added the debug log in *org.apache.fli