Re: Kafka source with multiple partitions loses data during savepoint recovery

2022-03-18 Thread Sharon Xie
Thanks Qingsheng, I'm using Flink SQL. I'll need to dig a bit deeper to see which Flink consumer class is actually used. I didn't find the log you linked. However I found the following log from the task manager. Just a note, in this test, I sent 10 records to the source topic which has 10 partiti

Re: Kafka source with multiple partitions loses data during savepoint recovery

2022-03-18 Thread Qingsheng Ren
Hi Sharon, Could you check the log after starting the job with savepoint? If you have INFO log enabled you will get an entry “Consumer subtask {} will start reading {} partitions with offsets in restored state: {}” [1] in the log, which shows the starting offset of partitions. This might be he

Kafka source with multiple partitions loses data during savepoint recovery

2022-03-17 Thread Sharon Xie
Hi, I'm seeing an odd behavior for Kafka source where some records are dropped during recovery. My test set up is: Kafka source topic -> pass through flink job -> Kafka sink topic There are 10 partitions in the source & sink topics. Test Steps * Start the flink job, send 5 records (first batch)