Idle source configuration per topic with the Kafka Table API connector

2021-05-29 Thread Svend
Hi everyone, My Flink streaming application consumes several Kafka topics, one of which receiving traffic in burst once per day. I would like that topic not to hold back the progress of the watermark. Most of my code is currently using the SQL API and in particular the Table API Kafka connecto

Re: Flink in k8s operators list

2021-05-29 Thread Svend
Hi Ilya, At my company we're currently using the GCP k8s operator (2nd on your list). Our usage is very moderate, but so far it works great for us. We appreciate that when upgrading the application, it triggers automatically a savepoint during shutdown and resumes from it when restarting. It al

Re: Idle source configuration per topic with the Kafka Table API connector

2021-05-29 Thread JING ZHANG
Hi Svend, Your solution could work well in Flink 1.13.0 and Flink 1.13.0+ because those version provides many related improvements. > as per [1] Yes, "table.exec.source.idle-timeout" is not table-level parameter, but a global parameter, It would apply to all those table sources which with watermar

Error about "Rejected TaskExecutor registration at the ResourceManger"

2021-05-29 Thread Kai Fu
Hi team, We encountered an issue during recovery from checkpoint. It's recovering because the downstream Kafka sink is full for a while and the job is failed and keeps trying to recover(The downstream is full for about 4 hours). The job cannot recover from checkpoint successfully even if after we