Hi Nic.

I do not have a solution (sorrrry), but have seen something similar. And have 
complained about it, already. Look up my mail in the archives.

I am also deploying our session cluster using Flink Kubernetes operator. I was 
setting “execution.checkpointing.interval: 300000”. This should set a global 
value for periodic checkpoints. They do work in Docker Compose.

I can see the value is read and accepted in Job Manager, expect the same in 
Task Manager – haven’t checked the logs. However, all running jobs are stating 
that “Periodic Checkpointing Disabled”. If I alter the job to set periodic 
checkpointing on its execution environment, then it works.

So, it seems like JM or TM is not passing certain options to our jobs. Mind 
you, all other checkpointing options were passed into the execution environment 
of the job.

Is this a bug or a feature?

Nix,

From: Nic Townsend <nictowns...@uk.ibm.com>
Date: Friday, January 10, 2025 at 4:22 PM
To: user@flink.apache.org <user@flink.apache.org>
Subject: table.exec.source.idle-timeout support
Hi, I’m deploying Flink 1.19 via the k8s operator.

I’m setting `table.exec.source.idle-timeout: 30 s` in the 
`spec.flinkConfiguration` section of the FlinkDeployment CR.

The Flink UI is showing the JobManager has been configured with the value and 
the JM and TM logs both show `INFO [] - Loading configuration property: 
table.exec.source.idle-timeout, 30 s`.

However, when I run a simple SQL job (select * from <kafkatable>) – where the 
Kafka topic has 3 partitions,  the watermark does not appear to update when a 
partition is idle (confirmed using the job graph UI).

If I instead use the option `'scan.watermark.idle-timeout'=30 s ` in the CREATE 
TABLE then the watermark does increase when the partition goes idle.

I can’t find an error in the JobManager to suggest the config option is invalid 
or malformed, and I’m not seeing a Jira issue that reflects this.

So I feel like I must be doing something wrong, but would appreciate any 
advice, thank you!

--

Nic Townsend
IBM Event Processing
Senior Engineer / Technical Lead

Slack: @nictownsend


Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road, 
Winchester, Hampshire SO21 2JN

Reply via email to