Hi, The difference is that the *table.exec.source.idle-timeout *is used for dealing with source idleness[1]. It is a problem that a watermark cannot advance if some of the partition become idle (do not produce any records). Watermark is always the minimum of watermarks of all input partitions. The setting makes flink ignore certain partitions in the calculation after the time threshold is reached.
The IdleStateRetention is Table API specific. As described in the link you provided it removes entries from a state for keys that were not seen for a given time threshold. As for your issue, I'd recommend first investigating what is causing the state to grow. Is it ever growing keyspace? Is it that a watermark does not progress (this should manifest in results as well). Or is it something else. Best, Dawid ** [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/event_timestamps_watermarks.html#dealing-with-idle-sources On 25/01/2021 20:12, Dcosta, Agnelo (HBO) wrote: > > Hi, > > What is the difference between *table.exec.source.idle-timeout* and > *setIdleStateRetentionTime* ? > > table.exec.source.idle-timeout: > https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/config.html#table-exec-source-idle-timeout > > > > setIdleStateRetentionTime: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/query_configuration.html#idle-state-retention-time > > > > Some context: > Hi we are using flink 1.12. > Our checkpoint size is constantly increasing once app is deployed. > After performing a restart, checkpoint size goes back to expected size. > Looking at actual checkpoint files generated, it seems our app is > holding on to state/events since the time app started up. > Based on our sql, the maximum time we would need to hold state is 10 > minutes. > > > > This e-mail is intended only for the use of the addressees. Any > copying, forwarding, printing or other use of this e-mail by persons > other than the addressees is not authorized. This e-mail may contain > information that is privileged, confidential and exempt from > disclosure. If you are not the intended recipient, please notify us > immediately by return e-mail (including the original message in your > reply) and then delete and discard all copies of the e-mail. Thank you. > HB75
signature.asc
Description: OpenPGP digital signature
