Hi,

The difference is that the *table.exec.source.idle-timeout *is used for
dealing with source idleness[1]. It is a problem that a watermark cannot
advance if some of the partition become idle (do not produce any
records). Watermark is always the minimum of watermarks of all input
partitions. The setting makes flink ignore certain partitions in the
calculation after the time threshold is reached.

The IdleStateRetention is Table API specific. As described in the link
you provided it removes entries from a state for keys that were not seen
for a given time threshold.

As for your issue, I'd recommend first investigating what is causing the
state to grow. Is it ever growing keyspace? Is it that a watermark does
not progress (this should manifest in results as well). Or is it
something else.

Best,

Dawid
**


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/event_timestamps_watermarks.html#dealing-with-idle-sources

On 25/01/2021 20:12, Dcosta, Agnelo (HBO) wrote:
>
> Hi,
>
> What is the difference between *table.exec.source.idle-timeout* and
> *setIdleStateRetentionTime* ?
>
> table.exec.source.idle-timeout:
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/config.html#table-exec-source-idle-timeout
>
>  
>
> setIdleStateRetentionTime:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/query_configuration.html#idle-state-retention-time
>
>  
>
> Some context:
> Hi we are using flink 1.12.
> Our checkpoint size is constantly increasing once app is deployed.
> After performing a restart, checkpoint size goes back to expected size.
> Looking at actual checkpoint files generated, it seems our app is
> holding on to state/events since the time app started up.
> Based on our sql, the maximum time we would need to hold state is 10
> minutes.
>
>  
>
> This e-mail is intended only for the use of the addressees. Any
> copying, forwarding, printing or other use of this e-mail by persons
> other than the addressees is not authorized. This e-mail may contain
> information that is privileged, confidential and exempt from
> disclosure. If you are not the intended recipient, please notify us
> immediately by return e-mail (including the original message in your
> reply) and then delete and discard all copies of the e-mail. Thank you.
> HB75

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to