Hi, I'm trying to investigate a SQL job using a time-windowed join that is exhibiting a large, growing state. The join syntax is most similar to the interval join ( https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/joins.html ).
A few questions: 1. Am I correct in understanding that State TTL is generally not applicable for TableAPI&SQL? So we cannot use State TTL to limit state size for a join? 2. It seems that Flink should be able to expire state even without explicit settings based on this: "In TableAPI&SQL and DataStream, the window aggregation and time-windowed join will clear expired state using Timers which is triggered by watermark." ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/join-state-TTL-td34760.html ) To clarify: Does the above mean that Flink is expected to detect expired state and clear it without explicit configuration to allow it to do so? 3. I've looked into setting the idle state retention time. From what I can understand, this particular setting is appropriate for my use case. "TableConfig#setIdleStateRetentionTime in TableAPI&SQL is a job level configuration which will enable state ttl for all non-time-based operator states." ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/join-state-TTL-td34760.html ) To clarify: Would enabling this setting control state growth? Is this only available for blink planner? Currently we are using the StreamPlanner. Is there any way to ensure that idle state has limited retention for applications using the StreamPlanner? Thanks ahead, Sofya
