Hey,

As for the MATCH_RECOGNIZE clause, I highly recommend applying a time
constraint[1]. The idle state retention time does not apply to the
MATCH_RECOGNIZE, but you can think of the time constraint as something
similar, but it is closer to the actual query logic.

If you are hitting FLINK-15160 unfortunately I don't have a good
solution for it. The only thing that comes to my mind is adding a
heartbeat event to the event stream to prune the partial matches, but I
understand it is quite invasive.

If you would be willing to help fixing the problem in FLINK, I could
also help review it and give pointers how it could be done.

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/streaming/match_recognize.html#time-constraint

On 26/01/2021 17:39, Dcosta, Agnelo (HBO) wrote:
>
> Hi Dawid, thanks for the clarification and it helps a lot.
> Reply to couple of points :
>
> what is causing the state to grow?
> We are using flink SQL and have 5 pattern match queries , 3 group by
> tumble windows. State growth over time is primarily coming from
> pattern match queries.
>
> Is it ever growing keyspace?
> Yes. By design our keyspace is ever growing. The expectation is that
> messages for one key will come in for couple of hours, then stop
> coming in. We would never see messages from that key again. New keys
> are constantly coming in.
>
> Is it that a watermark does not progress?
> Watermark on the subtask level is constantly updating and is in sync
> with other subtasks. We have not seen any issues with watermark
> updating as such.
>
> Looking through mailing list archive, our problem seems similar to
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Memory-in-CEP-queries-keep-increasing-td31045.html
> https://issues.apache.org/jira/browse/FLINK-15160 : Clean up is not
> applied if there are no incoming events for a key.
>
> By design we can have partial matched states/matches in pattern match
> queries. And key space is such that no new event comes in for those
> partial matches.
>
> thanks.
>
>  
>
> *From: *Dawid Wysakowicz <dwysakow...@apache.org>
> *Date: *Tuesday, January 26, 2021 at 3:14 AM
> *To: *Dcosta, Agnelo (HBO) <agnelo.dco...@hbo.com>,
> user@flink.apache.org <user@flink.apache.org>
> *Subject: *Re: Difference between table.exec.source.idle-timeout and
> setIdleStateRetentionTime ?
>
> **External Email received from: dwysakow...@apache.org **
>
>  
>
> Hi,
>
> The difference is that the *table.exec.source.idle-timeout *is used
> for dealing with source idleness[1]. It is a problem that a watermark
> cannot advance if some of the partition become idle (do not produce
> any records). Watermark is always the minimum of watermarks of all
> input partitions. The setting makes flink ignore certain partitions in
> the calculation after the time threshold is reached.
>
> The IdleStateRetention is Table API specific. As described in the link
> you provided it removes entries from a state for keys that were not
> seen for a given time threshold.
>
> As for your issue, I'd recommend first investigating what is causing
> the state to grow. Is it ever growing keyspace? Is it that a watermark
> does not progress (this should manifest in results as well). Or is it
> something else.
>
> Best,
>
> Dawid
>
>  
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/event_timestamps_watermarks.html#dealing-with-idle-sources
>
> On 25/01/2021 20:12, Dcosta, Agnelo (HBO) wrote:
>
> Hi,
>
> What is the difference between *table.exec.source.idle-timeout* and
> *setIdleStateRetentionTime* ?
>
> table.exec.source.idle-timeout:
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/config.html#table-exec-source-idle-timeout
>
>  
>
> setIdleStateRetentionTime:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/query_configuration.html#idle-state-retention-time
>
>  
>
> Some context:
> Hi we are using flink 1.12.
> Our checkpoint size is constantly increasing once app is deployed.
> After performing a restart, checkpoint size goes back to expected size.
> Looking at actual checkpoint files generated, it seems our app is
> holding on to state/events since the time app started up.
> Based on our sql, the maximum time we would need to hold state is 10
> minutes.
>
>  
>
> This e-mail is intended only for the use of the addressees. Any
> copying, forwarding, printing or other use of this e-mail by persons
> other than the addressees is not authorized. This e-mail may contain
> information that is privileged, confidential and exempt from
> disclosure. If you are not the intended recipient, please notify us
> immediately by return e-mail (including the original message in your
> reply) and then delete and discard all copies of the e-mail. Thank you.
> HB75
>

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to