Re: Non incremental window function accumulates unbounded state with RocksDb

2019-09-01 Thread Yun Tang
11:42 To: William Jonsson , "user@flink.apache.org" Cc: Fleet Perception for Maintenance Subject: Re: Non incremental window function accumulates unbounded state with RocksDb Hi William I don't believe the same job would have 70~80GB state for RocksDB while it&#x

Re: Non incremental window function accumulates unbounded state with RocksDb

2019-08-30 Thread William Jonsson
eption for Maintenance Subject: Re: Non incremental window function accumulates unbounded state with RocksDb Hi William I don't believe the same job would have 70~80GB state for RocksDB while it's only 200MB for HeapStateBackend even though RocksDB has some space amplification.

Re: Non incremental window function accumulates unbounded state with RocksDb

2019-08-30 Thread Yun Tang
link.apache.org Cc: Fleet Perception for Maintenance Subject: Non incremental window function accumulates unbounded state with RocksDb Hello, I have a Flink pipeline reading data from Kafka which is keyed (in the pseudocode example below it is keyed on the first letter in the string, in our

Non incremental window function accumulates unbounded state with RocksDb

2019-08-30 Thread William Jonsson
Hello, I have a Flink pipeline reading data from Kafka which is keyed (in the pseudocode example below it is keyed on the first letter in the string, in our pipeline it is keyed on a predefined key) and processed in sliding windows with a duration of 60m every 10:th minute. The time setting is e