Re: Flink long state TTL Concerns

2020-03-19 Thread Matthew Rafael Magsombol
he.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html ? ( this is a bit off topic by this point now! ) On Thu, Mar 19, 2020 at 11:25 PM Matthew Rafael Magsombol < raffy4...@gmail.com> wrote: > Also as a follow up question with respect to state cleanup, > I see

Re: Flink long state TTL Concerns

2020-03-19 Thread Matthew Rafael Magsombol
m them. It sounds like rocksdb cleans these up via compaction but what about for states where we use FSBackendState where we use the heap for in-flight data? On Thu, Mar 19, 2020 at 7:07 PM Matthew Rafael Magsombol < raffy4...@gmail.com> wrote: > I see... > The way we run our setup is t

Re: Flink long state TTL Concerns

2020-03-19 Thread Matthew Rafael Magsombol
I see... The way we run our setup is that we run these in a kubernetes cluster where we have one cluster running one job. The total parallelism of the whole cluster is equal to the number of taskmanagers where each task manager has 1 core cpu accounting for 1 slot. If we add a state ttl, do you hav