Jobmanager & taskmanager restarts

Hi All, I have a jobmanager and 3 taskmanager pods running in Kubernetes. Suddenly my taskmanager has following logs and it got restarted. ``` 2024-12-25 23:11:27.836 WARN org.apache.pekko.remote.transport.netty.NettyTransport [] - Remote connection to [/10.68.16.22:55200] failed with org

Understanding flink rocks db state and compaction

Hi All, I have a flink job with Rabbitmq source,keyby, tumbling window and aggregator function, sink. I’m using rockDB state backend along with incremental checkpoints. Every 10s checkpointing is happening with 5s pause between checkpoints. Constant load with same keys I used for my test. I

State size

Hi All, I'm having flink job with incremental checkpointing and rocksdb state backend enabled. I see that my state grows and check point size as well. To limit the state size, I am trying to limit the memory Available in Taskmanager pod. What is physical memory in task manager from flink web UI

Managed memory and state size

Hi All, As my incremental rocksdb check point size is increasing continuously, I am trying to reduce managed memory. I am expecting my states size wont grow more than managed memory (because of rocksdb compaction) so that my check points size also will be in control. I allocated very minimal m

stabilized check point size

Hi All, I have a flink job with Rabbitmq source, sink,keyby, tumbling window and aggregator function. Incremental rocksdb checkpointing is enabled. I understand that checkpoints size will become stabilized at some point and it won't increase after that. How can I prove that to my customer?. I

Tuning rocksdb configuration

Hi All, I have a flink job with RMQ Source, filters, tumbling window(uses processing time fires every 2s), aggregator, RMQ Sink. Enabled incremental rocksdb checkpoints for every 10s with minimum pause between checkpoints as 5s. My checkpoints size is keep on increasing , so I am planning to tune

Re: Flink state

Dear Community, Gentle reminder about my below email. Thanks Banu On Sat, 20 Jul, 2024, 4:37 pm banu priya, wrote: > Hi All, > > I have a flink job with RMQ Source, filters, tumbling window(uses > processing time fires every 2s), aggregator, RMQ Sink. > > I am trying t

Flink state

Hi All, I have a flink job with RMQ Source, filters, tumbling window(uses processing time fires every 2s), aggregator, RMQ Sink. I am trying to understand about states and checkpoints(enabled incremental rocksdb checkpoints). In local rocks db directory, I have .sst files, log, lock, options fil

Re: Event de duplication in flink with rabbitmq connector

Hi All, Gentle reminder about bow query. Thanks Banu On Tue, 9 Jul, 2024, 1:42 pm banu priya, wrote: > Hi All, > > I have a Flink job with a RMQ source, tumbling windows (fires for each > 2s), an aggregator, then a RMQ sink. Incremental RocksDB checkpointing is > enabled with a

Event de duplication in flink with rabbitmq connector

Hi All, I have a Flink job with a RMQ source, tumbling windows (fires for each 2s), an aggregator, then a RMQ sink. Incremental RocksDB checkpointing is enabled with an interval of 5 minutes. I was trying to understand Flink failure recovery. My checkpoint X is started, I have sent one event to m

Re: Checkpoints and windows size

On Wed, 19 Jun, 2024, 9:38 am banu priya, wrote: > Hi All, > > I have a flink job with key by, tumbling window(2sec window time &uses > processing time)and aggregator. > > How often should I run the check point??I don't need the data to be > retained after 2s. >

Checkpoints and windows size

Hi All, I have a flink job with key by, tumbling window(2sec window time &uses processing time)and aggregator. How often should I run the check point??I don't need the data to be retained after 2s. I want to use incremental check point with rocksdb. Thanks Banupriya