Re: Tuning rocksdb configuration

2024-07-25 Thread Zakelly Lan
Hi Banu, I'm trying to answer your question in brief: 1. Yes, when the memtable reaches the value you configured, a flush will be triggered. And no, sst files have different format with memtables, the size is smaller than 64mb IIUC. 2. Typically you don't need to change this value. If it is set

Access to S3 - checkpoints

2024-07-25 Thread Sigalit Eliazov
Hi, We are using Ceph buckets to store the checkpoints and savepoints, and the access is done via the S3 protocol. Since we don't have any integration with Hadoop, we added a dependency on flink-s3-fs-presto. Our Flink configuration looks like this: state.checkpoint-storage: filesystemstate.che

Tuning rocksdb configuration

2024-07-25 Thread banu priya
Hi All, I have a flink job with RMQ Source, filters, tumbling window(uses processing time fires every 2s), aggregator, RMQ Sink. Enabled incremental rocksdb checkpoints for every 10s with minimum pause between checkpoints as 5s. My checkpoints size is keep on increasing , so I am planning to tune