ming li created FLINK-28390:
-------------------------------
Summary: Allows RocksDB to configure FIFO Compaction to reduce CPU
overhead.
Key: FLINK-28390
URL: https://issues.apache.org/jira/browse/FLINK-28390
Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Reporter: ming li
We know that the fifo compaction strategy may silently delete data and may lose
data for the business. But in some scenarios, FIFO compaction can be a very
effective way to reduce CPU usage.
Flink's Taskmanager is usually some small-scale processes, such as allocating 4
CPUs and 16G memory. When the state size is small, the CPU overhead occupied by
RocksDB is not high, and as the state increases, RocksDB may frequently be in
the compaction operation, which will occupy a large amount of CPU and affect
the computing operation.
We usually configure a TTL for the state, so when using FIFO we can configure
it to be slightly longer than the TTL, so that the upper layer is the same as
before.
Although the FIFO Compaction strategy may bring space amplification, the disk
is cheaper than the CPU after all, so the overall cost is reduced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
