[
https://issues.apache.org/jira/browse/FLINK-33819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809419#comment-17809419
]
Yue Ma commented on FLINK-33819:
--------------------------------
[~pnowojski] [~masteryhx]
I conducted benchmark tests on Snappy and No_Compression. In the Value State
test, I increased the number of setupKeyCounts to 1000 because the default
value state test is too small, most of which may be in the memtable. The
conclusion is as follows:
Point LookUp operations can improve performance by over *130%+*
Scan type operations can improve performance by *60%*
The performance of write operations has no impact with before
The size of valueState has increased from *78M to 96M* after disabling
compression
The size of mapState has increased from *58m to 86m* (Perhaps this data is for
reference only, as the actual compression ratio depends on the characteristics
of the business data)
|Benchmark|(backendType)|Mode|Score
(Snappy)|Score（NoCompression）|Units|performance benfit|
|MapStateBenchmark.mapAdd|ROCKSDB|thrpt|654.362|679.276|ops/ms|{color:#FF0000}3.80737267750878%{color}|
|MapStateBenchmark.mapContains|ROCKSDB|thrpt|104.57|297.213|ops/ms|{color:#FF0000}184.223964808262%{color}|
|MapStateBenchmark.mapEntries|ROCKSDB|thrpt|573.153|933.967|ops/ms|{color:#FF0000}62.9524751680616%{color}|
|MapStateBenchmark.mapGet|ROCKSDB|thrpt|106.288|330.821|ops/ms|{color:#FF0000}211.249623664007%{color}|
|MapStateBenchmark.mapIsEmpty|ROCKSDB|thrpt|88.642|207.76|ops/ms|{color:#FF0000}134.380993208637%{color}|
|MapStateBenchmark.mapIterator|ROCKSDB|thrpt|572.848|912.097|ops/ms|{color:#FF0000}59.2214688713236%{color}|
|MapStateBenchmark.mapKeys|ROCKSDB|thrpt|580.244|949.094|ops/ms|{color:#FF0000}63.568085150385%{color}|
|MapStateBenchmark.mapPutAll|ROCKSDB|thrpt|129.965|130.054|ops/ms|{color:#FF0000}0.0684799753779853%{color}|
|MapStateBenchmark.mapRemove|ROCKSDB|thrpt|723.835|785.637|ops/ms|{color:#FF0000}8.53813369068916%{color}|
|MapStateBenchmark.mapUpdate|ROCKSDB|thrpt|697.409|652.893|ops/ms|{color:#FF0000}-6.38305499355471%{color}|
|MapStateBenchmark.mapValues|ROCKSDB|thrpt|579.399|935.651|ops/ms|{color:#FF0000}61.4864713263226%{color}|
|ValueStateBenchmark.valueAdd|ROCKSDB|thrpt|645.081|636.098|ops/ms|{color:#FF0000}-1.39253830139162%{color}|
|ValueStateBenchmark.valueGet|ROCKSDB|thrpt|103.393|297.646|ops/ms|{color:#FF0000}187.87828963276%{color}|
|ValueStateBenchmark.valueUpdate|ROCKSDB|thrpt|560.153|621.502|ops/ms|{color:#FF0000}10.9521862776777%{color}|
| | | | | | | |
| | | | | | | |
| |DBSize （SnappyCompression）|DBSize （NoCompression）| | | | |
|ValueStateBenchMark|78M|96M| | | | |
|MapStateBenchMark|58M|86M| | | | |
> Support setting CompressType in RocksDBStateBackend
> ---------------------------------------------------
>
> Key: FLINK-33819
> URL: https://issues.apache.org/jira/browse/FLINK-33819
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.18.0
> Reporter: Yue Ma
> Assignee: Yue Ma
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-12-14-11-32-32-968.png,
> image-2023-12-14-11-35-22-306.png
>
>
> Currently, RocksDBStateBackend does not support setting the compression
> level, and Snappy is used for compression by default. But we have some
> scenarios where compression will use a lot of CPU resources. Turning off
> compression can significantly reduce CPU overhead. So we may need to support
> a parameter for users to set the CompressType of Rocksdb.
> !image-2023-12-14-11-35-22-306.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
