[
https://issues.apache.org/jira/browse/FLINK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17635599#comment-17635599
]
Xintong Song commented on FLINK-30073:
--------------------------------------
I'm no more familiar with the background of FLINK-14484 &FLINK-15532 than you
are. So I cannot comment whether returning {{false}} from
{{useManagedMemory()}} when {{fixed-per-slot > 0}} is a desirable design.
Assuming the proposed change is a good design, I see a significant behavior
change in this. Let's say fixed-per-slot is configured to X, and a Y sized
memory calculated from the weights is reserved for rocksdb due to not setting
{{useManagedMemory()}} to {{false}}. Assuming currently the job is running well
currently, then either Y >= X and some memory is wasted, or Y < X and other
not-fully-used memory (framework.off-heap, jvm-overhead, etc.) is making up for
the gap. With the proposed change, Y will be shared by other managed memory use
cases, while rocksdb still uses X. That means when upgrading to the new
version, to keep a job with fixed-per-slot running with the same resources,
user needs to increase the framework.off-heap memory by min(X, Y), and either
also increase the total memory or reduce the managed memory.
I'm not oppose to this proposal. I don't have enough expertise in the rocksdb
state backend to decide whether this is a good design change. Just trying to
say that, considering the significant behavior changes, we should be very
cautious about this change and apply it only if it's absolutely necessary.
cc [~yuanmei]
> Managed memory can be wasted if RocksDB memory is fixed-per-slot
> ----------------------------------------------------------------
>
> Key: FLINK-30073
> URL: https://issues.apache.org/jira/browse/FLINK-30073
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Affects Versions: 1.16.0, 1.17.0, 1.15.2
> Reporter: Roman Khachatryan
> Priority: Major
>
> When
> [state.backend.rocksdb.memory.fixed-per-slot|https://github.com/apache/flink/blob/ba4b182955867fedfa9891bf0bf430e92eeab41a/flink-runtime/src/main/java/org/apache/flink/runtime/util/config/memory/ManagedMemoryUtils.java#L75]
> is set, RocksDB does not use managed memory (this option overrides the
> 'state.backend.rocksdb.memory.managed').
> However, the runtime [doesn't take this into
> account|https://github.com/apache/flink/blob/ba4b182955867fedfa9891bf0bf430e92eeab41a/flink-runtime/src/main/java/org/apache/flink/runtime/util/config/memory/ManagedMemoryUtils.java#L75]
> and still reserves the managed memory according to the configured weigths.
> cc: [~yunta]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
