[ https://issues.apache.org/jira/browse/FLINK-18473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786887#comment-17786887 ]
Yi Zhang commented on FLINK-18473: ---------------------------------- Thanks [~masteryhx] for the pointers! yes, round-robin works for users who can manage to match #slots%#disks=0, may go worser for users who can't do that. the code seems need a bit cleaning since it is only doing random pick. Will checkout the SharedResources. > Optimize RocksDB disk load balancing strategy > --------------------------------------------- > > Key: FLINK-18473 > URL: https://issues.apache.org/jira/browse/FLINK-18473 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends > Affects Versions: 1.12.0 > Reporter: Rui Fan > Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > In general, bigdata servers have many disks. For large-state jobs, if > multiple slots are running on a TM, then each slot will create a RocksDB > instance. We hope that multiple RocksDB instances use different disks to > achieve load balancing. > h3. The problem of current load balancing strategy: > When the current RocksDB is initialized, a random value nextDirectory is > generated according to the number of RocksDB dir: [code > link|https://github.com/apache/flink/blob/2d371eb5ac9a3e485d3665cb9a740c65e2ba2ac6/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java#L441] > {code:java} > nextDirectory = new Random().nextInt(initializedDbBasePaths.length); > {code} > Different slots generate different RocksDBStateBackend objects, so each slot > will generate its own *nextDirectory*. The random algorithm used here, so the > random value generated by different slots may be the same. For example: the > current RocksDB dir is configured with 10 disks, the *nextDirectory* > generated by slot0 and slot1 are both 5, then slot0 and slot1 will use the > same disk. This disk will be under a lot of pressure, other disks will not be > under pressure. > h3. Optimization ideas: > *{{nextDirectory}}* should belong to slot sharing, the initial value of > *{{nextDirectory}}* cannot be 0, it is still generated by random. But define > *nextDirectory* as +_{{static AtomicInteger()}}_+ and execute > +_{{nextDirectory.incrementAndGet()}}_+ every time RocksDBKeyedStateBackend > is applied for. > {{nextDirectory}} takes the remainder of {{initializedDbBasePaths.length}} to > decide which disk to use. > Is there any problem with the above ideas? > -- This message was sent by Atlassian Jira (v8.20.10#820010)