[
https://issues.apache.org/jira/browse/HDDS-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDDS-8316.
-----------------------------------
Resolution: Won't Fix
Resolving as Won't Fix: this optimization of the legacy persistent collections
is superseded by the optimized SnapDiff implementation developed under
HDDS-9154.
The new SnapDiffJobStore already uses ManagedWriteBatch, with a default batch
size of 1,000, for intermediate state and report writes. This incorporates the
batching proposed here without adding a separate optimization to the legacy
collections.
Repository evidence at d0949f291791c399f3e54409fcb92656a80597a4:
* Default batch size and shared batched job store:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapDiffJobStore.java#L82
* Batch flush implementation:
https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapDiffJobStore.java#L550
The existing SnapshotDiffManager still uses the legacy persistent collections
and individual writes in this repository baseline. This resolution does not
claim that the legacy execution path has already been replaced. Integration,
rollout and removal remain tracked by HDDS-15397, HDDS-15400 and HDDS-15402
under HDDS-9154. We will not separately optimize the legacy collections slated
for replacement.
> SnapDiff: Persistent collection optimization
> --------------------------------------------
>
> Key: HDDS-8316
> URL: https://issues.apache.org/jira/browse/HDDS-8316
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Hemant Kumar
> Assignee: Hemant Kumar
> Priority: Minor
>
> To store the snapDiff and intermediate objects, we implemented persistent
> collections (List, Map, Set) backed by RocksDB. Currently we do individual
> RocksDb put for every add/put operation to the collection. Which results to
> lots of I/O. We can optimize it by using WriteBatch and do batch updates of
> like 1000 entries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
