Chu Cheng Li created HDDS-17173:
-----------------------------------
Summary: OM table iterators should return applied entries that are
not flushed yet
Key: HDDS-17173
URL: https://issues.apache.org/jira/browse/HDDS-17173
Project: Apache Ozone
Issue Type: Improvement
Reporter: Chu Cheng Li
*Problem*
OM applies a write request in two steps. First, {{validateAndUpdateCache}}
updates the table cache. Later, the double buffer writes the change to RocksDB.
{{Table.get}} reads the table cache first. Thus, it returns the applied state.
{{Table.iterator}} reads RocksDB only. Thus, it does not return an entry that
is applied but not flushed. It also returns an entry that the cache marks as
deleted. {{keyIterator}}, {{valueIterator}} and
{{OmMetadataManagerImpl.countRowsInTable}} use {{Table.iterator}}. Thus, they
have the same problem.
*Effect*
* Each caller that iterates a table must merge the cache by hand. OM has about
20 of these merges, for example in {{ListIterator}},
{{KeyManagerImpl.listStatus}} and {{OMFileRequest.hasChildren}}. A merge that
is missing or wrong is a bug, for example HDDS-14600 and HDDS-9233.
* Some callers do not merge the cache. For example,
{{OmMetadataManagerImpl.listOpenFiles}} does not show a new open key until the
flush.
* A test that reads a table right after a request can see old data. Thus, tests
add {{awaitDoubleBufferFlush}} calls, for example in HDDS-16523, HDDS-13865 and
HDDS-13880.
*Fix*
* {{TypedTable.iterator}} also returns the cache entries that RocksDB does not
contain yet. A cache value replaces the DB value of the same key. A deleted
cache entry hides the DB entry. The merge uses the RocksDB key order (unsigned
bytes), not {{String.compareTo}}.
* The new class {{FlushedCacheEpoch}} records the highest cache epoch in each
committed batch. When a DB iterator opens, it gets the epoch that it contains.
The iterator merges only the cache entries with a higher epoch. If the open
overlaps a commit, the open runs again.
* The epoch check is necessary for two reasons. First, a cache entry stays in
the cache for a short time after the flush. Second, snapshot create deletes a
range of {{deletedTable}} at flush time, without a cache entry. Without the
check, an iterator can return keys that moved to the snapshot.
* Only a store built with {{DBStoreBuilder.setIteratorsReadTableCache(true)}}
merges the cache. The OM DB uses this setting. SCM, Recon, datanodes and OM
snapshot DBs do not change. Full-cache tables (volume and bucket) do not change.
* The double buffer commits each batch with {{commitBatchOperation(batch,
lastIndex)}}. It builds the batch inside {{FlushedCacheEpoch.readDbOnly}}.
Thus, the responses still read the DB as of the previous batch. They do not
read the cache entries of later transactions.
* The checkpoint servlet reads the snapshot directories from the DB only. A
snapshot directory exists only after the flush.
* {{KeyLifecycleService.getSubDirectory}} does not scan the cache separately
anymore. Otherwise, it counts each cached sub-directory two times.
* {{TestHSync}} does not call {{awaitDoubleBufferFlush}} before it reads
{{openFileTable}} anymore (HDDS-16523).
* Requests do not add cache entries for {{deletedTable}}, {{deletedDirTable}}
and {{snapshotRenamedTable}}. Thus, this change does not affect these tables.
Related: HDDS-17077.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
