924060929 commented on PR #66717:
URL: https://github.com/apache/doris/pull/66717#issuecomment-5289400702
Design suggestion after reviewing the current head `57a8d5d7`: I think the
production estimator can be simplified substantially around the two actual
goals of this PR:
1. Keep the long-lived External MetaCache footprint approximately bounded
during normal operation.
2. Let the JVM sacrifice unused metadata cache values when heap is under
pressure.
For these goals we do not need a precise retained-object-graph size. I
suggest using `maximumWeight + softValues`, with a cheap cache-specific linear
weight based only on cardinalities already available from the loaded value:
```java
long weight = BASE_WEIGHT
+ partitionCount * PARTITION_WEIGHT
+ fileCount * FILE_WEIGHT;
```
The unit can be approximate KiB. The constants can be calibrated offline
with the existing benchmarks/full estimator and rounded up to simple powers of
two. Production weighing should only read O(1) collection sizes or counters
already produced by the normal loader. It must not reflect over object fields,
build an identity set, materialize lazy state, read manifests remotely, or
serialize/clone the value.
Suggested formulas for the currently managed and adjacent unbounded entries:
```text
Hive partition_values
BASE
+ partitionCount * HIVE_PARTITION_WEIGHT
+ partitionColumnCount * partitionCount * HIVE_PARTITION_VALUE_WEIGHT
Hive file entry, when brought into the managed scope
BASE
+ fileCount * HIVE_FILE_WEIGHT
Iceberg table
BASE
+ snapshotCount * ICEBERG_SNAPSHOT_WEIGHT
+ schemaCount * ICEBERG_SCHEMA_WEIGHT
+ specCount * ICEBERG_SPEC_WEIGHT
+ sortOrderCount * ICEBERG_SORT_ORDER_WEIGHT
+ propertyCount * ICEBERG_PROPERTY_WEIGHT
Iceberg snapshot
BASE
+ partitionCount * ICEBERG_PARTITION_WEIGHT
+ nameMappingEntryCount * ICEBERG_NAME_MAPPING_WEIGHT
+ manifestCount * ICEBERG_MANIFEST_WEIGHT
// only when the manifest list is already retained/materialized by the
normal load path;
// never perform remote IO solely to obtain this count
Iceberg manifest
BASE
+ dataFileCount * ICEBERG_DATA_FILE_WEIGHT
+ deleteFileCount * ICEBERG_DELETE_FILE_WEIGHT
Paimon snapshot projection
BASE
+ partitionCount * PAIMON_PARTITION_WEIGHT
+ schemaFieldCount * PAIMON_SCHEMA_FIELD_WEIGHT
+ optionCount * PAIMON_OPTION_WEIGHT
```
For Paimon, `Partition.fileCount()` is only a scalar retained in each
partition record; it should not be charged as if the cache retained every file
object. If a future Paimon entry actually retains file objects, that entry can
add `retainedFileCount * PAIMON_FILE_WEIGHT`.
The formula should include only collections actually retained by that cache
value. Hudi, MaxCompute, Doris, Hive single-partition, and other small/bounded
entries can remain count-based until one of their values retains an unbounded
collection; then the same `BASE + cardinality * unit` rule can be added.
`softValues()` provides the second property: a value still used by a query
remains strongly reachable from the query, while a value retained only for
cache reuse can be collected under JVM memory pressure. `maximumWeight` remains
the predictable normal bound; soft values are the emergency pressure-release
path.
There is one important lifecycle requirement in the current implementation:
`MetaCacheEntry.ReservationRecord<V>` and `RefreshRecord<V>` strongly retain
`value`. Adding Caffeine `softValues()` without removing those strong
references would make the soft policy ineffective. Reservation
ownership/removal therefore needs to be generation/token based without strongly
retaining `V`. `COLLECTED`, delayed removal, replacement, invalidation, and
close must release only the matching generation. Accounting may be
conservatively late, but must never release a newer live generation because an
older soft value was collected.
I would keep the existing global/catalog/entry budget hierarchy,
admission-before-publication protocol, generation fencing, and rejection
behavior. I would replace the production `OwnedObjectSizeEstimator` and SDK
field-signature machinery with these coarse weighers, retaining the full
estimator only in tests/benchmarks to calibrate constants.
The key tests should be:
- weight grows approximately linearly from 1K to 10K to 100K
partitions/files;
- weighing performs no remote IO or lazy materialization;
- a value held by a query is not lost when soft references are collected;
- an otherwise unreferenced cache value can be collected and its matching
reservation is eventually released;
- a delayed `COLLECTED` callback cannot release a replacement generation;
- no long-lived catalog/table object outside MetaCache strongly retains the
cached value.
This gives a substantially cheaper and more maintainable implementation
while still improving by orders of magnitude over pure entry-count limits. As
with the current PR scope, it controls retained MetaCache memory after load; it
does not bound temporary memory used by the query that constructs the value
before admission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
