qzsee opened a new pull request, #70337:
URL: https://github.com/apache/doris/pull/70337
### What problem does this PR solve?
Related PR: #66378, #66389, #66447, #66451
Problem Summary:
Every round, `CloudTabletRebalancer` rebuilds a current and a future
generation of three routing indexes:
- global: `be -> tabletIds`
- table: `tableId -> be -> tabletIds`
- partition: `partitionId -> indexId -> be -> tabletIds`
All inner containers were `ConcurrentHashMap` /
`ConcurrentHashMap.newKeySet()` holding boxed `Long`s. The partition and table
levels create one concurrent set per (partition index, backend) and per (table,
backend), so the per-container and per-element overhead dominates. With
millions of routes, these indexes become one of the largest objects on the FE
heap.
#### Test cluster and metadata
| Item | Value |
|---|---|
| Compute groups / backends | 3 compute groups, 5 backends (1 + 2 + 2) |
| Tables | 192 (77 partitioned, 2 of them with 1 rollup; 115
non-partitioned) |
| Partitions | ~36,800 |
| Tablets per compute group | 947,075 |
| Routes per generation (all compute groups) | 2,841,225 |
| Tablets per backend | 947,075 (single-backend group) / ~473,500
(two-backend groups) |
| Data | none, the rebalancer only depends on metadata |
Table specs (empty tables created by a script):
| Buckets | Partitioned tables (count × partitions) | Non-partitioned tables
|
|---|---|---|
| 16 | 30 × 200, 10 × 500, 6 × 1000, 4 × 2000 | 50 |
| 32 | 8 × 200, 2 × 200 (with 1 rollup), 4 × 500, 2 × 1000, 1 × 2000 | 30 |
| 64 | 3 × 200, 2 × 500, 1 × 1000 | 20 |
| 128 | 2 × 200, 1 × 500 | 10 |
| 256 | 1 × 200 | 5 |
With this metadata, a heap dump taken while the indexes were alive showed
`CloudTabletRebalancer` retaining 839.82 MB, which is 4.6 times
`CloudInternalCatalog` (182.16 MB) and 16 times `CloudTabletInvertedIndex`
(52.56 MB).
#### Changes
This PR replaces the inner containers with primitive fastutil collections.
Scheduling logic and decisions are unchanged.
1. Partition level: `ConcurrentHashMap<partitionId,
Long2ObjectOpenHashMap<indexId, Long2ObjectOpenHashMap<be, LongArrayList>>>`. A
partition index usually has only a few tablets per backend, so a list is
smaller than a set and its linear removal is cheap. Containers are created with
small initial capacities.
2. Table level: `ConcurrentHashMap<tableId, Long2ObjectOpenHashMap<be,
LongOpenHashSet>>`. A large table can have many tablets on one backend, so a
hash set keeps removal O(1).
3. The partition and table indexes are only read and written by the
rebalancer daemon thread (`statRouteInfo`, `balanceImpl`, warmup rollback in
`processFailedWarmupTasks`), so their inner containers are intentionally not
thread-safe and are accessed without locking. The outer maps are still
`ConcurrentHashMap`.
4. A tablet is added to the partition / table containers only when it is
newly added to the global set of the same generation. Lists are not idempotent
like the sets they replace, so this keeps them free of duplicates (e.g. two
replicas of one tablet routed to the same backend).
5. Global level: the inner sets become `LongOpenHashSet`. Unlike the other
two levels, these sets are copied by other threads through
`getSnapshotTabletsIn*ByBeId` (tablet report in `CloudReportHandler`, warm up
in `CacheHotspotManager`, multi-cluster sync load in
`StmtExecutor.syncLoadForTablets`). An open-addressing removal shifts unrelated
elements, so an unguarded copy could miss a tablet that is not moving, and the
tablet report would then drop a tablet that is still in use. Each set is
therefore guarded by its own monitor:
- the rebalancer thread modifies a set only while holding its monitor
(`fillBeToTablets`, `updateBeToTablets`); the lock is uncontended while
`statRouteInfo` fills the sets, because they are published afterwards by the
volatile field write;
- snapshot methods copy a set only while holding its monitor, and
`addAll` takes fastutil's primitive copy path, so the lock is held for a few
milliseconds even for a backend with ~950k tablets;
- the rebalancer thread's own reads and `getTabletNumByBackendId`
(`size()` only) do not lock.
The lock is never nested and never held together with a table or catalog
lock. A snapshot is now an exact point-in-time copy; previously the weakly
consistent iteration could include or miss the tablet being moved.
6. Snapshot results change from `HashSet<Long>` to `LongOpenHashSet` and are
still returned as `Set<Long>`. Callers only use `contains` / `removeAll`.
7. The global set presize hint is no longer capped at `1 << 16` and uses the
previous round's size directly. With `LongOpenHashSet` the table is the whole
storage: under the cap, a backend with ~950k tablets regrew 4 times every
round, discarding about as much memory as the final set. An over-sized hint
after a catalog shrink lasts one round and is never larger than the previous
round's set, which is still alive while the new one is built.
8. Balancing helpers take `Map<Long, ? extends Collection<Long>>`.
`reservoirPick` / `containsTablet` iterate `LongCollection`s without boxing,
and `pickedTabletIds` is a `LongOpenHashSet`.
#### Result
Measured on the same metadata (MAT retained heap, both generations, taken
while the indexes were alive; `currentRoundTabletScanCount` 5,682,450 vs
5,682,462):
| Field | Before | After | Change |
|---|---|---|---|
| `partitionToTablets` + `futurePartitionToTablets` | 299.58 MB | 96.36 MB |
-67.8% |
| `beToTabletsInTable` + `futureBeToTabletsInTable` | 235.74 MB | 107.36 MB
| -54.5% |
| `beToTabletsGlobal` + `futureBeToTabletsGlobal` | 232.16 MB | 100.66 MB |
-56.6% |
| Others (mostly boxed tablet ids) | 72.34 MB | 3.31 MB | -95.4% |
| **`CloudTabletRebalancer` retained** | **839.82 MB** | **307.69 MB** |
**-63.4%** |
The global sets were the last holders of boxed tablet ids, so the boxed
`Long`s retained by the rebalancer (about 68 MB, 24 bytes each) are gone as
well. The remaining 300 MB is almost entirely the `long[]` tables of the three
levels.
### Release note
Reduce FE memory used by the cloud tablet rebalancer by storing its routing
indexes in primitive collections.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test <!-- TODO: cloud_p0/balance -->
- [x] Unit Test
- `CloudTabletRebalancerTest` (37 tests), including:
- `testSnapshotDoesNotMissUnmovedTabletsWhileTabletsAreMoving`:
one thread keeps moving tablets between two backends while another keeps taking
snapshots; every unmoved tablet must be present. It fails with the locks
removed.
- `testStatRouteInfoPresizesLargeGlobalTabletSetsWithoutBound`
- `testFillBeToTabletsAddsDuplicateRouteOnce`
- `testPickTabletPreferCold_primitiveCandidates`
- `CacheHotspotManagerTest` (15 tests)
- [x] Manual test (add detailed scripts or steps below)
- Metadata: see "Test cluster and metadata" above.
- Held the rebalancer thread after `statRouteInfo` and took `jmap
-dump:live` with the old and new builds on the same metadata; the retained
sizes are in "Result" above.
- TODO: compare `cloud_tablet_rebalancer_last_round_duration_ms`
before and after.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
