Ryan19929 opened a new pull request, #64492:
URL: https://github.com/apache/doris/pull/64492
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
When a backup/restore job involves a large number of tablets (e.g. 50k+
tablets), persisting the job to the edit log causes severe FE heap pressure (an
81MB `OP_RESTORE_JOB` journal entry transiently allocates dozens of times its
size), which can lead to FE Full GC / OOM. There are two layers of tree-mode
(DOM) materialization in the Gson path:
1. **Field level**: the Gson adapters for Guava `Table` work in tree mode
(`JsonSerializer`/`JsonDeserializer`): they materialize the whole `JsonElement`
DOM of the huge `snapshotInfos` / `restoredVersionInfo` fields.
2. **Job level**: `RuntimeTypeAdapterFactory` (polymorphic dispatch on
`clazz`) converts the **entire job** to a `JsonElement` DOM on write
(`toJsonTree` + clone to put `clazz` first) and parses the full stream into a
DOM on read (`Streams.parse`), regardless of any field-level streaming.
This PR fixes both layers:
**Commit 1-3 (field level)**: a streaming `TypeAdapterFactory`
(`GuavaTableTypeAdapterFactory`) writes directly to `JsonWriter` / reads
directly from `JsonReader` without building the DOM, applied **per-field via
`@JsonAdapter`, only on `RestoreJob.snapshotInfos` and
`RestoreJob.restoredVersionInfo`**. The global tree-mode
`GuavaTableAdapter`/`GuavaMultimapAdapter` registrations are untouched, so all
other persisted metadata keeps the battle-tested legacy path.
**Commit 4 (job level)**: an opt-in `withStreamingDispatch()` mode for
`RuntimeTypeAdapterFactory`, enabled **only for `jobBackupTypeAdapterFactory`
(BackupJob / RestoreJob / CloudRestoreJob)**:
- Write: a pass-through `TypeFieldInjectingJsonWriter` injects `"clazz":
label` right after the first `beginObject()`, producing byte-identical output
to tree mode, and fails fast on a conflicting top-level field name.
- Read: requires `clazz` to be the first field (always true for
Doris-generated journals/images since tree mode always emits it first); after
consuming it, the remaining stream is handed to the subtype delegate via a
pass-through `EnteredObjectJsonReader`. Gson's `MapTypeAdapterFactory` promotes
map keys through the global `JsonReaderInternalAccess` hook which manipulates
`JsonReader` internals directly, so the hook is replaced once with an
unwrapping version (behavior for all other readers unchanged).
Allocation probe results (`ThreadMXBean.getThreadAllocatedBytes`,
100k-tablet RestoreJob, ratio = transient allocation / serialized size):
| path | before | field-level only | field-level + streaming dispatch |
|---|---|---|---|
| write (`toJson`) | 55.9x | 41.6x | **15.9x** (steady state) |
| read (replay via `AbstractJob`) | 112.5x | 30.2x | **10.3x** |
Real-cluster verification (51200 tablets, 8GB heap FE, this exact branch
build):
- backup + restore full pass, restore FINISHED in 403s, 0
`JsonParseException`
- `editlog_max_interval_jump` = 81.20MB, byte-identical to the previous
build (format unchanged)
- old-write-new-read: FE restarted on pre-existing metadata written by the
previous build, replay clean
- new-write-new-read: FE restarted after the run, replayed the 81MB
streaming-written `OP_RESTORE_JOB` entry, replay clean, data queryable
Compatibility:
- The json format is byte-identical between streaming and tree mode at both
layers, guarded by exact-format unit tests and cross-mode round-trip tests
(new-write-old-read / old-write-new-read). Old journals can be replayed by new
code and vice versa; rolling upgrade and rollback are safe.
- Streaming readers fail fast on field-order mismatch instead of silently
misreading.
### Release note
None
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Unit Test: GsonSerializationTest (exact json format pinning,
streaming-vs-legacy byte comparison), RuntimeTypeAdapterFactoryStreamingTest
(byte-identical output, cross-mode round-trip both directions, clazz-first
enforcement, conflicting field fail-fast, unknown subtype error),
RestoreJobTest, BackupJobTest, BackupHandlerTest, SchemaChangeJobV2Test,
GsonDerivedClassSerializationTest — full `org.apache.doris.persist.gson.*` +
`org.apache.doris.backup.*` regression: 60 tests, 0 failures
- Manual test: real-cluster backup+restore of 51200 tablets with double FE
restart/replay verification (old-write-new-read and new-write-new-read), built
from this branch
- Behavior changed: No (json format unchanged, byte-identical)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
