Doris-Breakwater commented on issue #70655: URL: https://github.com/apache/doris/issues/70655#issuecomment-6937166405
Breakwater-GitHub-Analysis-Slot: slot_586cbd41118b Initial triage: this is an FE metadata decoding failure with a confirmed process-exit path. The evidence does **not yet establish why the JSON presented to Gson was malformed**, or identify a matching LoadJob serialization fix. I checked the exact reported commit, `59de8c4c524008e8ab2e43b79312f716a3a423a8`, rather than assuming a release/tag mapping. The issue currently has no labels; FE persistence/journal replay is the primary area for maintainer routing, with the statistics exception tracked separately. **What the matching code establishes** - `BDBJournalCursor.next()` retrieves a record with `READ_COMMITTED`, then calls `JournalEntity.readFields()`. Its deserialization exception handler logs this exact fatal message and calls `System.exit(-1)`. `DorisFE` registers the shutdown hook that prints “Received shutdown signal”; that message also follows an internal `System.exit`, so it does not establish an external kill. See [cursor lines 120–131](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBJournalCursor.java#L120-L131) and [shutdown hook](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/DorisFE.java#L160-L169). - At the reported `JournalEntity.java:559`, the dispatch case is `OP_CREATE_LOAD_JOB` (230). This is a code-based inference from the stack, pending confirmation from the actual opcode bytes. It does not identify the job ID, label, or load subtype. The record layout is a two-byte opcode followed, for this operation, by a four-byte UTF-8 byte length and the JSON bytes. `LoadJob.write()` uses `GsonUtils.GSON.toJson(this)` and `Text.writeString()`; `read()` reverses that path. This is internal metadata JSON, with no evidence that malformed user input caused it. See [dispatch](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java#L558-L561), [LoadJob read/write](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJob.java#L827-L829), and [Text framing](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b 79312f716a3a423a8/fe/fe-common/src/main/java/org/apache/doris/common/io/Text.java#L404-L427). - The exception occurs at `Streams.parse(in)`, before subtype selection and object reconstruction. A missing subtype registration is therefore not supported by this trace. Column 660 is a parser position in the decoded text, not necessarily byte offset 660 in the BDB value. See [adapter read path](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/persist/gson/RuntimeTypeAdapterFactory.java#L303-L330). - Successful per-entry replay is logged at DEBUG in this build. Absence of `19295330` from normal healthy-node logs is inconclusive; a later replay ID alone also does not distinguish replay from loading a newer image. **Exporting the failing record without successful deserialization** The matching build already contains a useful tool: `BDBTool.getValueByKey()` catches the decode exception and subsequently prints `key`, `op code`, `num bytes`, and escaped `bytes`, even when `value` is null. It opens the JE environment/database read-only. See [tool implementation](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBTool.java#L144-L184). 1. Preserve FE-A's full metadata directory and incident logs before restarting or rebuilding it. Run diagnostics against a disposable, consistent offline copy, using the exact FE package and JE/Gson libraries. For a healthy FE, obtain a consistent backup/snapshot through the deployment's supported procedure; do not stop either remaining voting FE merely to get a comparison copy. 2. Use an isolated copied FE installation. Set its `DORIS_HOME`, `PID_DIR`, logging paths, and effective `meta_dir` to the diagnostic locations, including any `fe_custom.conf` overrides. With that package's Java 17 options and `lib/*` classpath, the Java main-class/argument forms are: ```text org.apache.doris.DorisFE --bdb --listdb org.apache.doris.DorisFE --bdb --db <segment-db-name> --from 19295330 --to 19295330 ``` These are JVM main-class arguments, not commands for `start_fe.sh`: that version's shell wrapper does not expose the `--bdb` options. List the numeric segment databases and choose the greatest segment start no greater than `19295330`; do not assume the segment name equals the journal ID. Preserve stdout, stderr, and diagnostic logs. The tool supports `--metaversion` if decoding requires the recorded metadata version; do not substitute the product version number. 3. Keep the original export private: LoadJob metadata can contain identifiers and source properties. For a lossless comparison, have a maintainer export the raw JE `DatabaseEntry` value slice (respecting offset/size) as hex/base64 and hash those original bytes before sanitization. The lookup key uses JE `TupleBinding<Long>`, not a decimal-text key. The tool's mixed printable/escaped `bytes:` output is useful for inspection but is not an unambiguous binary interchange format. One diagnostic limitation matters: this `BDBTool` explicitly disables JE log checksum reads. Successful extraction therefore does **not** establish storage integrity. If it cannot open the offline copy, retain that error for a version-matched JE diagnostic procedure rather than repairing the evidence copy. **How the comparison would narrow the cause** - Different values for the same journal ID in confirmed copies of the same cluster would direct investigation toward record divergence and the storage/replication path; it would not by itself prove a disk fault. - Identical values that fail the matching decoder on both copies would direct investigation toward the persisted representation, framing, and writer path. - Identical values that decode successfully offline while FE-A fails would require comparison of actual loaded libraries, JVM/configuration, and repeatability before attributing a cause. - `NOT FOUND` on a healthy copy can reflect retention/compaction and does not prove that node replayed or rejected the entry. Please provide: the opcode, value length and original-byte hashes from both copies if retained; a sanitized structural excerpt around the failing JSON position; SHA-256 checksums for `doris-fe.jar`, Gson and JE JARs across all FEs; actual JVM versions/options and effective metadata read settings, especially `metadata_text_read_max_batch_bytes`; FE-A's full exception/startup/shutdown window; healthy-node image-load and replay-range logs covering this entry; and available JE, filesystem, device, and service-manager diagnostics. Please align timestamps across sources: the incident time is `2026-11-13 17:36:35 UTC`. Once the record identifies the load subtype/job, its creation SQL and preceding DDL/load sequence would help construct a targeted reproduction. A query execution profile is not the first diagnostic needed for this metadata parser failure. **Recovery of this single non-Master Follower** Keep the two healthy voting FEs running; the OBSERVER does not supply the missing vote. After preserving evidence, the preferred recovery direction is a controlled rebuild/rejoin of FE-A from authoritative healthy metadata, with the exact membership procedure confirmed for this deployment. Do not treat force-skipping `19295330` as a routine fix: skipping a job-creation record can leave later metadata operations without their prerequisite state. Cluster-wide metadata recovery mode is also inappropriate as a default response to one failed follower; the [4.x metadata operations guide](https://doris.apache.org/docs/4.x/admin-manual/trouble-shooting/metadata-operation/) documents its risks. A valid same-cluster image with journal sequence **N >= 19295330** can avoid decoding this record: `Env.loadImage()` sets the replay position to N, and replay starts at N+1. This is conditional recovery, not proof that the underlying cause has been fixed. Verify the completed image's integrity and package compatibility, continuous availability of journals after N, correct FE-A identity/role and cluster membership, and absence of skip/recovery overrides. Preserve FE-A's identity files appropriately; do not copy another node's `ROLE`/node identity wholesale. After rejoin, confirm `Alive=true`, advancing/caught-up replay position, and successful metadata operations. See [image loading](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java#L2192-L2203) and [replay starting position](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/ca talog/Env.java#L3068-L3078). Simply adding `--helper` to an existing installation does not guarantee a newer image: when both local `ROLE` and `VERSION` exist, this build follows the local identity path and ignores helpers. Rebuilding/reseeding must therefore follow the deliberate node recovery procedure rather than assuming a restart downloads an image. **The statistics exception has a separate matching fix** The reported commit dereferences `table.getColumn(column.colName)` without checking for null. A queued column can become stale after a column is dropped while its table still exists. Public [PR #63882](https://github.com/apache/doris/pull/63882) adds the null check, and [PR #63947](https://github.com/apache/doris/pull/63947) is its merged 4.0 backport. The reported commit lacks that change; release inclusion should be checked against the intended upgrade package. `Daemon.run()` catches and logs this exception and continues subsequent cycles. The inspected statistics path processes queued columns and sends a statistics RPC; it does not serialize LoadJob journal records. Treat this as a separate known defect unless additional evidence links the two failures. Applying that fix is not an established remedy for the fatal JSON error. No source changes were made, and no production metadata was available for reproduction. The exit mechanism and diagnostic/recovery conditions above are verified from the matching source; the origin of the malformed JSON remains unresolved pending the record-level evidence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
