gavinchou opened a new pull request, #67183:
URL: https://github.com/apache/doris/pull/67183
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
`meta_tool --operation=show_segment_data` previously read a fixed 10-row
sample, disabled page checksum verification, retained all returned values in
one batch, and did not reliably propagate read failures through its process
exit code. It could therefore reproduce a bad page, but could not prove that
every logical row in a repaired segment was readable.
Metadata-driven reads are also insufficient for forensic analysis when the
segment footer or ordinal indexes contain stale physical offsets. A valid page
may still exist at a corrected physical offset, but the normal reader will
continue to use the stored `PagePointer`.
This PR:
- adds bounded row controls to `show_segment_data`:
- `--rows` (`-1` reads all remaining rows)
- `--row_start`
- `--batch_rows`
- `--check_only`
- `--verify_checksum`
- `--scan_segment_pages`
- makes segment inspection return meaningful status and exit codes:
- `0`: success
- `1`: corruption, I/O, checksum, decoding, or boundary failure
- `2`: invalid command-line input
- adds `--operation=check_page_crc` for explicit physical `OFFSET:SIZE`
ranges
- adds `--operation=scan_page_crc` for deriving a consecutive page chain
between a known start and an exact expected end
- validates CRC32C, page footer size, and `PageFooterPB` with bounded
one-MiB reads
- documents operation selection, output modes, limitations, and a practical
inserted-extent investigation workflow
The raw CRC operations are read-only forensic checks. They do not modify a
segment, discover an arbitrary first-page offset, prove that a page is
referenced by metadata, or replace full logical decoding.
### Release note
Add bounded full-segment validation and raw physical page CRC operations to
the BE `meta_tool`.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Manual validation:
- `sh build.sh --be --meta-tool -j 32`
- `sh format_code.sh be/src/tools/meta_tool.cpp`
- `build-support/check-format.sh` with clang-format 16
- `git diff --check`
- decoded all 43 columns and 1,211,072 rows per column with `--rows=-1
--check_only=true --verify_checksum=true --scan_segment_pages=true`
- validated three representative explicit page ranges, including data and
index pages
- scanned and validated a continuous 815-page range, ending exactly at the
requested end offset
- confirmed that the known corrupt extent returns corruption and exit code 1
- checked invalid range syntax, conflicting range inputs, missing scan
bounds, invalid output mode, and out-of-file ranges
`run-clang-tidy.sh` was also attempted. The new output-parameter findings
were fixed, but the command remains blocked by existing/toolchain issues
including an unmatched `NOLINTEND` in `be/src/core/types.h` and the clang-20
toolchain failing to resolve `stddef.h`.
- Behavior changed:
- [ ] No.
- [x] Yes. `show_segment_data` now verifies checksums by default and
reports failures through non-zero exit codes; new row/page flags and raw CRC
operations are available.
- Does this need documentation?
- [ ] No.
- [x] Yes. `be/src/tools/README.md` is added and expanded in this PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
