This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 13735f9f797 [improvement](be) Add complete segment validation and raw
CRC checks to meta tool (#67183)
13735f9f797 is described below
commit 13735f9f7973b041dca2072c00337456202944f0
Author: Gavin Chou <[email protected]>
AuthorDate: Thu Aug 27 15:57:46 2026 +0800
[improvement](be) Add complete segment validation and raw CRC checks to
meta tool (#67183)
Problem Summary:
`meta_tool --operation=show_segment_data` previously read a fixed 10-row
sample, disabled page checksum verification, retained all returned
values in one batch, and did not reliably propagate read failures
through its process exit code. It could therefore reproduce a bad page,
but could not prove that every logical row in a repaired segment was
readable.
Metadata-driven reads are also insufficient for forensic analysis when
the segment footer or ordinal indexes contain stale physical offsets. A
valid page may still exist at a corrected physical offset, but the
normal reader will continue to use the stored `PagePointer`.
This PR:
- adds bounded row controls to `show_segment_data`:
- `--rows` (`-1` reads all remaining rows)
- `--row_start`
- `--batch_rows`
- `--check_only`
- `--verify_checksum`
- `--scan_segment_pages`
- makes segment inspection return meaningful status and exit codes:
- `0`: success
- `1`: corruption, I/O, checksum, decoding, or boundary failure
- `2`: invalid command-line input
- adds `--operation=check_page_crc` for explicit physical `OFFSET:SIZE`
ranges
- adds `--operation=scan_page_crc` for deriving a consecutive page chain
between a known start and an exact expected end
- validates CRC32C, page footer size, and `PageFooterPB` with bounded
one-MiB reads
- documents operation selection, output modes, limitations, and a
practical inserted-extent investigation workflow
The raw CRC operations are read-only forensic checks. They do not modify
a segment, discover an arbitrary first-page offset, prove that a page is
referenced by metadata, or replace full logical decoding.
### Release note
Add bounded full-segment validation and raw physical page CRC operations
to the BE `meta_tool`.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Manual validation:
- `sh build.sh --be --meta-tool -j 32`
- `sh format_code.sh be/src/tools/meta_tool.cpp`
- `build-support/check-format.sh` with clang-format 16
- `git diff --check`
- decoded all 43 columns and 1,211,072 rows per column with `--rows=-1
--check_only=true --verify_checksum=true --scan_segment_pages=true`
- validated three representative explicit page ranges, including data
and index pages
- scanned and validated a continuous 815-page range, ending exactly at
the requested end offset
- confirmed that the known corrupt extent returns corruption and exit
code 1
- checked invalid range syntax, conflicting range inputs, missing scan
bounds, invalid output mode, and out-of-file ranges
`run-clang-tidy.sh` was also attempted. The new output-parameter
findings were fixed, but the command remains blocked by
existing/toolchain issues including an unmatched `NOLINTEND` in
`be/src/core/types.h` and the clang-20 toolchain failing to resolve
`stddef.h`.
- Does this need documentation?
- [ ] No.
- [x] Yes. `be/src/tools/README.md` is added and expanded in this PR.
---
be/src/tools/README.md | 439 +++++++++++++++++++++++
be/src/tools/meta_tool.cpp | 860 ++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 1210 insertions(+), 89 deletions(-)
diff --git a/be/src/tools/README.md b/be/src/tools/README.md
new file mode 100644
index 00000000000..7e919b94e1e
--- /dev/null
+++ b/be/src/tools/README.md
@@ -0,0 +1,439 @@
+# Doris BE Meta Tool
+
+`meta_tool` inspects and maintains Doris BE metadata. This document focuses on
the read-only
+segment inspection operations used for diagnosing and validating segment files.
+
+## Build
+
+Build the BE meta tool from the Doris repository root:
+
+```bash
+sh build.sh --be --meta-tool -j 32
+```
+
+The installed binary is normally available at:
+
+```text
+be/output/lib/meta_tool
+```
+
+If the loader reports `libjvm.so: cannot open shared object file`, add the JDK
server library before
+running the examples:
+
+```bash
+export LD_LIBRARY_PATH="${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH:-}"
+```
+
+## Inspect a segment footer
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_footer \
+ --file=/path/to/segment.dat
+```
+
+This parses and prints `SegmentFooterPB`. It does not decode column values.
+
+## Check physical page CRCs
+
+Doris segment pages use this physical trailer layout:
+
+```text
+page body | PageFooterPB | footer_size (4-byte little endian) | crc32c (4-byte
little endian)
+```
+
+The stored CRC covers every byte except the final four-byte CRC itself. The
page CRC operations
+also validate that `footer_size` stays inside the supplied page range and that
the footer parses as
+a `PageFooterPB` with its required logical fields.
+
+There are three related but different page checks:
+
+| Check | Page locations come from | Intended use |
+|---|---|---|
+| `show_segment_data --scan_segment_pages=true` | Segment footer and ordinal
indexes | Validate metadata-referenced data and dictionary pages while reading
the segment normally. |
+| `--operation=check_page_crc` | Caller-supplied `offset:size` values | Test
known or corrected physical `PagePointer` candidates without trusting segment
metadata. |
+| `--operation=scan_page_crc` | A caller-supplied start and end offset |
Derive consecutive page sizes from CRC trailers when the first page boundary is
already known. |
+
+### When to use each operation
+
+Start with the normal metadata-driven path whenever the segment footer and
indexes are expected to
+be trustworthy. The raw CRC operations are forensic tools for the cases where
normal reading has
+already failed or where a candidate physical layout must be tested
independently of the offsets
+stored in the segment.
+
+| Situation | Recommended operation | Reason |
+|---|---|---|
+| Validate that a healthy or repaired segment can really serve all logical
rows | `show_segment_data --rows=-1 --check_only=true --verify_checksum=true` |
Exercises normal footer, index, iterator, decompression, decoding, and checksum
paths. |
+| Inventory metadata-referenced data and dictionary pages | Add
`--scan_segment_pages=true` to `show_segment_data` | Uses the same PagePointers
that Doris will use in production and reports page-level layout/CRC results. |
+| Check one or more page locations whose exact physical offsets and sizes are
already known | `check_page_crc` | Directly tests each supplied `offset:size`
range without following the possibly wrong PagePointer stored in metadata. |
+| Test whether adding a suspected constant delta to metadata offsets recovers
pages | `check_page_crc` | The corrected offset and the original metadata size
form an exact candidate range. Sampling pages near the start, middle, and end
tests whether one displacement explains the suffix. |
+| Validate a candidate first-page boundary when individual page sizes are
unknown | `scan_page_crc` | Derives each page end from CRC/footer evidence and
requires a gap-free chain to the supplied end. |
+| Search an entire file for an unknown first-page offset | Neither raw
operation directly supports this | `scan_page_crc` verifies a caller-provided
start; it does not enumerate possible starts. First derive candidates from
metadata, an S3 part boundary, byte comparison, logs, or another independent
source. |
+| Determine whether bytes are semantically correct merely because their CRC
matches | Neither raw operation is sufficient | CRC proves physical consistency
of a selected page range, not business-value correctness. Follow with normal
full decoding. |
+| Modify or repair the segment | Neither operation | Both operations are
read-only. Repair must be performed separately and then validated through the
normal read path. |
+
+The distinction matters because Doris page reads are address based, not
self-synchronizing. The
+reader obtains an absolute `PagePointer(offset, size)` from the footer or an
index and reads exactly
+that byte range. It does not search forward for another valid page after a
checksum mismatch. If an
+unaccounted extent is inserted in the middle of the file, the footer at the
physical EOF can still
+parse successfully while every PagePointer after the insertion addresses the
wrong bytes:
+
+```text
+metadata/logical layout
+... previous page | next page at logical offset L | following pages ...
+ ^ PagePointer offset L
+
+damaged physical layout
+... previous page | unaccounted bytes | next page at physical offset P |
following pages ...
+ ^ reader still uses L ^ candidate P = L + delta
+```
+
+This is the main use case for the two raw operations. `scan_page_crc` can test
whether `P` begins a
+continuous page chain, while `check_page_crc` can test representative metadata
ranges after adding
+the same `delta` to their offsets.
+
+### Practical investigation workflow: inserted extent at an S3 boundary
+
+The following workflow comes from a real `Bad page: checksum mismatch`
investigation. The segment
+footer was readable, pages before logical offset `5,203,052` were readable,
and pages after that
+offset failed. Metadata independently placed both the end of the
key/primary-index group and the
+first value-column page at logical offset `5,203,052`.
+
+The configured S3 write-buffer boundary was 5 MiB:
+
+```text
+5 MiB = 5,242,880
+candidate delta = 5,242,880 - 5,203,052 = 39,828
+```
+
+The 5 MiB value supplied a candidate physical start; the CRC scanner did not
discover that start.
+Use the following sequence when investigating a similar layout.
+
+1. Preserve the original file and record its length and checksum. Run
`show_segment_footer` and a
+ normal `show_segment_data` check first. Record the first failing
column/page and whether all
+ metadata-referenced pages after some logical boundary fail.
+
+2. Establish the logical boundary `L` from metadata. Useful independent
signals include the end of
+ the last readable PagePointer, the start of the first failing data page,
vertical-compaction
+ key/value group boundaries, and the expected logical EOF derived from the
footer.
+
+3. Propose one or more physical starts `P` from evidence outside the CRC
scanner. Examples include
+ S3 multipart boundaries, file-cache block boundaries, direct object-range
comparison, known
+ alignment, or a constant difference between the actual and expected file
sizes.
+
+4. Check whether `[L,P)` itself forms a page chain. For the example, scanning
the suspected extra
+ extent is expected to fail:
+
+ ```bash
+ be/output/lib/meta_tool \
+ --operation=scan_page_crc \
+ --file=/path/to/bad-segment.dat \
+ --page_scan_start=5203052 \
+ --page_scan_end=5242880 \
+ --page_output=errors
+ ```
+
+ The observed result was `no_page_boundary offset=5203052 remaining=39828`
and exit code 1. This
+ does not by itself prove that the bytes are corrupt; it says only that the
suspected extent is
+ not a consecutive Doris page chain beginning at `L`.
+
+5. Scan forward from candidate `P` to an independently expected end. In the
example, the following
+ range was expected to contain all 91 pages of column 1:
+
+ ```bash
+ be/output/lib/meta_tool \
+ --operation=scan_page_crc \
+ --file=/path/to/bad-segment.dat \
+ --page_scan_start=5242880 \
+ --page_scan_end=5919643 \
+ --page_output=all
+ ```
+
+ It produced 91 consecutive CRC- and footer-valid pages, with no gaps or
overlaps, and ended
+ exactly at `5,919,643`. The page count and end position also matched the
metadata after adding
+ `39,828`. A long exact chain is much stronger boundary evidence than one
isolated CRC match.
+
+6. Use `check_page_crc` to test the same delta at widely separated locations.
Keep the sizes from
+ metadata and add `39,828` only to offsets:
+
+ ```bash
+ be/output/lib/meta_tool \
+ --operation=check_page_crc \
+ --file=/path/to/bad-segment.dat \
+ --page_ranges=5242880:6485,5919754:47,264635689:1384,274697492:2932 \
+ --page_output=all
+ ```
+
+ These candidates cover the first displaced data page, another small data
page, a middle/late
+ ordinal root, and a page near the file tail. If all pass while their
uncorrected metadata offsets
+ fail, one inserted extent and a constant suffix displacement is a better
explanation than many
+ unrelated page corruptions.
+
+7. If creating a non-destructive forensic copy by removing the suspected
extent, preserve the
+ original and verify byte identity on both sides of the edit. Then validate
the copy through the
+ normal path, not only with raw CRCs:
+
+ ```bash
+ be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/repaired-copy.dat \
+ --rows=-1 \
+ --batch_rows=4096 \
+ --check_only=true \
+ --verify_checksum=true \
+ --scan_segment_pages=true
+ ```
+
+ In the incident, deleting only `[5,203,052,5,242,880)` made the suffix
byte-for-byte identical
+ to the expected logical layout; 9,060 data pages and six dictionary pages
passed, and all 11
+ columns were readable. That establishes the repaired copy as forensic
evidence, but does not by
+ itself authorize replacing a production object or its rowset metadata.
+
+### Interpreting raw CRC results
+
+| Observation | What it supports | What it does not prove |
+|---|---|---|
+| One explicit range has matching CRC and a valid footer | The supplied offset
and size delimit an internally consistent Doris page. | The page is referenced
by this segment, contains the intended values, or is at the correct logical
position. |
+| Corrected range passes while the metadata range of the same size fails | The
candidate physical displacement is plausible for that page. | A single page is
not enough to prove one constant displacement for the whole suffix. |
+| Many corrected ranges near the start, middle, and tail pass with the same
delta | A single inserted/omitted physical extent is a strong explanation. |
Whether the discrepancy originated in object upload, object storage, cache,
warm-up, range download, or local copying. |
+| `scan_page_crc` finds a long continuous chain and ends exactly at an
independently expected end | The proposed start is strongly supported as a real
page boundary. | How the start was created or that it would have been found
without the supplied candidate. |
+| `no_page_boundary` occurs at the first offset | No valid page chain was
found beginning exactly there within the supplied range. | The range contains
no valid page at some later, unknown start. |
+| A chain stops after several valid pages | The valid prefix is physically
consistent; the next page is corrupt, the requested end is wrong, or non-page
bytes follow. | Which of those causes applies without more metadata or
byte-layout evidence. |
+| Raw page checks pass but normal full decoding fails | Physical page framing
is intact for the checked ranges. | Compression, encoding, index linkage,
schema, logical ordering, or value semantics may still be wrong. |
+
+For an unknown start, avoid running `scan_page_crc` across a multi-gigabyte
range and treating its
+failure as discovery. The implementation performs a byte-by-byte running CRC
search from exactly
+the supplied start, so the work is proportional to the distance to the next
accepted boundary.
+Generate a small set of defensible candidate starts first, then test each one.
+
+Raw CRC operation flags:
+
+| Flag | Default | Operation | Description |
+|---|---:|---|---|
+| `--file` | empty | both | Local segment path; required. |
+| `--page_ranges` | empty | `check_page_crc` | Comma-separated decimal
`OFFSET:SIZE` values. |
+| `--page_ranges_file` | empty | `check_page_crc` | File containing decimal
`OFFSET SIZE` pairs. |
+| `--page_scan_start` | `0` | `scan_page_crc` | Inclusive known first-page
offset; the flag must be explicitly supplied even when the value is zero. |
+| `--page_scan_end` | `0` | `scan_page_crc` | Exclusive expected end offset;
the flag must be explicitly supplied. |
+| `--page_output` | `all` | both | Per-page output level: `all`, `errors`, or
`summary`. |
+
+### Check known page ranges
+
+Use `--page_ranges` for a small number of decimal `OFFSET:SIZE` pairs:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=check_page_crc \
+ --file=/path/to/segment.dat \
+ --page_ranges=5242880:18349,32162899:2387
+```
+
+Whitespace around comma-separated ranges is accepted, but hexadecimal input is
not. Ranges are
+checked in their input order and duplicates are preserved. Every range is
checked even if an
+earlier range fails; the final exit status is non-zero when any range is
invalid, unreadable, has a
+bad CRC, or contains an invalid footer.
+
+For a large list, use `--page_ranges_file` instead of `--page_ranges`:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=check_page_crc \
+ --file=/path/to/segment.dat \
+ --page_ranges_file=/path/to/page_ranges.txt
+```
+
+The ranges file contains one decimal `OFFSET SIZE` pair per line. Blank lines,
leading/trailing
+whitespace, full-line comments, and trailing comments are accepted:
+
+```text
+# column 1 data page 0
+5242880 18349
+
+32162899 2387 # column 1 ordinal root
+```
+
+Exactly one of `--page_ranges` and `--page_ranges_file` is required.
+
+### Scan consecutive pages from a known boundary
+
+Use `scan_page_crc` when the first candidate page offset and the expected end
of a continuous page
+range are known, but individual page sizes are not:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=scan_page_crc \
+ --file=/path/to/segment.dat \
+ --page_scan_start=5242880 \
+ --page_scan_end=32203820
+```
+
+The range is `[page_scan_start, page_scan_end)`. Starting at
`page_scan_start`, the scanner extends
+CRC32C one byte at a time until the following four bytes match the running
CRC, then validates the
+candidate page footer. A successful page becomes the start of the next search.
Success requires at
+least one page and requires the final page to end exactly at `page_scan_end`.
+
+`scan_page_crc` does **not** search for an arbitrary first page offset. If
`page_scan_start` points
+to padding, a corrupt extent, or the middle of a page, it reports
`no_page_boundary`. Candidate
+physical starts such as an S3 multipart boundary must be proposed from
independent evidence before
+using this operation.
+
+### Page output control
+
+Both raw CRC operations accept `--page_output`:
+
+| Value | Output |
+|---|---|
+| `all` | Print every checked or discovered page, followed by the summary.
This is the default. |
+| `errors` | Print only failed page records and `no_page_boundary`, followed
by the summary. |
+| `summary` | Suppress individual page records and print only headers and the
final summary. |
+
+A successful page record looks like:
+
+```text
+page=0 offset=5242880 size=18349 range_valid=true readable=true
actual=929431102 expect=929431102 checksum_ok=true footer_size=15
footer_ok=true page_type=DATA_PAGE
+```
+
+The final result is designed for shell automation:
+
+```text
+=== Page CRC Summary ===
+Mode: ranges
+Pages Checked: 2
+Valid: 2
+Bad CRC: 0
+Bad Footer: 0
+Unreadable: 0
+Invalid Range: 0
+Status: OK
+```
+
+The implementation uses bounded one-MiB reads for CRC calculation and
scanning. It does not load
+the complete segment or complete scan range into memory.
+
+## Dump or check segment data
+
+The default command prints the first 10 logical rows from every top-level
scalar column:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/segment.dat
+```
+
+### Row-reading flags
+
+| Flag | Default | Description |
+|---|---:|---|
+| `--rows` | `10` | Maximum logical rows read per column. Use `-1` for all
remaining rows and `0` to read no values. |
+| `--row_start` | `0` | First logical row ordinal to read. |
+| `--batch_rows` | `4096` | Maximum rows decoded by each `next_batch` call.
This bounds working memory and does not change the selected row range. |
+| `--check_only` | `false` | Decode selected rows without printing every
value. A compact result is printed for each column. |
+| `--verify_checksum` | `true` | Verify page checksums through the normal
`ColumnReader`/`PageIO` path. |
+| `--scan_segment_pages` | `false` | Independently scan data pages referenced
by ordinal indexes and dictionary pages, and print CRC/layout summaries. |
+
+`--rows` applies to each selected top-level column. Doris segments are
columnar, so dumping all
+rows from a 43-column segment prints approximately 43 values per logical row,
not 1 output line per
+row.
+
+If `--rows` exceeds the number of rows remaining after `--row_start`, the
range is clamped at the
+segment row count. `--row_start` greater than the segment row count and
`--batch_rows=0` are invalid.
+
+### Examples
+
+Print the first 1,000 rows from every scalar column:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/segment.dat \
+ --rows=1000
+```
+
+Print logical rows `[100000, 101000)`:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/segment.dat \
+ --row_start=100000 \
+ --rows=1000
+```
+
+Dump every row. Redirecting stdout is recommended for large segments:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/segment.dat \
+ --rows=-1 \
+ --verify_checksum=true \
+ > segment.dump.txt
+```
+
+Decode every row without printing values:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/segment.dat \
+ --rows=-1 \
+ --batch_rows=4096 \
+ --check_only=true \
+ --verify_checksum=true
+```
+
+For a stronger forensic check, combine full decoding with the independent page
scan:
+
+```bash
+be/output/lib/meta_tool \
+ --operation=show_segment_data \
+ --file=/path/to/segment.dat \
+ --rows=-1 \
+ --batch_rows=4096 \
+ --check_only=true \
+ --verify_checksum=true \
+ --scan_segment_pages=true
+```
+
+Successful full decoding ends with a summary similar to:
+
+```text
+=== Data Read Summary ===
+Columns Checked: 43
+Rows Per Column: 1211072
+Row Range: [0,1211072)
+Check Only: true
+Verify Checksum: true
+Status: OK
+```
+
+## Validation scope and limitations
+
+Full `check_only` decoding verifies that every selected logical row can be
read through the column
+iterator, including page checksum verification when `--verify_checksum=true`.
It exercises data,
+dictionary, encoding, compression, nullable-column, and ordinal-index read
paths needed for the
+selected columns.
+
+`--scan_segment_pages=true` independently checks every data page exposed by
each ordinal index and
+each dictionary page. It does not currently enumerate every auxiliary
zone-map, Bloom-filter, or
+bitmap-index page, so success must not be described as exhaustive validation
of all auxiliary
+indexes.
+
+`check_page_crc` and `scan_page_crc` operate on raw physical ranges and do not
prove that those
+pages are referenced by the segment footer or rowset metadata. A single CRC32C
match is not by
+itself proof that an arbitrary offset is a page boundary; footer parsing,
exact range completion,
+multiple consecutive pages, and agreement with independent metadata materially
strengthen the
+evidence. Neither operation modifies or repairs its input file.
+
+Top-level complex columns are not yet supported by `check_only`; the command
returns a non-zero
+status instead of silently reporting success. Displayed string-like values are
escaped and truncated
+after 50 bytes, so `show_segment_data` is a readability/debug dump rather than
a round-trippable
+logical export.
+
+## Exit status
+
+| Code | Meaning |
+|---:|---|
+| `0` | All requested rows or pages completed successfully; CRC scanning ended
exactly at its requested end. |
+| `1` | The file/ranges file could not be read, or a footer, checksum, page,
iterator, decoding, boundary, or early-EOF error occurred. |
+| `2` | Required flags, row/page options, range syntax, output mode, or scan
bounds are invalid. |
diff --git a/be/src/tools/meta_tool.cpp b/be/src/tools/meta_tool.cpp
index 99970f8a938..bdf060164e0 100644
--- a/be/src/tools/meta_tool.cpp
+++ b/be/src/tools/meta_tool.cpp
@@ -21,14 +21,18 @@
#include <gflags/gflags.h>
#include <cctype>
+#include <charconv>
#include <cstring>
#include <filesystem>
#include <fstream>
#include <iomanip>
#include <iostream>
+#include <limits>
#include <set>
#include <sstream>
#include <string>
+#include <string_view>
+#include <vector>
#include "common/logging.h"
#include "common/status.h"
@@ -82,7 +86,8 @@ using namespace doris;
DEFINE_string(root_path, "", "storage root path");
DEFINE_string(operation, "get_meta",
"valid operation: get_meta, flag, load_meta, delete_meta,
show_meta, "
- "show_segment_footer, show_segment_data, gen_empty_segment");
+ "show_segment_footer, show_segment_data, check_page_crc,
scan_page_crc, "
+ "gen_empty_segment");
DEFINE_int64(tablet_id, 0, "tablet_id for tablet meta");
DEFINE_int32(num_rows_per_block, 1024, "num rows per block");
DEFINE_int32(schema_hash, 0, "schema_hash for tablet meta");
@@ -94,6 +99,22 @@ DEFINE_string(output_path, "", "output directory path
(default: current director
DEFINE_int32(num_short_key_columns, 0, "number of short key columns");
DEFINE_bool(has_sequence_col, false, "whether has sequence column");
DEFINE_bool(enable_unique_key_merge_on_write, false, "whether enable unique
key merge on write");
+DEFINE_bool(scan_segment_pages, false,
+ "scan every data page referenced by the ordinal index and report
checksum ranges");
+DEFINE_int64(rows, 10,
+ "maximum logical rows to read per column for show_segment_data;
-1 means all rows");
+DEFINE_uint64(row_start, 0, "first logical row ordinal to read for
show_segment_data");
+DEFINE_uint64(batch_rows, 4096, "maximum rows per read batch for
show_segment_data");
+DEFINE_bool(check_only, false, "decode rows without printing values for
show_segment_data");
+DEFINE_bool(verify_checksum, true,
+ "verify page checksums while decoding data for show_segment_data");
+DEFINE_string(page_ranges, "",
+ "comma-separated OFFSET:SIZE ranges for the check_page_crc
operation");
+DEFINE_string(page_ranges_file, "",
+ "file containing OFFSET SIZE pairs for the check_page_crc
operation");
+DEFINE_uint64(page_scan_start, 0, "known first page offset for the
scan_page_crc operation");
+DEFINE_uint64(page_scan_end, 0, "exclusive scan end offset for the
scan_page_crc operation");
+DEFINE_string(page_output, "all", "page CRC output level: all, errors, or
summary");
std::string get_usage(const std::string& progname) {
std::stringstream ss;
@@ -110,7 +131,15 @@ std::string get_usage(const std::string& progname) {
ss << "./meta_tool --operation=batch_delete_meta
--tablet_file=file_path\n";
ss << "./meta_tool --operation=show_meta --pb_meta_path=path\n";
ss << "./meta_tool --operation=show_segment_footer
--file=/path/to/segment/file\n";
- ss << "./meta_tool --operation=show_segment_data
--file=/path/to/segment/file\n";
+ ss << "./meta_tool --operation=show_segment_data
--file=/path/to/segment/file "
+ "[--row_start=N] [--rows=N|-1] [--batch_rows=N] [--check_only] "
+ "[--verify_checksum]\n";
+ ss << "./meta_tool --operation=check_page_crc --file=/path/to/segment/file
"
+ "(--page_ranges=OFFSET:SIZE,... |
--page_ranges_file=/path/to/ranges) "
+ "[--page_output=all|errors|summary]\n";
+ ss << "./meta_tool --operation=scan_page_crc --file=/path/to/segment/file "
+ "--page_scan_start=OFFSET --page_scan_end=OFFSET "
+ "[--page_output=all|errors|summary]\n";
ss << "./meta_tool --operation=gen_empty_segment
[--output_path=/path/to/output]\n";
ss << " Generates an empty segment file (0 rows) at specified path or
current directory\n";
ss << " Default output file name: empty.dat\n";
@@ -572,23 +601,589 @@ std::string format_column_value(const doris::IColumn&
column, size_t row,
}
}
-// Read and print column data values
-void print_column_data_values(const doris::segment_v2::ColumnMetaPB&
column_meta,
- const FileReaderSPtr& file_reader, uint64_t
num_segment_rows,
- int indent_level) {
+enum class PageOutputMode { ALL, ERRORS, SUMMARY };
+
+struct RawPageRange {
+ uint64_t offset = 0;
+ uint64_t size = 0;
+};
+
+struct PageChecksumScanResult {
+ uint64_t offset = 0;
+ uint64_t size = 0;
+ bool range_valid = false;
+ bool readable = false;
+ bool checksum_ok = false;
+ bool footer_size_ok = false;
+ bool footer_ok = false;
+ uint32_t actual_checksum = 0;
+ uint32_t expected_checksum = 0;
+ uint32_t footer_size = 0;
+ PageTypePB page_type = UNKNOWN_PAGE_TYPE;
+ std::string error;
+
+ bool ok() const {
+ return range_valid && readable && checksum_ok && footer_size_ok &&
footer_ok;
+ }
+};
+
+struct PageCrcSummary {
+ uint64_t pages_checked = 0;
+ uint64_t valid = 0;
+ uint64_t bad_crc = 0;
+ uint64_t bad_footer = 0;
+ uint64_t unreadable = 0;
+ uint64_t invalid_range = 0;
+
+ void add(const PageChecksumScanResult& result) {
+ ++pages_checked;
+ if (result.ok()) {
+ ++valid;
+ return;
+ }
+ if (!result.range_valid) {
+ ++invalid_range;
+ } else if (!result.readable) {
+ ++unreadable;
+ } else {
+ if (!result.checksum_ok) {
+ ++bad_crc;
+ }
+ if (!result.footer_size_ok || !result.footer_ok) {
+ ++bad_footer;
+ }
+ }
+ }
+
+ bool ok() const { return pages_checked > 0 && valid == pages_checked; }
+};
+
+Status read_file_exact(const FileReaderSPtr& file_reader, uint64_t offset,
char* data,
+ size_t size) {
+ size_t bytes_read = 0;
+ Status status =
+ file_reader->read_at(static_cast<size_t>(offset), Slice(data,
size), &bytes_read);
+ if (!status.ok()) {
+ status.prepend("failed to read file at offset " +
std::to_string(offset) + ": ");
+ return status;
+ }
+ if (bytes_read != size) {
+ return Status::IOError("short read at offset {}: expected={},
actual={}", offset, size,
+ bytes_read);
+ }
+ return Status::OK();
+}
+
+PageChecksumScanResult check_page_checksum(const FileReaderSPtr& file_reader,
uint64_t offset,
+ uint64_t size) {
+ PageChecksumScanResult result;
+ result.offset = offset;
+ result.size = size;
+ if (size < 8) {
+ result.error = "size_less_than_8";
+ return result;
+ }
+
+ const uint64_t file_size = file_reader->size();
+ if (offset > file_size || size > file_size - offset) {
+ result.error = "out_of_range";
+ return result;
+ }
+ result.range_valid = true;
+
+ char trailer[8];
+ if (!read_file_exact(file_reader, offset + size - sizeof(trailer),
trailer, sizeof(trailer))
+ .ok()) {
+ result.error = "read_failed";
+ return result;
+ }
+ result.footer_size = doris::decode_fixed32_le(reinterpret_cast<const
uint8_t*>(trailer));
+ result.expected_checksum =
+ doris::decode_fixed32_le(reinterpret_cast<const uint8_t*>(trailer
+ 4));
+
+ constexpr size_t crc_chunk_size = 1024 * 1024;
+ size_t buffer_size = static_cast<size_t>(std::min<uint64_t>(size - 4,
crc_chunk_size));
+ std::vector<char> buffer(buffer_size);
+ uint64_t cursor = offset;
+ uint64_t remaining = size - 4;
+ uint32_t checksum = 0;
+ while (remaining > 0) {
+ size_t bytes_to_read =
static_cast<size_t>(std::min<uint64_t>(remaining, buffer.size()));
+ if (!read_file_exact(file_reader, cursor, buffer.data(),
bytes_to_read).ok()) {
+ result.error = "read_failed";
+ return result;
+ }
+ checksum = crc32c::Extend(checksum, reinterpret_cast<const
uint8_t*>(buffer.data()),
+ bytes_to_read);
+ cursor += bytes_to_read;
+ remaining -= bytes_to_read;
+ }
+ result.readable = true;
+ result.actual_checksum = checksum;
+ result.checksum_ok = result.actual_checksum == result.expected_checksum;
+
+ result.footer_size_ok = result.footer_size > 0 && result.footer_size <=
size - 8 &&
+ result.footer_size <=
std::numeric_limits<int>::max();
+ if (result.footer_size_ok) {
+ std::vector<char> footer_buffer(result.footer_size);
+ uint64_t footer_offset = offset + size - 8 - result.footer_size;
+ Status footer_status = read_file_exact(file_reader, footer_offset,
footer_buffer.data(),
+ footer_buffer.size());
+ if (footer_status.ok()) {
+ PageFooterPB footer;
+ result.footer_ok = footer.ParseFromArray(footer_buffer.data(),
+
static_cast<int>(footer_buffer.size())) &&
+ footer.has_type() &&
footer.has_uncompressed_size();
+ if (result.footer_ok) {
+ result.page_type = footer.type();
+ }
+ }
+ }
+
+ if (!result.checksum_ok) {
+ result.error = "checksum_mismatch";
+ } else if (!result.footer_size_ok) {
+ result.error = "invalid_footer_size";
+ } else if (!result.footer_ok) {
+ result.error = "invalid_footer";
+ }
+ return result;
+}
+
+PageChecksumScanResult scan_page_checksum(const FileReaderSPtr& file_reader,
+ const PagePointer& page_pointer) {
+ return check_page_checksum(file_reader, page_pointer.offset,
page_pointer.size);
+}
+
+std::string_view trim_ascii(std::string_view value) {
+ while (!value.empty() && std::isspace(static_cast<unsigned
char>(value.front()))) {
+ value.remove_prefix(1);
+ }
+ while (!value.empty() && std::isspace(static_cast<unsigned
char>(value.back()))) {
+ value.remove_suffix(1);
+ }
+ return value;
+}
+
+Status parse_uint64_value(std::string_view text, const std::string&
field_name, uint64_t& value) {
+ text = trim_ascii(text);
+ if (text.empty()) {
+ return Status::InvalidArgument("{} is empty", field_name);
+ }
+ auto [end, error] = std::from_chars(text.data(), text.data() +
text.size(), value, 10);
+ if (error != std::errc() || end != text.data() + text.size()) {
+ return Status::InvalidArgument("invalid decimal {}: '{}'", field_name,
text);
+ }
+ return Status::OK();
+}
+
+Status parse_page_range_token(std::string_view token, RawPageRange& range) {
+ token = trim_ascii(token);
+ size_t separator = token.find(':');
+ if (separator == std::string_view::npos ||
+ token.find(':', separator + 1) != std::string_view::npos) {
+ return Status::InvalidArgument("invalid page range '{}', expected
OFFSET:SIZE", token);
+ }
+ RETURN_IF_ERROR(parse_uint64_value(token.substr(0, separator), "page
offset", range.offset));
+ RETURN_IF_ERROR(parse_uint64_value(token.substr(separator + 1), "page
size", range.size));
+ return Status::OK();
+}
+
+Status parse_inline_page_ranges(const std::string& input,
std::vector<RawPageRange>& ranges) {
+ size_t begin = 0;
+ while (begin <= input.size()) {
+ size_t end = input.find(',', begin);
+ std::string_view token(input.data() + begin,
+ (end == std::string::npos ? input.size() : end)
- begin);
+ if (trim_ascii(token).empty()) {
+ return Status::InvalidArgument("page_ranges contains an empty
range");
+ }
+ RawPageRange range;
+ RETURN_IF_ERROR(parse_page_range_token(token, range));
+ ranges.push_back(range);
+ if (end == std::string::npos) {
+ break;
+ }
+ begin = end + 1;
+ }
+ return Status::OK();
+}
+
+Status parse_page_ranges_file(const std::string& file_name,
std::vector<RawPageRange>& ranges) {
+ std::ifstream input(file_name);
+ if (!input.is_open()) {
+ return Status::IOError("failed to open page ranges file {}",
file_name);
+ }
+
+ std::string line;
+ uint64_t line_number = 0;
+ while (std::getline(input, line)) {
+ ++line_number;
+ size_t comment = line.find('#');
+ std::string_view content(line.data(), comment == std::string::npos ?
line.size() : comment);
+ content = trim_ascii(content);
+ if (content.empty()) {
+ continue;
+ }
+
+ std::istringstream line_stream {std::string(content)};
+ std::string offset_text;
+ std::string size_text;
+ std::string extra;
+ if (!(line_stream >> offset_text >> size_text) || (line_stream >>
extra)) {
+ return Status::InvalidArgument("invalid page ranges file line {}:
expected OFFSET SIZE",
+ line_number);
+ }
+ RawPageRange range;
+ RETURN_IF_ERROR(parse_uint64_value(offset_text, "page offset",
range.offset));
+ RETURN_IF_ERROR(parse_uint64_value(size_text, "page size",
range.size));
+ ranges.push_back(range);
+ }
+ if (!input.eof()) {
+ return Status::IOError("failed while reading page ranges file {}",
file_name);
+ }
+ return Status::OK();
+}
+
+Status load_page_ranges(std::vector<RawPageRange>& ranges) {
+ bool has_inline_ranges = !FLAGS_page_ranges.empty();
+ bool has_ranges_file = !FLAGS_page_ranges_file.empty();
+ if (has_inline_ranges == has_ranges_file) {
+ return Status::InvalidArgument(
+ "check_page_crc requires exactly one of page_ranges or
page_ranges_file");
+ }
+ if (has_inline_ranges) {
+ RETURN_IF_ERROR(parse_inline_page_ranges(FLAGS_page_ranges, ranges));
+ } else {
+ RETURN_IF_ERROR(parse_page_ranges_file(FLAGS_page_ranges_file,
ranges));
+ }
+ if (ranges.empty()) {
+ return Status::InvalidArgument("no page ranges were provided");
+ }
+ return Status::OK();
+}
+
+Status parse_page_output_mode(PageOutputMode& mode) {
+ if (FLAGS_page_output == "all") {
+ mode = PageOutputMode::ALL;
+ } else if (FLAGS_page_output == "errors") {
+ mode = PageOutputMode::ERRORS;
+ } else if (FLAGS_page_output == "summary") {
+ mode = PageOutputMode::SUMMARY;
+ } else {
+ return Status::InvalidArgument("page_output must be all, errors, or
summary, got '{}'",
+ FLAGS_page_output);
+ }
+ return Status::OK();
+}
+
+bool should_print_page(PageOutputMode mode, const PageChecksumScanResult&
result) {
+ return mode == PageOutputMode::ALL || (mode == PageOutputMode::ERRORS &&
!result.ok());
+}
+
+void print_page_checksum_result(uint64_t page_index, const
PageChecksumScanResult& result) {
+ std::cout << "page=" << page_index << " offset=" << result.offset << "
size=" << result.size
+ << " range_valid=" << (result.range_valid ? "true" : "false")
+ << " readable=" << (result.readable ? "true" : "false")
+ << " actual=" << result.actual_checksum << " expect=" <<
result.expected_checksum
+ << " checksum_ok=" << (result.checksum_ok ? "true" : "false")
+ << " footer_size=" << result.footer_size
+ << " footer_ok=" << (result.footer_ok ? "true" : "false")
+ << " page_type=" << PageTypePB_Name(result.page_type);
+ if (!result.error.empty()) {
+ std::cout << " error=" << result.error;
+ }
+ std::cout << std::endl;
+}
+
+void print_page_crc_summary(const std::string& mode, const PageCrcSummary&
summary,
+ const std::string& status) {
+ std::cout << "\n=== Page CRC Summary ===" << std::endl;
+ std::cout << "Mode: " << mode << std::endl;
+ std::cout << "Pages Checked: " << summary.pages_checked << std::endl;
+ std::cout << "Valid: " << summary.valid << std::endl;
+ std::cout << "Bad CRC: " << summary.bad_crc << std::endl;
+ std::cout << "Bad Footer: " << summary.bad_footer << std::endl;
+ std::cout << "Unreadable: " << summary.unreadable << std::endl;
+ std::cout << "Invalid Range: " << summary.invalid_range << std::endl;
+ std::cout << "Status: " << status << std::endl;
+}
+
+Status open_page_crc_file(FileReaderSPtr& file_reader) {
+ Status status =
doris::io::global_local_filesystem()->open_file(FLAGS_file, &file_reader);
+ if (!status.ok()) {
+ status.prepend("failed to open page CRC input " + FLAGS_file + ": ");
+ }
+ return status;
+}
+
+Status check_page_crc_ranges() {
+ PageOutputMode output_mode;
+ RETURN_IF_ERROR(parse_page_output_mode(output_mode));
+ std::vector<RawPageRange> ranges;
+ RETURN_IF_ERROR(load_page_ranges(ranges));
+
+ FileReaderSPtr file_reader;
+ RETURN_IF_ERROR(open_page_crc_file(file_reader));
+ std::cout << "=== Page CRC Check ===" << std::endl;
+ std::cout << "File: " << FLAGS_file << std::endl;
+ std::cout << "File Size: " << file_reader->size() << std::endl;
+ std::cout << "Ranges: " << ranges.size() << std::endl;
+
+ PageCrcSummary summary;
+ for (size_t i = 0; i < ranges.size(); ++i) {
+ PageChecksumScanResult result =
+ check_page_checksum(file_reader, ranges[i].offset,
ranges[i].size);
+ summary.add(result);
+ if (should_print_page(output_mode, result)) {
+ print_page_checksum_result(i, result);
+ }
+ }
+
+ print_page_crc_summary("ranges", summary, summary.ok() ? "OK" :
"CORRUPTION");
+ if (!summary.ok()) {
+ return Status::Corruption(
+ "page CRC check failed: valid={}, checked={}, bad_crc={},
bad_footer={}, "
+ "unreadable={}, invalid_range={}",
+ summary.valid, summary.pages_checked, summary.bad_crc,
summary.bad_footer,
+ summary.unreadable, summary.invalid_range);
+ }
+ return Status::OK();
+}
+
+Status find_next_crc_page(const FileReaderSPtr& file_reader, uint64_t
page_offset,
+ uint64_t scan_end, PageChecksumScanResult& result,
bool& found) {
+ constexpr size_t scan_chunk_size = 1024 * 1024;
+ std::vector<char> buffer(scan_chunk_size + 4);
+ uint64_t cursor = page_offset;
+ uint32_t checksum = 0;
+ found = false;
+
+ while (scan_end - cursor >= 5) {
+ size_t bytes_to_process =
+ static_cast<size_t>(std::min<uint64_t>(scan_chunk_size,
scan_end - cursor - 4));
+ size_t bytes_to_read = bytes_to_process + 4;
+ RETURN_IF_ERROR(read_file_exact(file_reader, cursor, buffer.data(),
bytes_to_read));
+ for (size_t i = 0; i < bytes_to_process; ++i) {
+ checksum = crc32c::Extend(checksum, reinterpret_cast<const
uint8_t*>(buffer.data() + i),
+ 1);
+ uint64_t payload_size = cursor + i - page_offset + 1;
+ if (payload_size < 4) {
+ continue;
+ }
+ uint32_t expected = doris::decode_fixed32_le(
+ reinterpret_cast<const uint8_t*>(buffer.data() + i + 1));
+ if (checksum != expected) {
+ continue;
+ }
+
+ PageChecksumScanResult candidate =
+ check_page_checksum(file_reader, page_offset, payload_size
+ 4);
+ if (candidate.ok()) {
+ result = std::move(candidate);
+ found = true;
+ return Status::OK();
+ }
+ }
+ cursor += bytes_to_process;
+ }
+ return Status::OK();
+}
+
+Status scan_page_crc_range() {
+ PageOutputMode output_mode;
+ RETURN_IF_ERROR(parse_page_output_mode(output_mode));
+ if (gflags::GetCommandLineFlagInfoOrDie("page_scan_start").is_default ||
+ gflags::GetCommandLineFlagInfoOrDie("page_scan_end").is_default) {
+ return Status::InvalidArgument(
+ "scan_page_crc requires explicit page_scan_start and
page_scan_end");
+ }
+ if (FLAGS_page_scan_end <= FLAGS_page_scan_start) {
+ return Status::InvalidArgument("page_scan_end {} must be greater than
page_scan_start {}",
+ FLAGS_page_scan_end,
FLAGS_page_scan_start);
+ }
+
+ FileReaderSPtr file_reader;
+ RETURN_IF_ERROR(open_page_crc_file(file_reader));
+ if (FLAGS_page_scan_end > file_reader->size()) {
+ return Status::InvalidArgument("page_scan_end {} exceeds file size
{}", FLAGS_page_scan_end,
+ file_reader->size());
+ }
+
+ std::cout << "=== Page CRC Scan ===" << std::endl;
+ std::cout << "File: " << FLAGS_file << std::endl;
+ std::cout << "File Size: " << file_reader->size() << std::endl;
+ std::cout << "Range: [" << FLAGS_page_scan_start << "," <<
FLAGS_page_scan_end << ")"
+ << std::endl;
+
+ PageCrcSummary summary;
+ uint64_t page_offset = FLAGS_page_scan_start;
+ uint64_t page_index = 0;
+ while (page_offset < FLAGS_page_scan_end) {
+ PageChecksumScanResult result;
+ bool found = false;
+ RETURN_IF_ERROR(
+ find_next_crc_page(file_reader, page_offset,
FLAGS_page_scan_end, result, found));
+ if (!found) {
+ if (output_mode != PageOutputMode::SUMMARY) {
+ std::cout << "no_page_boundary offset=" << page_offset
+ << " remaining=" << (FLAGS_page_scan_end -
page_offset) << std::endl;
+ }
+ print_page_crc_summary("scan", summary, "CORRUPTION");
+ return Status::Corruption("no page boundary at offset {},
remaining={}", page_offset,
+ FLAGS_page_scan_end - page_offset);
+ }
+
+ summary.add(result);
+ if (should_print_page(output_mode, result)) {
+ print_page_checksum_result(page_index, result);
+ }
+ page_offset += result.size;
+ ++page_index;
+ }
+
+ print_page_crc_summary("scan", summary, summary.ok() ? "OK" :
"CORRUPTION");
+ if (!summary.ok()) {
+ return Status::Corruption("page CRC scan failed");
+ }
+ return Status::OK();
+}
+
+// Keep the forensic scan linear so every counter and printed example follows
page order.
+// NOLINTNEXTLINE(readability-function-size)
+Status print_column_page_checksums(const std::shared_ptr<ColumnReader>&
column_reader,
+ const doris::segment_v2::ColumnMetaPB&
column_meta,
+ const FileReaderSPtr& file_reader, int
indent_level) {
std::string indent(indent_level * 2, ' ');
+ doris::OlapReaderStatistics stats;
+ OrdinalIndexReader* ordinal_index = nullptr;
+ Status status = column_reader->get_ordinal_index_reader(ordinal_index,
&stats);
+ if (!status.ok()) {
+ status.prepend("failed to load ordinal index for column " +
+ std::to_string(column_meta.column_id()) + ": ");
+ return status;
+ }
- doris::FieldType field_type =
static_cast<doris::FieldType>(column_meta.type());
+ constexpr uint64_t s3_part_size = 5 * 1024 * 1024;
+ int valid_pages = 0;
+ int bad_pages = 0;
+ int unreadable_pages = 0;
+ int noncontiguous_pages = 0;
+ int multipart_crossing_pages = 0;
+ bool dict_bad = false;
+ bool dict_unreadable = false;
+ uint64_t span_begin = file_reader->size();
+ uint64_t span_end = 0;
+ uint64_t previous_end = 0;
+ bool have_previous = false;
+ int bad_examples = 0;
+
+ std::cout << indent << "Page checksum scan:" << std::endl;
+ for (auto iter = ordinal_index->begin(); iter.valid(); iter.next()) {
+ const PagePointer& pp = iter.page();
+ span_begin = std::min(span_begin, pp.offset);
+ span_end = std::max(span_end, pp.offset + pp.size);
+ if (have_previous && pp.offset != previous_end) {
+ ++noncontiguous_pages;
+ }
+ previous_end = pp.offset + pp.size;
+ have_previous = true;
- // Skip complex types for now
- if (!doris::is_scalar_type(field_type)) {
- std::cout << indent << "(Complex type - cannot display values)" <<
std::endl;
- return;
+ bool crosses_part =
+ pp.size > 0 && pp.offset / s3_part_size != (pp.offset +
pp.size - 1) / s3_part_size;
+ if (crosses_part) {
+ ++multipart_crossing_pages;
+ }
+
+ PageChecksumScanResult page = scan_page_checksum(file_reader, pp);
+ if (!page.readable) {
+ ++unreadable_pages;
+ } else if (page.ok()) {
+ ++valid_pages;
+ } else {
+ ++bad_pages;
+ }
+
+ if (iter.page_index() < 2 || crosses_part || (!page.ok() &&
bad_examples < 3)) {
+ std::cout << indent << " page=" << iter.page_index()
+ << " ordinals=" << iter.first_ordinal() << ".." <<
iter.last_ordinal()
+ << " offset=" << pp.offset << " size=" << pp.size
+ << " readable=" << (page.readable ? "true" : "false")
+ << " checksum_ok=" << (page.checksum_ok ? "true" :
"false")
+ << " actual=" << page.actual_checksum << " expect=" <<
page.expected_checksum
+ << " footer_size=" << page.footer_size
+ << " footer_ok=" << (page.footer_ok ? "true" : "false")
+ << " crosses_s3_part=" << (crosses_part ? "true" :
"false") << std::endl;
+ if (!page.ok()) {
+ ++bad_examples;
+ }
+ }
}
- if (num_segment_rows == 0) {
- std::cout << indent << "(No data)" << std::endl;
- return;
+ if (column_meta.has_dict_page()) {
+ PagePointer dict_page(column_meta.dict_page());
+ PageChecksumScanResult page = scan_page_checksum(file_reader,
dict_page);
+ std::cout << indent << " dict offset=" << dict_page.offset << "
size=" << dict_page.size
+ << " readable=" << (page.readable ? "true" : "false")
+ << " checksum_ok=" << (page.checksum_ok ? "true" : "false")
+ << " actual=" << page.actual_checksum << " expect=" <<
page.expected_checksum
+ << " footer_size=" << page.footer_size
+ << " footer_ok=" << (page.footer_ok ? "true" : "false") <<
std::endl;
+ if (!page.readable) {
+ dict_unreadable = true;
+ } else if (!page.ok()) {
+ dict_bad = true;
+ }
+ }
+
+ std::cout << indent << " summary pages=" <<
ordinal_index->num_data_pages()
+ << " valid=" << valid_pages << " bad=" << bad_pages
+ << " unreadable=" << unreadable_pages << " span=[" << span_begin
<< "," << span_end
+ << ") noncontiguous=" << noncontiguous_pages
+ << " crosses_s3_part=" << multipart_crossing_pages << std::endl;
+
+ if (bad_pages > 0 || unreadable_pages > 0 || dict_bad || dict_unreadable) {
+ return Status::Corruption(
+ "page checksum scan failed for column {}: data_bad={},
data_unreadable={}, "
+ "dict_bad={}, dict_unreadable={}",
+ column_meta.column_id(), bad_pages, unreadable_pages,
dict_bad, dict_unreadable);
+ }
+ return Status::OK();
+}
+
+Status validate_segment_data_options(uint64_t num_segment_rows, uint64_t*
rows_to_read) {
+ if (FLAGS_rows < -1) {
+ return Status::InvalidArgument("rows must be -1 or non-negative, got
{}", FLAGS_rows);
+ }
+ if (FLAGS_batch_rows == 0 || FLAGS_batch_rows >
std::numeric_limits<size_t>::max()) {
+ return Status::InvalidArgument("batch_rows must be in [1, {}], got {}",
+ std::numeric_limits<size_t>::max(),
FLAGS_batch_rows);
+ }
+ if (FLAGS_row_start > num_segment_rows) {
+ return Status::InvalidArgument("row_start {} exceeds segment row count
{}", FLAGS_row_start,
+ num_segment_rows);
+ }
+
+ uint64_t remaining_rows = num_segment_rows - FLAGS_row_start;
+ *rows_to_read = FLAGS_rows == -1
+ ? remaining_rows
+ :
std::min<uint64_t>(static_cast<uint64_t>(FLAGS_rows), remaining_rows);
+ return Status::OK();
+}
+
+// Read and print column data values. Keep reader setup and the bounded decode
loop together so
+// failures retain exact column/row context.
+// NOLINTNEXTLINE(readability-function-size)
+Status print_column_data_values(const doris::segment_v2::ColumnMetaPB&
column_meta,
+ const FileReaderSPtr& file_reader, uint64_t
num_segment_rows,
+ uint64_t row_start, uint64_t rows_to_read, int
indent_level) {
+ std::string indent(indent_level * 2, ' ');
+
+ auto field_type = static_cast<doris::FieldType>(column_meta.type());
+
+ if (!doris::is_scalar_type(field_type)) {
+ return Status::NotSupported("cannot read complex column {} as scalar
data",
+ column_meta.column_id());
}
// Create a virtual TabletColumn for the column
@@ -602,24 +1197,24 @@ void print_column_data_values(const
doris::segment_v2::ColumnMetaPB& column_meta
// Create column reader
ColumnReaderOptions reader_opts;
- reader_opts.verify_checksum = false; // Don't verify checksum for
performance
+ reader_opts.verify_checksum = FLAGS_verify_checksum;
std::shared_ptr<ColumnReader> column_reader;
Status status = ColumnReader::create(reader_opts, column_meta,
num_segment_rows, file_reader,
&column_reader);
if (!status.ok()) {
- std::cout << indent << "(Failed to create column reader: " <<
status.to_string() << ")"
- << std::endl;
- return;
+ status.prepend("failed to create reader for column " +
+ std::to_string(column_meta.column_id()) + ": ");
+ return status;
}
// Create column iterator
ColumnIteratorUPtr iterator;
status = column_reader->new_iterator(&iterator, &tablet_column);
if (!status.ok()) {
- std::cout << indent << "(Failed to create column iterator: " <<
status.to_string() << ")"
- << std::endl;
- return;
+ status.prepend("failed to create iterator for column " +
+ std::to_string(column_meta.column_id()) + ": ");
+ return status;
}
// Initialize iterator
@@ -630,77 +1225,116 @@ void print_column_data_values(const
doris::segment_v2::ColumnMetaPB& column_meta
status = iterator->init(iter_opts);
if (!status.ok()) {
- std::cout << indent << "(Failed to initialize column iterator: " <<
status.to_string()
- << ")" << std::endl;
- return;
+ status.prepend("failed to initialize iterator for column " +
+ std::to_string(column_meta.column_id()) + ": ");
+ return status;
+ }
+
+ if (FLAGS_scan_segment_pages) {
+ RETURN_IF_ERROR(
+ print_column_page_checksums(column_reader, column_meta,
file_reader, indent_level));
+ }
+
+ if (rows_to_read == 0) {
+ if (FLAGS_check_only) {
+ std::cout << indent << "Data check: rows=0 range=[" << row_start
<< "," << row_start
+ << ") batches=0 verify_checksum="
+ << (FLAGS_verify_checksum ? "true" : "false") << "
status=OK" << std::endl;
+ } else {
+ std::cout << indent << "Data Values (rows [" << row_start << ","
<< row_start << ") of "
+ << num_segment_rows << "):" << std::endl;
+ }
+ return Status::OK();
}
- // Seek to the beginning
- status = iterator->seek_to_ordinal(0);
+ status = iterator->seek_to_ordinal(row_start);
if (!status.ok()) {
- std::cout << indent << "(Failed to seek to ordinal 0: " <<
status.to_string() << ")"
- << std::endl;
- return;
+ status.prepend("failed to seek column " +
std::to_string(column_meta.column_id()) +
+ " to row " + std::to_string(row_start) + ": ");
+ return status;
}
- // Create destination column for reading data
auto data_type =
doris::DataTypeFactory::instance().create_data_type(column_meta);
if (!data_type) {
- std::cout << indent << "(Failed to create data type for field type "
- << static_cast<int>(field_type) << ")" << std::endl;
- return;
+ return Status::InternalError("failed to create data type for column
{}, field type {}",
+ column_meta.column_id(),
static_cast<int>(field_type));
}
- doris::MutableColumnPtr dst_column = data_type->create_column();
-
- // Determine how many rows to display (max 10 rows for readability)
- const size_t max_display_rows = 10;
- size_t rows_to_read = std::min(static_cast<size_t>(num_segment_rows),
max_display_rows);
- size_t rows_read = rows_to_read;
-
- status = iterator->next_batch(&rows_read, dst_column);
- if (!status.ok()) {
- std::cout << indent << "(Failed to read column data: " <<
status.to_string() << ")"
- << std::endl;
- return;
+ if (!FLAGS_check_only) {
+ std::cout << indent << "Data Values (rows [" << row_start << ","
+ << (row_start + rows_to_read) << ") of " << num_segment_rows
<< "):" << std::endl;
}
- if (rows_read == 0) {
- std::cout << indent << "(No data read)" << std::endl;
- return;
- }
+ uint64_t decoded_rows = 0;
+ uint64_t batches = 0;
+ while (decoded_rows < rows_to_read) {
+ uint64_t current_row = row_start + decoded_rows;
+ size_t requested_rows = static_cast<size_t>(
+ std::min<uint64_t>(FLAGS_batch_rows, rows_to_read -
decoded_rows));
+ size_t rows_read = requested_rows;
+ doris::MutableColumnPtr dst_column = data_type->create_column();
+
+ status = iterator->next_batch(&rows_read, dst_column);
+ if (!status.ok()) {
+ status.prepend("failed to read column " +
std::to_string(column_meta.column_id()) +
+ " at row " + std::to_string(current_row) + ": ");
+ return status;
+ }
+ if (rows_read == 0) {
+ return Status::Corruption(
+ "column {} reached an unexpected EOF at row {}, expected
range end {}",
+ column_meta.column_id(), current_row, row_start +
rows_to_read);
+ }
+ if (rows_read > requested_rows || dst_column->size() != rows_read) {
+ return Status::Corruption(
+ "column {} returned an invalid batch at row {}:
requested={}, read={}, "
+ "values={}",
+ column_meta.column_id(), current_row, requested_rows,
rows_read,
+ dst_column->size());
+ }
- // Print the values
- std::cout << indent << "Data Values (" << rows_read << " of " <<
num_segment_rows
- << " rows, showing first " << std::min(rows_read,
max_display_rows)
- << "):" << std::endl;
-
- for (size_t i = 0; i < rows_read; ++i) {
- std::cout << indent << " [" << i << "] ";
- if (column_meta.is_nullable()) {
- const auto& nullable_col = assert_cast<const
doris::ColumnNullable&>(*dst_column);
- if (nullable_col.is_null_at(i)) {
- std::cout << "NULL";
- } else {
- const doris::IColumn& nested_col =
nullable_col.get_nested_column();
- std::cout << format_column_value(nested_col, i, field_type);
+ if (!FLAGS_check_only) {
+ for (size_t i = 0; i < rows_read; ++i) {
+ std::cout << indent << " [" << (current_row + i) << "] ";
+ if (column_meta.is_nullable()) {
+ const auto& nullable_col =
+ assert_cast<const
doris::ColumnNullable&>(*dst_column);
+ if (nullable_col.is_null_at(i)) {
+ std::cout << "NULL";
+ } else {
+ const doris::IColumn& nested_col =
nullable_col.get_nested_column();
+ std::cout << format_column_value(nested_col, i,
field_type);
+ }
+ } else {
+ std::cout << format_column_value(*dst_column, i,
field_type);
+ }
+ std::cout << std::endl;
}
- } else {
- std::cout << format_column_value(*dst_column, i, field_type);
}
- std::cout << std::endl;
+
+ decoded_rows += rows_read;
+ ++batches;
}
- if (num_segment_rows > max_display_rows) {
- std::cout << indent << " ... (" << (num_segment_rows -
max_display_rows) << " more rows)"
- << std::endl;
+ if (FLAGS_check_only) {
+ std::cout << indent << "Data check: rows=" << decoded_rows << "
range=[" << row_start << ","
+ << (row_start + decoded_rows) << ") batches=" << batches
+ << " verify_checksum=" << (FLAGS_verify_checksum ? "true" :
"false")
+ << " status=OK" << std::endl;
+ } else if (rows_to_read < num_segment_rows) {
+ std::cout << indent << " ... (" << (num_segment_rows - rows_to_read)
+ << " rows outside selected range)" << std::endl;
}
+
+ return Status::OK();
}
-// Helper function to print column metadata
-void print_column_meta(const doris::segment_v2::ColumnMetaPB& column_meta,
- const FileReaderSPtr& file_reader, uint64_t
num_segment_rows,
- int indent_level) {
+// Helper function to print column metadata. The output intentionally mirrors
protobuf field order
+// for forensic readability.
+// NOLINTNEXTLINE(readability-function-size)
+Status print_column_meta(const doris::segment_v2::ColumnMetaPB& column_meta,
+ const FileReaderSPtr& file_reader, uint64_t
num_segment_rows,
+ uint64_t row_start, uint64_t rows_to_read, int
indent_level) {
std::string indent(indent_level * 2, ' ');
std::string column_name;
if (column_meta.has_column_path_info() &&
column_meta.column_path_info().has_path()) {
@@ -709,7 +1343,7 @@ void print_column_meta(const
doris::segment_v2::ColumnMetaPB& column_meta,
column_name = "column_id_" + std::to_string(column_meta.column_id());
}
- doris::FieldType field_type =
static_cast<doris::FieldType>(column_meta.type());
+ auto field_type = static_cast<doris::FieldType>(column_meta.type());
std::cout << indent << "=== " << column_name << ": type=" <<
get_field_type_string(field_type)
<< ", nullable=" << (column_meta.is_nullable() ? "true" :
"false")
<< ", encoding=" << get_encoding_string(column_meta.encoding())
@@ -740,7 +1374,9 @@ void print_column_meta(const
doris::segment_v2::ColumnMetaPB& column_meta,
if (column_meta.indexes_size() > 0) {
std::cout << indent << "Indexes: ";
for (int i = 0; i < column_meta.indexes_size(); ++i) {
- if (i > 0) std::cout << ", ";
+ if (i > 0) {
+ std::cout << ", ";
+ }
const auto& index_meta = column_meta.indexes(i);
if (index_meta.has_type()) {
switch (index_meta.type()) {
@@ -767,19 +1403,27 @@ void print_column_meta(const
doris::segment_v2::ColumnMetaPB& column_meta,
// Handle complex types recursively
if (column_meta.children_columns_size() > 0) {
+ if (FLAGS_check_only) {
+ return Status::NotSupported(
+ "check_only does not yet support complex column {} with {}
children",
+ column_meta.column_id(),
column_meta.children_columns_size());
+ }
std::cout << indent << "Sub-columns: " <<
column_meta.children_columns_size() << std::endl;
for (int i = 0; i < column_meta.children_columns_size(); ++i) {
- print_column_meta(column_meta.children_columns(i), file_reader,
num_segment_rows,
- indent_level + 1);
+ RETURN_IF_ERROR(print_column_meta(column_meta.children_columns(i),
file_reader,
+ num_segment_rows, row_start,
rows_to_read,
+ indent_level + 1));
}
- return;
+ return Status::OK();
}
// Print column data values for scalar types
if (doris::is_scalar_type(field_type)) {
- print_column_data_values(column_meta, file_reader, num_segment_rows,
indent_level);
+ return print_column_data_values(column_meta, file_reader,
num_segment_rows, row_start,
+ rows_to_read, indent_level);
} else {
- std::cout << indent << "(Complex type - cannot display values)" <<
std::endl;
+ return Status::NotSupported("cannot display values for column {} with
type {}",
+ column_meta.column_id(),
get_field_type_string(field_type));
}
}
@@ -791,7 +1435,9 @@ ACCESS_PRIVATE_FIELD(ExecEnv_orphan_mem_tracker,
doris::ExecEnv,
ACCESS_PRIVATE_STATIC_FIELD(ExecEnv_tracking_memory, doris::ExecEnv,
std::atomic_bool,
_s_tracking_memory);
-void show_segment_data(const std::string& file_name) {
+// Keep report sections in execution order so a failure never prints a
misleading final summary.
+// NOLINTNEXTLINE(readability-function-size)
+Status show_segment_data(const std::string& file_name) {
// Initialize ExecEnv components needed for ColumnReader
// Use macro to access private members temporarily
auto* exec_env = doris::ExecEnv::GetInstance();
@@ -815,23 +1461,30 @@ void show_segment_data(const std::string& file_name) {
doris::io::FileReaderSPtr file_reader;
Status status = doris::io::global_local_filesystem()->open_file(file_name,
&file_reader);
if (!status.ok()) {
- std::cout << "open file failed: " << status << std::endl;
- return;
+ status.prepend("failed to open segment file " + file_name + ": ");
+ return status;
}
SegmentFooterPB footer;
status = get_segment_footer(file_reader.get(), &footer);
if (!status.ok()) {
- std::cout << "get footer failed: " << status.to_string() << std::endl;
- return;
+ status.prepend("failed to read segment footer from " + file_name + ":
");
+ return status;
}
+ uint64_t rows_to_read = 0;
+ RETURN_IF_ERROR(validate_segment_data_options(footer.num_rows(),
&rows_to_read));
+
// Print basic info
std::cout << "\n=== Segment File Info ===" << std::endl;
std::cout << "File: " << file_name << std::endl;
std::cout << "Num Rows: " << footer.num_rows() << std::endl;
std::cout << "Num Columns: " << footer.columns_size() << std::endl;
std::cout << "Compression: " <<
get_compression_string(footer.compress_type()) << std::endl;
+ std::cout << "Selected Row Range: [" << FLAGS_row_start << ","
+ << (FLAGS_row_start + rows_to_read) << ")" << std::endl;
+ std::cout << "Check Only: " << (FLAGS_check_only ? "true" : "false") <<
std::endl;
+ std::cout << "Verify Checksum: " << (FLAGS_verify_checksum ? "true" :
"false") << std::endl;
if (footer.has_version()) {
std::cout << "Version: " << footer.version() << std::endl;
}
@@ -849,7 +1502,8 @@ void show_segment_data(const std::string& file_name) {
// Print each column
for (int i = 0; i < footer.columns_size(); ++i) {
const auto& column_meta = footer.columns(i);
- print_column_meta(column_meta, file_reader, footer.num_rows(), 0);
+ RETURN_IF_ERROR(print_column_meta(column_meta, file_reader,
footer.num_rows(),
+ FLAGS_row_start, rows_to_read, 0));
// Collect statistics
if (column_meta.has_compressed_data_bytes()) {
@@ -918,6 +1572,16 @@ void show_segment_data(const std::string& file_name) {
<< std::setprecision(2) << (footer.data_footprint() /
1024.0) << " KB)"
<< std::endl;
}
+
+ std::cout << "\n=== Data Read Summary ===" << std::endl;
+ std::cout << "Columns Checked: " << footer.columns_size() << std::endl;
+ std::cout << "Rows Per Column: " << rows_to_read << std::endl;
+ std::cout << "Row Range: [" << FLAGS_row_start << "," << (FLAGS_row_start
+ rows_to_read) << ")"
+ << std::endl;
+ std::cout << "Check Only: " << (FLAGS_check_only ? "true" : "false") <<
std::endl;
+ std::cout << "Verify Checksum: " << (FLAGS_verify_checksum ? "true" :
"false") << std::endl;
+ std::cout << "Status: OK" << std::endl;
+ return Status::OK();
}
void init_common_components() {
@@ -1089,11 +1753,29 @@ int main(int argc, char** argv) {
show_segment_footer(FLAGS_file);
} else if (FLAGS_operation == "show_segment_data") {
if (FLAGS_file == "") {
- std::cout << "no file flag for show_segment_data" << std::endl;
- return -1;
+ std::cerr << "no file flag for show_segment_data" << std::endl;
+ return 2;
+ }
+ init_common_components();
+ Status status = show_segment_data(FLAGS_file);
+ if (!status.ok()) {
+ std::cerr << "show_segment_data failed: " << status.to_string() <<
std::endl;
+ gflags::ShutDownCommandLineFlags();
+ return status.is<ErrorCode::INVALID_ARGUMENT>() ? 2 : 1;
+ }
+ } else if (FLAGS_operation == "check_page_crc" || FLAGS_operation ==
"scan_page_crc") {
+ if (FLAGS_file.empty()) {
+ std::cerr << "no file flag for " << FLAGS_operation << std::endl;
+ return 2;
}
init_common_components();
- show_segment_data(FLAGS_file);
+ Status status = FLAGS_operation == "check_page_crc" ?
check_page_crc_ranges()
+ :
scan_page_crc_range();
+ if (!status.ok()) {
+ std::cerr << FLAGS_operation << " failed: " << status.to_string()
<< std::endl;
+ gflags::ShutDownCommandLineFlags();
+ return status.is<ErrorCode::INVALID_ARGUMENT>() ? 2 : 1;
+ }
} else if (FLAGS_operation == "gen_empty_segment") {
gen_empty_segment();
} else {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
