HappenLee opened a new issue, #68240:
URL: https://github.com/apache/doris/issues/68240
### Search before asking
Related issues #67994 and #67995 describe reader-cache and ZoneMap failures.
This report tracks the broader consistency problem across hidden-column
pruning, statistics, and direct reads, including short-circuit point queries
and two-phase row-ID fetches that do not use the normal scan materialization
path.
### Version
Apache Doris master. The existing reader/cache and pruning reports describe
baseline `846d9b2ebfc595ddfd77579447db16a674c901f7`.
Related proposed implementation: #68125, inspected at
`0022410d4b3cddbb48c548c47bc22b4a7e99d5d8`. Its regression scenarios are
referenced below. No new cluster reproduction or test execution was performed
when filing this report.
### What's Wrong?
Some hidden columns store placeholders in segment files and obtain their
logical values from the owning rowset during a read:
- `__DORIS_VERSION_COL__`: a single-version rowset can supply its version
instead of the physical `0`.
- `__DORIS_COMMIT_TSO_COL__`: a published single-version rowset can supply
its assigned commit TSO instead of the physical `0`.
- `__DORIS_BINLOG_TSO__`: the row-binlog read path can materialize a TSO
instead of its physical placeholder, subject to the row-binlog read context.
These semantics are not interchangeable with reading the physical column or
its indexes. Several paths can bypass or precede normal hidden-column
materialization:
1. **Pruning:** physical segment/page ZoneMaps, Bloom filters, or inverted
indexes can reject rows based on placeholders rather than the logical values
that row evaluation will see.
2. **Statistics:** a pushed-down MIN/MAX path can consume placeholder
statistics rather than the logical hidden-column values.
3. **Reader reuse:** a reader cache keyed only by column identity can return
a physical reader when a logical constant is required, or reuse a
context-dependent constant where a physical read is intended.
4. **Direct reads:** short-circuit point queries and lazy two-phase row-ID
fetches can read physical column values or deserialize a JSONB row-store
payload without performing the replacement used by a normal `SegmentIterator`
scan. A JSONB row cache also lacks the owning rowset context needed to derive
the logical value.
As a result, the returned hidden value or matching row set can depend on the
execution path, cache warmup order, or whether row storage is enabled.
### What You Expected?
Logical projection, predicate evaluation, index pruning, and pushed-down
statistics must agree on the hidden-column values derived from the exact owning
rowset.
Point queries and two-phase reads must return the same logical
VERSION/COMMIT_TSO values as normal scans. Multi-version rowsets, unpublished
internal reads, deliberately physical accesses, and row-binlog-specific
BINLOG_TSO handling must retain their respective semantics; they must not
receive an unconditional constant replacement.
### How to Reproduce?
The following point-query scenario is adapted from the public regression
case in #68125. It is a proposed comparison for an affected build; it has not
been rerun for this report.
```sql
CREATE TABLE hidden_column_point_read (
k INT,
v INT
) UNIQUE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"enable_unique_key_merge_on_write" = "true",
"light_schema_change" = "true",
"store_row_column" = "true",
"disable_auto_compaction" = "true"
);
INSERT INTO hidden_column_point_read VALUES (1, 10);
SET show_hidden_columns = true;
SELECT __DORIS_VERSION_COL__
FROM hidden_column_point_read WHERE k = 1;
SELECT /*+ SET_VAR(enable_short_circuit_query=false) */
__DORIS_VERSION_COL__
FROM hidden_column_point_read WHERE k = 1;
```
Verify that the first query actually uses the short-circuit path. Both
queries should return the same positive published version, rather than one path
returning the physical `0`. Repeat with partial row storage and with an
ordinary-column point query first to warm the row cache.
Additional validation should cover:
- Two singleton rowsets whose sort keys interleave in a lazy TopN result.
Assert that the plan contains `PhysicalLazyMaterialize`, and compare each
fetched row's hidden version with its source rowset. Exercise both column
storage and full row storage; replacement must apply only to the newly appended
rows.
- COMMIT_TSO projection and predicates on a ROW-binlog-enabled base table,
using source rowset metadata as an independent oracle.
- Ordinary predicates and common expressions at both segment and page
levels, with reader caches initially cold and warm.
- MIN/MAX with execution-plan assertions proving that the statistics
pushdown path is active. A query that silently falls back to a normal scan does
not validate that optimization.
- BINLOG_TSO through the actual row-binlog scan context, plus controls for
physical/internal and multi-version reads.
### Anything Else?
The proposed direction in #68125 is to centralize hidden-column
identification and read-time value resolution, use logical summaries where
available, and avoid pruning from physical indexes that do not describe those
values. Context-dependent logical constants should not pollute the shared
physical reader cache. Direct reads must retain the exact owning rowset and
resolve VERSION/COMMIT_TSO accordingly; BINLOG_TSO materialization needs the
row-binlog scan context.
Statistics fast paths should be used only when the reader can supply the
correct logical result; otherwise they should fall back to normal iteration.
Public implementation and test references:
- [Hidden-column value
resolution](https://github.com/apache/doris/blob/0022410d4b3cddbb48c548c47bc22b4a7e99d5d8/be/src/storage/read_time_hidden_column.cpp).
- [Point-query and interleaved lazy-fetch regression
scenarios](https://github.com/apache/doris/blob/0022410d4b3cddbb48c548c47bc22b4a7e99d5d8/regression-test/suites/point_query_p0/test_point_query_read_time_hidden_columns.groovy).
- #68125 addresses the broader hidden-column read contract.
- #68097 addresses the COMMIT_TSO reader-entry-point and direct-read subset.
- #67994 and #67995 contain the earlier cache and ZoneMap analyses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
