morningman opened a new issue, #70278: URL: https://github.com/apache/doris/issues/70278
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master @ 72e91896bbdd8760f8bd4e488e5672406b40ce52 (2026-11-03). The code paths below are unchanged on master as of 2026-11-05. ### What's Wrong? On a Unique Key Merge-on-Write table with `changelog.enable = true` and `changelog.need_historical_value = false`, DETAIL reads are accepted but return `UPDATE_BEFORE` rows that carry the new values: - `@incr(... 'incrementType' = 'DETAIL')` is accepted. The binder rejects only `MIN_DELTA` when historical values are disabled ([BindRelation.java#L778-L793](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java#L778-L793)). - `CREATE STREAM ... "type" = "detail"` is accepted. `checkAsTableStreamBaseTable()` checks only `MIN_DELTA` ([OlapTable.java#L4351-L4372](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java#L4351-L4372)); [test_create_append_only_stream_without_historical_value.groovy#L49-L63](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/regression-test/suites/table_stream_p0/test_create_append_only_stream_without_historical_value.groovy#L49-L63) creates such a stream. - Partial-column updates, including `UPDATE` statements rewritten into partial updates, still record `op = UPDATE` with the full after image. - The DETAIL reader splits every `UPDATE` into `UPDATE_BEFORE` and `UPDATE_AFTER`, and reads `UPDATE_BEFORE` through the before-column mapping ([block_reader.cpp#L395-L414](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/iterator/block_reader.cpp#L395-L414)). Without before columns, `_init_before_column_ordinals()` maps every column to itself ([schema.cpp#L197-L216](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/be/src/storage/schema.cpp#L197-L216)), so `UPDATE_BEFORE` carries the after values. `DELETE` rows are read the same way, so they carry the after image of the delete (defaults or NULL for non-key columns) instead of the deleted row. Consumers cannot tell these rows from real before images. For example, a Flink Doris Connector job with `source.changelog.increment-type = 'detail'` receives `-U(new)` followed by `+U(new)`; any downstream operator that retracts on `-U`, such as an aggregation, retracts the wrong value. The incremental overview page states that UPDATE_BEFORE / DELETE in `min_delta` and `detail` need before images, so the base table must be MoW with `changelog.need_historical_value` enabled. ### What You Expected? Either reject DETAIL reads and `detail` streams on MoW tables without historical values, as is already done for `MIN_DELTA`, or never emit rows that look like before images when none were recorded (for example, emit only `UPDATE_AFTER`). ### How to Reproduce? ```sql -- fe.conf: enable_feature_changelog = true CREATE TABLE t2 (k INT, v1 INT, v2 INT, v3 INT) UNIQUE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "enable_unique_key_merge_on_write" = "true", "changelog.enable" = "true" -- changelog.need_historical_value is left at false ); INSERT INTO t2 VALUES (1, 10, 10, 10); SET enable_unique_key_partial_update = true; INSERT INTO t2 (k, v1) VALUES (1, 20); -- recorded as UPDATE with the full after image SET enable_unique_key_partial_update = false; SELECT __DORIS_CHANGELOG_OP__, k, v1, v2, v3 FROM t2@incr('startTimestamp' = '<a time before the first INSERT>', 'incrementType' = 'DETAIL') ORDER BY __DORIS_CHANGELOG_TSO__, __DORIS_CHANGELOG_LSN__, __DORIS_CHANGELOG_OP__; -- suspected: the UPDATE_BEFORE row (op = 2) shows v1 = 20, identical to UPDATE_AFTER (op = 3) ``` ### Anything Else? Found by code reading while documenting the changelog and Table Stream features; the reproduction above has not been run yet. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
