Yukang-Lian opened a new pull request, #67239:
URL: https://github.com/apache/doris/pull/67239
### What problem does this PR solve?
Issue Number: None
Related PR: #63173
Problem Summary:
Light schema change uses column unique IDs to match FE slots with columns
stored
in existing segments. Some supported legacy metadata can contain a
`maxColUniqueId` lower than an ID that is still present in the current
schema.
The `DROP COLUMN` path replaced the index schema before preserving its
unique-ID
high-water mark. Dropping the highest-ID column could therefore lose that
historical ID, and a later `ADD COLUMN` could assign it again. Existing
segments
could then be interpreted as containing values for the new column.
This change normalizes `maxColUniqueId` when metadata is deserialized and
preserves the maximum from both the old and new schemas during light schema
changes. It prevents future drops from losing a high-water mark that is still
available in current metadata.
This is a preventive fix for metadata whose high-water mark is still
inferable.
It cannot reconstruct an ID that was already absent from both the schema and
`maxColUniqueId` before upgrade.
### Release note
Prevent future light schema changes from reusing dropped column unique IDs
when
a legacy high-water mark is stale. Metadata that had already lost its
historical
high-water mark before upgrade is not repaired by this change.
### Check List (For Author)
- Test
- [x] Regression test
- Added a Docker fault-injection test that writes an old segment,
drops
and re-adds a column, verifies the old row returns `NULL`, and
verifies
a new row returns the newly written value.
- Ran `LOCAL_DORIS_PATH=$PWD/.doris-compose ./run-regression-test.sh
--run -d fault_injection_p0 -s test_readd_dropped_column_unique_id -image
doris-pr63173-test:latest -runMode not_cloud`.
- [x] Unit Test
- Added DDL coverage for non-reused IDs and metadata deserialization
coverage for repairing a stale maximum while preserving a higher
one.
- Ran `./run-fe-ut.sh --run
'org.apache.doris.alter.SchemaChangeHandlerTest#testReaddDroppedValueColumnUsesNewUniqueId,org.apache.doris.catalog.MaterializedIndexMetaTest'`.
- Ran `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe`.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Light schema changes preserve the column unique-ID high-water
mark so newly added columns do not reuse IDs still known to the
metadata.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
