airborne12 opened a new pull request, #69729:
URL: https://github.com/apache/doris/pull/69729
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: In cloud mode, a RESTORE of selected partitions into an
existing table creates the tablets of the restored partitions with the target
table's default inverted index storage format, and registers the partitions
without a format of their own. The restored files, however, were written in the
format of the backed-up table, which can differ from the target's default: an
SNII partition restored into a V2 table, for instance. With the partition-level
format rollout on, the tablet metadata then disagrees with the files, and a
MATCH on the restored data fails with "CLucene read past EOF" until a full
compaction rewrites it.
The format is now taken from the backup: the backed-up partition's own entry
in its PartitionInfo, or else the backed-up table's default. CloudRestoreJob
remembers it per restored partition when it resets the partition, because the
reset replaces the remote partition id that the entry is keyed by, and uses it
when it creates the tablets. When the partition joins the table, both
RestoreJob paths (execution and replay) record it in the table's PartitionInfo,
but only when the partition-level format rollout is on, as for a partition
created in the table.
Cloud mode cannot BACKUP, so a restore end to end needs a local-mode source
cluster and a shared repository. That was not run for this change.
### Release note
Fix cloud-mode RESTORE of selected partitions into a table whose inverted
index storage format differs from the backed-up table's.
### Compatibility
- FE only: no BE, thrift, protobuf or meta-service change.
- Nothing changes by default.
`enable_partition_inverted_index_storage_format_rollout` is false by default.
With it off, the new code records nothing in the `PartitionInfo`, and the
format in the create-tablets request is ignored by the meta-service for a
schema (index id, schema version) that already exists, so a restored tablet
keeps the format it has today. The change takes effect only for a cluster that
turned the rollout on.
- With the rollout on, the tablets of a restored partition and its
`PartitionInfo` entry carry the format the backed-up files were written in. A
partition restored from a backup of the same format gets the same value as
before.
- Persistence: the map `CloudRestoreJob` keeps from restored partition id to
format has no `@SerializedName`, which the FE's Gson configuration skips, so
the persisted job, the edit log and the image are unchanged. The map is only
read while `checkAndPrepareMeta` creates the tablets, and that method fills it
again if the job is run again. The `PartitionInfo` entry is an existing
persisted field written through its existing setter, from both
`checkAndPrepareMeta` and `replayCheckAndPrepareMeta`, so a follower or a
restarted FE ends with the same entry as the master.
### Performance
FE only and once per restored partition: one map insert and one lookup while
the tablets are created. Nothing on a query or load path changes.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- Added RestoreJobTest.testBackedUpInvertedIndexFileStorageFormat,
RestoreJobTest.testRecordBackedUpInvertedIndexFileStorageFormat,
CloudRestoreJobTest.testCreateReplicasUsesTheBackedUpPartitionFormat and
CloudRestoreJobTest.testResetPartitionRecordsTheBackedUpFormat; RestoreJobTest
and CloudRestoreJobTest pass. Reintroducing each of the two bugs the last two
guard against (tablets created with the table default, format looked up after
the partition id was reset) makes the matching test fail.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. A restored partition keeps the format of its backed-up files
instead of the target table's default; nothing changes when the two formats are
equal.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
