JNSimba opened a new pull request, #68079:
URL: https://github.com/apache/doris/pull/68079
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
CDC streaming jobs persist the source schema used by their readers. If a
schema change cannot be applied to Doris, an administrator can repair the
target table, but the job cannot discard the stale persisted source schema or
identify the exact source position without recreating the job.
This change adds a one-shot `reload_source_schema` ALTER JOB property. It
clears the persisted CDC source schema so the rebuilt reader loads the current
source schema, and it can be used alone or together with an explicit offset.
CDC record-processing errors now include the source table, source offset when
available, failed DDL, and root cause. The regression coverage verifies that
data written before the DDL, after the failed DDL, and after recovery is
preserved.
Example recovery:
```sql
ALTER JOB job_name PROPERTIES (
"offset" = '{"file":"mysql-bin.000001","pos":1234}',
"reload_source_schema" = "true"
);
```
### Release note
CDC streaming jobs can reload their persisted source schema through ALTER
JOB, and schema-change failures expose the failed DDL and source offset when
available.
### Check List (For Author)
- Test
- [ ] Regression test
- Added MySQL manual recovery coverage and extended MySQL/PostgreSQL
schema-change error coverage. Tests were not run locally.
- [ ] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test
- Behavior changed:
- [ ] No.
- [x] Yes. ALTER JOB accepts the one-shot `reload_source_schema`
property for CDC jobs, and CDC task errors include source record context.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new ALTER JOB property should be added to the streaming
job documentation.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
