damokelis opened a new pull request, #66790:
URL: https://github.com/apache/doris/pull/66790
### What problem does this PR solve?
Issue Number: close #66784
Related PR: #65868
Problem Summary:
The connector SPI path on `master` can read Paimon tables, but it still uses
Paimon 1.3.1 and does not provide a complete external-table write path. The
implementation from `branch-4.1` cannot be applied directly because catalog,
transaction, planning, scan, and write contracts have moved to the connector
modules on `master`.
This draft:
- upgrades the FE connector, BE JNI scanner/writer, and shaded Hive runtime
to Apache Paimon 2.0.0;
- adds Connector SPI write planning and a Paimon table sink backed by the
Paimon Java SDK;
- aggregates BE commit messages in the FE transaction and provides
acknowledgement, retry reconciliation, abort cleanup, and statement-level
commit;
- supports external catalog `CREATE TABLE`, `DROP TABLE`, `INSERT INTO`,
`INSERT OVERWRITE`, and CTAS paths;
- maps primitive and complex values through Arrow, including Paimon 2.0
BLOB, VARIANT, and VECTOR representations;
- adds focused Java, C++, and regression coverage for schema mapping, JNI
conversion, write planning, commit transport, transaction handling, budgets,
acknowledgements, and CTAS failure cleanup.
Live filesystem/HMS/REST and S3-compatible integration validation is in
progress; results will be added before this draft is marked ready.
### Release note
Support transactional writes to Apache Paimon 2.0 external catalogs,
including CREATE/DROP, INSERT, INSERT OVERWRITE, and CTAS.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Current validation:
- Java focused tests: 86 passed
- C++ focused tests: 17 passed
- FE Maven reactor: 58 modules passed
- Checkstyle: 0 violations
- clang-format and `git diff --check`: passed
- Full Linux FE/BE build and live catalog validation: in progress
- Behavior changed:
- [ ] No.
- [x] Yes. Paimon external catalogs gain transactional DDL/DML writes
and use Paimon 2.0.0.
- Does this need documentation?
- [ ] No.
- [x] Yes. A follow-up documentation PR will describe supported catalog
properties, statements, and type mappings.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
