danny0405 opened a new pull request, #20122: URL: https://github.com/apache/hudi/pull/20122
### Describe the issue this Pull Request addresses Flink initializes a table for each write bucket and repeatedly validates the same writer schema against commit metadata. With the default settings, disabling Avro compatibility validation does not disable column-drop protection, so these repeated checks can still load the timeline and schema. Validate once before the coordinator publishes each new instant, while retaining column-drop protection and skipping metadata-table schema validation. ### Summary and Changelog - Move Flink schema and table-property validation into `HoodieFlinkWriteClient.preTxn`; remove schema checks from the bucket and prepared-record write methods. - Include the complex-key-generator encoding check in an operation-aware table-property validator. Keep the original two-argument public validator and retain validation during initialization for other engines through `doInitTable`. - Refresh the coordinator meta client before validation so changes to table properties and secondary-index definitions are visible alongside the current timeline. - Cover validation frequency, failure before instant publication, metadata/prepared writes, property and index changes after coordinator startup, and Java encoding initialization. No code was copied from another project. ### Impact Flink validates against the metadata visible when each instant is created, instead of validating each bucket separately. Direct users of the Flink write client must call `preTxn` to perform these validations before writing. Schema errors retain the underlying schema-compatibility exception without the insert/upsert wrapper. The existing public table-property validator remains available; the new overload accepts the meta client and operation type. There are no configuration-default or storage-format changes. Internal-schema loading during table construction is unchanged and remains separate follow-up work. ### Risk Level medium Validation moves across the coordinator/task boundary, and the common initialization hook changes. Refreshing properties and index definitions once per instant prevents stale coordinator metadata from bypassing validation. This adds one metadata refresh per instant while eliminating repeated bucket-level schema checks; no throughput benchmark was run. Validation: Maven reactor build and Checkstyle passed. Of 115 selected tests, 114 passed and one pre-existing disabled coordinator test was skipped: - `TestBaseHoodieWriteClient`: 27 property-validation and encoding tests. - `TestHoodieJavaWriteClientComplexKeyGenEncoding`: 1 test. - `TestFlinkWriteClient`: 9 tests. - `TestStreamWriteOperatorCoordinator`: full class, 57 passed and 1 disabled (existing issue #19922), including concurrent requests and recovery coverage. - `TestFlinkWriteClientFunctional`: 19 tests. - `ITTestDataStreamWrite#testColumnDroppingIsNotAllowed`: 1 integration test. `git diff --check` also passed. The full repository test suite was not run. ### Documentation Update none — no new feature, configuration, default, or storage format. Updated source documentation explains the coordinator validation lifecycle. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
