fhan688 opened a new pull request, #20281:
URL: https://github.com/apache/hudi/pull/20281
### Describe the issue this Pull Request addresses
Part of #19779.
This PR builds on the append-only Flink SQL sink path introduced in #20198
and adds the first bounded mutable Copy-On-Write path to Hudi Architect.
Previously, mutable Flink workloads were detected but always blocked. This
change allows a deliberately narrow subset of new-table, single-writer mutable
workloads to reach `CONFIG_VALIDATED`, while unsupported mutable configurations
continue to fail closed.
This PR does not close #19779. MOR, existing-table takeover, external
catalogs, multiple writers, alternative indexes, custom mergers, and broader
CDC semantics remain deferred.
### Summary and Changelog
This PR introduces design-contract schema 2 for the first mutable Flink
SQL COW sink path.
An executable mutable design must satisfy all of the following conditions:
- Apache Hudi 1.2.0 and Apache Flink 1.20, with Flink 1.20.1 used by the
pinned fixture;
- a new Hudi table managed through this Architect path;
- one confirmed writer and no external-catalog requirement;
- streaming execution with checkpointing enabled and an interval of at
least 1000 ms;
- a stable, non-null record key shared by the source primary key and
target primary key;
- exactly one non-null event-time ordering field using `BIGINT` or
`TIMESTAMP(p)`, where `p <= 6`;
- a normalized UPSERT source changelog containing `INSERT`,
`UPDATE_AFTER`, and optional `DELETE`, without `UPDATE_BEFORE`;
- full-row delete records when deletes are emitted;
- immutable partition values for an existing record key; and
- an explicitly empty connector pass-through option map.
For an eligible mutable design, the validator renders a canonical COW
configuration with:
- `table.type=COPY_ON_WRITE`;
- `write.operation=upsert`;
- `ordering.fields=<validated ordering field>`;
- `hoodie.write.record.merge.mode=EVENT_TIME_ORDERING`;
- `index.type=FLINK_STATE`;
- `index.global.enabled=true`;
- `index.state.ttl=0`;
- `index.bootstrap.enabled=true`; and
- `changelog.enabled=false`.
`index.bootstrap.enabled=true` closes the cold-start gap for an
Architect-managed table when Flink index state must be rebuilt from existing
table data. It does not make arbitrary existing-table takeover eligible; that
lifecycle remains blocked by the existing-table gate.
The generated table provides latest-state behavior. This PR does not claim
full changelog-history preservation or Hudi CDC query support.
Additional changes include:
- advancing the capability manifest to schema 3 and recording both
supported design-contract schemas;
- adding immutable source anchors for the Hudi 1.2.0 changelog,
merge-mode, FLINK_STATE, global-index, and bootstrap implementations;
- routing otherwise eligible mutable workloads from the PR1 safety gates
into schema-2 validation;
- retaining `FLINK_MUTABLE_COW_DEFERRED` for mutable requests still
presented through the schema-1 append-only contract;
- adding deterministic findings for unsupported mutable identity,
ordering, changelog, delete, partition, index, bootstrap, and write-path
combinations;
- adding a mutable JSON contract and exact SQL golden fixture;
- extending the pinned Hudi 1.2.0/Flink 1.20.1 Java fixture to verify the
`I/UA/D` changelog contract, absence of `UPDATE_BEFORE`, Hudi stream writer,
and index-bootstrap planner node;
- adding the mutable golden contract to Agent Gateway CI; and
- updating the Hudi Architect skill, runbook, capability reference,
question flow, warnings, decision rules, and output templates.
The existing schema-1 append-only contract and its generated SQL remain
unchanged.
### Impact
This change affects only the Hudi Architect Flink design and
SQL-generation path.
For eligible mutable workloads, Hudi Architect can now emit canonical
streaming COW upsert SQL after deterministic static validation. Requests
outside the bounded path continue to return `INCOMPLETE`, `BLOCKED`, or
`REVIEW_REQUIRED` without emitting partial SQL.
The following mutable configurations remain fail closed:
- auto-generated record keys;
- missing, nullable, multiple, or unsupported ordering fields;
- retract streams or changelogs containing `UPDATE_BEFORE`;
- key-only or otherwise incomplete delete records;
- partition movement for an existing key;
- alternative index implementations;
- non-global FLINK_STATE;
- expiring index state;
- disabled index bootstrap;
- custom record mergers;
- MOR tables;
- existing-table takeover;
- external catalogs; and
- multiple writers.
The validation evidence now reports capability manifest schema 3 and the
supported design-contract schemas `[1, 2]`.
There are no changes to Hudi storage format, core Flink connector
behavior, or public Java APIs.
### Risk Level
Medium.
The change enables executable SQL generation for mutable workloads, so an
incorrect contract or generated option could affect record routing, replay
behavior, deletes, or conflict resolution.
The risk is mitigated by:
- keeping the executable surface intentionally narrow;
- rejecting every unsupported deviation before SQL is emitted;
- pinning behavior to the immutable Hudi 1.2.0 source revision;
- verifying all referenced source files by SHA-256;
- preserving the existing schema-1 SQL goldens;
- comparing schema-2 output with an exact SQL golden;
- validating the generated DDL and `INSERT INTO` with the released Hudi
1.2.0 Flink bundle and Flink 1.20.1 planner;
- verifying the Hudi UPSERT changelog contains `INSERT`, `UPDATE_AFTER`,
and `DELETE`, but not `UPDATE_BEFORE`;
- verifying that the generated plan contains both the Hudi stream writer
and index-bootstrap operator; and
- running the complete Agent Gateway test, lint, and type-check matrix on
Python 3.11 and 3.12.
### Documentation Update
Updated the in-repository Hudi Architect documentation, including:
- `SKILL.md`;
- `README.md`;
- `RUNBOOK.md`;
- the Flink capability baseline;
- the Flink question flow;
- decision overrides;
- warning definitions; and
- executable and non-executable output templates.
No Hudi website documentation is required because this PR changes the Hudi
Architect skill and does not add or modify a core Hudi connector configuration.
### Tests
- Python 3.11: `278 passed, 4 skipped`
- Python 3.12: `278 passed, 4 skipped`
- Pinned Hudi 1.2.0 / Flink 1.20.1 Java fixture: passed
- Ruff: passed
- Mypy: passed
- Hudi configuration-key validation: passed
- Pinned source-revision and SHA-256 validation: passed
- Hudi Architect skill validation: passed
- `git diff --check`: passed
Commands used:
```bash
python skills/hudi-architect/validate_config_keys.py
python skills/hudi-architect/validate_flink_capabilities.py \
--verify-source
python skills/hudi-architect/validate_flink_design.py \
--input tests/fixtures/hudi_architect/flink_pr2/stable_key.json \
--emit-sql
python skills/hudi-architect/validate_flink_design.py \
--input tests/fixtures/hudi_architect/flink_pr3/mutable_cow.json \
--emit-sql
python -m pytest tests -q
ruff check \
src \
tests \
skills/hudi-architect/redact_sensitive_values.py \
skills/hudi-architect/validate_flink_capabilities.py \
skills/hudi-architect/validate_flink_design.py
mypy \
src \
skills/hudi-architect/validate_flink_capabilities.py \
skills/hudi-architect/validate_flink_design.py
mvn -B \
-f hudi-agent-gateway/fixtures/hudi-architect-flink-1.20/pom.xml \
test
```
### Contributor's checklist
- [x] Read through contributor's guide
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
