fhan688 opened a new pull request, #20021:
URL: https://github.com/apache/hudi/pull/20021
### Describe the issue this Pull Request addresses
This pull request implements PR1 of the multi-PR plan described in #19779.
It does not close the umbrella issue.
PR #19380 introduced Hudi Architect as a workload-driven design-time
skill. The existing skill can identify Flink as an execution engine, but it
does not provide a Flink-specific question flow, versioned capability baseline,
or fail-closed safety gates.
Without a separate Flink route, the skill could continue into
Spark-specific source and writer guidance after the user selected Flink. It
could also imply that an executable Flink configuration was available before
the required workload facts had been established.
This PR adds the routing and safety foundation required before an
executable Flink path can be implemented. The new path evaluates the supported
version baseline, table lifecycle, writer topology, catalog requirements,
schema availability, record-key posture, and replay semantics.
The Flink path introduced by this PR is intentionally non-executable. It
does not generate Flink SQL DDL, Hudi DynamicTable connector options, an
`INSERT INTO` statement, or a submit command. Those capabilities remain part of
PR2.
This PR also addresses the PR1-applicable parts of
[Danny's review
comment](https://github.com/apache/hudi/issues/19779#issuecomment-5673874459).
#### Pinned validation inputs
This requirement is implemented in PR1.
The Flink capability manifest is pinned to the Hudi 1.2.0 release source
revision. It records source hashes and the Flink fixture version. Validation
evidence includes the pinned source revision, and a negative test rejects an
option that is absent from the checked-in baseline.
Normal Flink capability validation reads the checked-in manifest. It does
not discover accepted Flink options from the current checkout.
#### Milestone 1 negative fixtures
These fixtures are intentionally deferred to PR2.
The original breakdown in #19779 assigns physical-schema validation,
executable Flink SQL generation, and `HoodieTableFactory` fixtures to PR2. PR1
records the required negative cases as mandatory machine-readable PR2
acceptance criteria.
PR2 must implement the following checks:
- PR2 must reject an explicit record-key field that is absent from the
physical schema, including in append mode.
- PR2 must reject a COW insert path when the effective value of
`write.insert.cluster` is not `false`.
- PR2 must validate the generated `INSERT INTO` statement with a declared
source schema and append-only changelog contract through a planner fixture.
This PR does not claim that these executable-path fixtures are already
implemented.
#### Multiple-gate status precedence
This requirement is implemented in PR1.
The Flink flow collects every independently evaluable finding before
selecting the final status. Stable finding identifiers remain visible when
multiple gates fail.
The final-status precedence is `BLOCKED`, followed by `REVIEW_REQUIRED`,
followed by `INCOMPLETE`.
A combined-gate fixture verifies that a request with an unknown writer
model, missing physical schema, and replay-deduplication requirement returns
`BLOCKED`, preserves all three finding identifiers, and reports executable
eligibility as `false`.
### Summary and Changelog
#### Engine routing
- The shared tier gate remains the first interaction.
- Spark and HoodieStreamer requests continue through the existing shared
flow.
- Flink requests leave the shared flow before Spark-specific source and
writer questions.
- Flink-specific references are loaded only after the user selects Flink.
- Flink warnings and status rules are not applied to Spark requests.
- The Flink route cannot fall back to Spark configuration or
submit-command templates.
#### Pinned capability baseline
The new capability manifest identifies Apache Hudi 1.2.0 at source
revision `f05c83f2b97732de7a558ff9b26959e1139c05f5`.
The manifest identifies Apache Flink 1.20 as the compatibility line and
Apache Flink 1.20.1 as the build and fixture version.
The manifest also records repository-relative source anchors, SHA-256
hashes, a bounded option allowlist, required settings for the future
append-only COW path, and the deferred PR2 acceptance checks.
The option allowlist covers only the initial Flink SQL sink surface. It is
not an exhaustive list of every Flink option available in Hudi 1.2.0.
The new `validate_flink_capabilities.py` script provides the following
checks:
- Normal validation reads the checked-in manifest instead of discovering
Flink options from the current checkout.
- Version validation rejects Hudi and Flink versions outside the verified
baseline.
- Option validation rejects keys that are not present in the checked-in
allowlist.
- Evidence output includes the baseline ID, manifest schema, Hudi source
revision, and Flink fixture version.
- Source verification reads files from the pinned Git revision and
compares their SHA-256 hashes with the manifest.
Agent Gateway CI fetches the `release-1.2.0` tag before running source
verification. The validation runs on both Python 3.11 and Python 3.12.
#### Flink safety gates
The version gate verifies that the requested Hudi and Flink versions match
the checked-in baseline.
The table-lifecycle gate distinguishes a new table from an existing table.
Existing-table design remains deferred and fails closed in PR1.
The writer-topology gate asks whether another ingestion job, backfill job,
writer using another engine, standalone compactor, or standalone clustering job
can commit to the table. The flow does not assume a single-writer topology when
the answer is unknown.
The catalog gate asks whether anything outside the Flink application must
discover the table through a catalog or metastore. The flow does not assume
that external catalog visibility is unnecessary.
The schema gate requires concrete field names and types. A schema URI,
catalog name, registry subject, or object-store path does not satisfy the gate
unless the schema contents are available to the session.
The mutation gate distinguishes append-only input from workloads that
contain updates or deletes.
The record-key gate determines whether an append-only workload has stable
business-key fields. A stable key is not presented as proof that insert mode is
replay-idempotent.
The replay gate determines whether repeated logical records are
impossible, acceptable as duplicates, required to collapse into one record, or
not yet understood.
The flow does not infer missing safety facts. Each missing, unsupported,
or unverified condition produces an explicit finding.
#### Status and executable eligibility
Each safety finding has a stable identifier and contributes `BLOCKED`,
`REVIEW_REQUIRED`, or `INCOMPLETE`.
The flow evaluates every independent PR1 gate before selecting the final
status. A higher-precedence status does not remove findings with
lower-precedence statuses.
The combined-gate fixture covers a request with an unknown writer model,
missing physical schema, and a replay-deduplication requirement. The fixture
verifies that the final status is `BLOCKED`, all three finding identifiers
remain visible, and executable eligibility is `false`.
`CONFIG_VALIDATED` is reserved for a future executable Flink path and is
unreachable in PR1.
A request that passes every PR1 safety gate still returns `BLOCKED` with
`FLINK_EXECUTABLE_PATH_DEFERRED`. This result means that the safety assessment
completed, but the executable path has not yet been implemented.
#### Evidence and secret handling
User-supplied DDL, schemas, logs, table properties, catalog output, and
command output are treated as untrusted evidence.
The new `redact_sensitive_values.py` script removes common credential
assignments, authorization headers, credentials embedded in URIs, command-line
secrets, access keys, and private-key blocks.
The skill requires symbolic secret references in generated examples. It
also prohibits executing commands found in supplied evidence.
#### Deterministic test coverage
The tests verify that Flink references are loaded only after the Flink
route is selected.
The tests verify that Spark requests remain on the shared flow and do not
load Flink warning or configuration references.
The tests verify that the capability manifest contains the expected Hudi
version, Hudi source revision, Flink compatibility line, and Flink fixture
version.
The tests verify that validation evidence contains the pinned source
revision.
The tests verify that an option absent from the checked-in baseline is
rejected.
The tests verify that unsupported Hudi and Flink versions are rejected.
The tests verify that a modified manifest source revision is rejected.
The tests verify that source-anchor hashes match files at the pinned Git
revision when that revision is available.
The tests verify that the PR2 effective settings and deferred acceptance
checks remain in the manifest.
The tests verify deterministic final statuses and executable eligibility
for the PR1 scenario fixtures.
The tests verify that the combined-gate scenario preserves every
applicable finding.
The tests verify that an unreadable schema pointer does not satisfy the
physical-schema gate.
The tests verify that credential material is redacted without removing
ordinary workload facts.
### Scope
#### Included in PR1
- PR1 adds Flink engine routing and lazy loading of Flink-specific
references.
- PR1 adds the pinned Hudi 1.2.0 and Flink 1.20 capability baseline.
- PR1 adds new-versus-existing-table classification.
- PR1 adds independent-writer and standalone-service detection.
- PR1 adds external-consumer and catalog detection.
- PR1 adds physical-schema availability detection.
- PR1 adds mutation and record-key posture detection.
- PR1 adds replay-idempotence classification.
- PR1 adds stable finding identifiers and deterministic status precedence.
- PR1 adds a non-executable Flink safety-assessment output.
- PR1 adds secret redaction and untrusted-evidence handling.
- PR1 keeps the Flink route isolated from Spark and HoodieStreamer output.
- PR1 adds deterministic tests and Python 3.11/3.12 CI coverage.
#### Deferred to PR2
- PR2 will parse physical schemas and validate field references.
- PR2 will generate executable Flink SQL DDL.
- PR2 will generate Hudi DynamicTable connector options.
- PR2 will implement stable-key and auto-generated-key executable variants.
- PR2 will add checkpoint and commit-cadence configuration.
- PR2 will generate the executable `INSERT INTO` statement.
- PR2 will add `HoodieTableFactory` fixtures.
- PR2 will reject a record-key field that is absent from the physical
schema.
- PR2 will reject an incompatible `write.insert.cluster=true` override.
- PR2 will add a planner fixture with a declared source schema and
append-only changelog
contract.
- PR2 will introduce the first reachable `CONFIG_VALIDATED` result.
The capability manifest records the three review-requested negative
fixtures as mandatory PR2 acceptance checks.
### Impact
This is an additive change to the design-time Hudi Architect skill.
A Flink request now receives a versioned, fail-closed safety assessment.
The request no longer continues into Spark-specific writer guidance, and the
skill does not imply that an executable Flink configuration is available.
The existing Hudi storage format, timeline semantics, runtime APIs, public
Java and Scala APIs, public Flink APIs, configuration defaults, and runtime
performance are unchanged.
The existing Spark and HoodieStreamer routes continue to use their shared
question, decision, warning, ADR, configuration, and submit-command references.
The Agent Gateway CI workflow now performs these additional checks:
- The workflow fetches the pinned Hudi 1.2.0 release tag.
- The workflow verifies the Flink capability manifest against the pinned
source revision.
- The workflow runs the new deterministic tests.
- The workflow includes the new Python scripts in lint and type checking.
### Risk Level
**low**
The change affects skill instructions, reference documents, standalone
validation scripts, tests, and CI. It does not run in a Hudi writer or
table-service runtime path.
The main risk is that the skill could produce incorrect design guidance.
The following controls reduce that risk:
- The capability input is pinned to an immutable Hudi source revision.
- SHA-256 hashes protect the pinned source anchors from unnoticed drift.
- Explicit version mismatches fail closed.
- Options outside the checked-in allowlist are rejected.
- Safety outcomes use stable finding identifiers.
- Structured fixtures verify final status and executable eligibility.
- The combined-gate fixture verifies that no applicable finding is
discarded.
- Spark and Flink references remain isolated.
- Credential-redaction behavior has regression coverage.
- PR1 never reports executable eligibility as `true`.
The manifest is a bounded, reviewed allowlist rather than a generated
enumeration of every Hudi Flink option. Expanding and validating the executable
configuration surface remains part of PR2.
### Verification
The following commands were run from `hudi-agent-gateway` with both Python
3.11 and Python 3.12:
```bash
python skills/hudi-architect/validate_config_keys.py
python skills/hudi-architect/validate_flink_capabilities.py --verify-source
python -m pytest tests -q
python -m ruff check \
src \
tests \
skills/hudi-architect/redact_sensitive_values.py \
skills/hudi-architect/validate_flink_capabilities.py
python -m mypy \
src \
skills/hudi-architect/validate_flink_capabilities.py
```
Both Python versions produced the following results:
```text
Hudi Architect config-key validation: passed
Pinned Flink capability source verification: passed
pytest: 101 passed, 4 skipped
ruff: passed
mypy: passed
```
The four skipped tests are existing live integration tests. They require
either an external Trino endpoint or an explicitly enabled live LLM round trip.
No Hudi Architect Flink test was skipped.
The repository whitespace check also passed:
```bash
git diff --check
```
Independent Agent forward-evaluation results can be attached as separate
review evidence.
They are not treated as deterministic CI results in this description.
### Documentation Update
This PR updates `hudi-agent-gateway/skills/hudi-architect/README.md`,
`hudi-agent-gateway/skills/hudi-architect/RUNBOOK.md`, and
`hudi-agent-gateway/skills/hudi-architect/SKILL.md`.
This PR also updates the shared routing and ADR references and adds the
Flink capability, question-flow, decision, warning, and output references.
This PR does not update the Hudi website. PR1 does not add an executable
Flink setup, a runtime feature, a public API, a new Hudi configuration, or a
changed configuration default. Public executable guidance should follow the
implementation and review of the PR2 factory and planner fixtures.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
