AHeise opened a new pull request, #29034:
URL: https://github.com/apache/flink/pull/29034
## What is the purpose of the change
`CREATE OR ALTER MATERIALIZED TABLE` derives its schema delta via
`MaterializedTableUtils.validateAndExtractColumnChanges`, which appends new
columns but never diffs column position. Adding a column anywhere but the end
of the query projection therefore stores a schema whose column order disagrees
with the query. Nothing rejects it at DDL time (`validateChanges` passes); the
mismatch only surfaces later, when the refresh runs a positional `INSERT INTO
<table> <expandedQuery>` and the wrong query column binds to the wrong sink
column.
`ALTER MATERIALIZED TABLE ... AS` already diffs position via
`buildSchemaTableChanges`, so the two query-evolution paths disagree. This PR
makes `CREATE OR ALTER` honor query column position the same way.
Example — `mt = (a, b, c, d)`:
```sql
CREATE OR ALTER MATERIALIZED TABLE mt AS SELECT a, b, c, 42 AS mid, d FROM
t1;
```
Expected stored schema: `(a, b, c, mid, d)` (the query projection order, as
`ALTER ... AS` produces).
Before this change: `(a, b, c, d, mid)` — `mid` appended at the end.
## Brief change log
- `validateAndExtractColumnChanges` emits `modifyColumnPosition` for
name-matched columns whose index changed, mirroring `buildSchemaTableChanges`.
- Old columns are positioned by their rank among the columns that survive
into the new schema, so retained non-persisted columns (kept by `CREATE OR
ALTER` but absent from the query projection) do not skew the position
comparison.
## Verifying this change
This change added tests and can be verified as follows:
- `MaterializedTableUtilsTest#testCreateOrAlter` — a physical
mid-projection insert and a case with a retained virtual-metadata column,
asserting the position-aware `TableChange` list.
- `SqlNodeToOperationSqlCreateOrAlterMaterializedTableConverterTest` —
end-to-end over real `CREATE OR ALTER` SQL, exercising schema resolution, the
position diff, and `validateChanges`: inserting before the last column stores
the query order and is accepted; inserting before multiple trailing columns is
rejected up front with a clear error instead of miscompiling at refresh time.
- `AlterMaterializedTableAsQueryOperationValidationTest` — guard boundary:
a reposition anchored after a newly-added column is accepted, one anchored
after an existing column is rejected.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
