[
https://issues.apache.org/jira/browse/SPARK-58118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098366#comment-18098366
]
Naveen Kumar Puppala edited comment on SPARK-58118 at 7/23/26 5:20 AM:
-----------------------------------------------------------------------
I'd like to take this one.
My plan, following the two parts in the description:
1. Relax validateUserSpecifiedSchemas so the declared schema can differ from
the inferred one only by the reserved metadata column. Any other mismatch still
fails. Pipelines that already declare the reserved column today keep working.
2. In DatasetManager.materializeTable, when the table has a user schema and is
fed by an AUTO CDC flow, append the reserved column(s) to the schema the table
actually gets created with. Otherwise the merge fails at runtime as you
described.
Two questions:
- Do we want to keep allowing the reserved column in user schemas, or deprecate
that with a warning?
- I'm planning to cover SCD1 only for now, since AutoCdcMergeFlow.schema still
throws AUTOCDC_SCD2_NOT_SUPPORTED. SCD2's reserved columns can get the same
treatment when that path is wired up. Sound right?
For tests I'll extend UserSpecifiedSchemaValidationSuite: omitting the metadata
column accepted, wrong data columns still rejected, declaring the metadata
column still works, plus a check that the created table actually ends up with
the column.
was (Author: JIRAUSER312582):
I'd like to take this one.
My plan, following the two parts in the description:
1. Relax \{{validateUserSpecifiedSchemas}} so the declared schema can differ
from the inferred one only by the reserved metadata column. Any other mismatch
still fails. Pipelines that already declare the reserved column today keep
working.
2. In \{{DatasetManager.materializeTable}}, when the table has a user schema
and is fed by an AUTO CDC flow, append the reserved column(s) to the schema the
table actually gets created with. Otherwise the merge fails at runtime as you
described.
Two questions:
- Do we want to keep allowing the reserved column in user schemas, or deprecate
that with a warning?
- I'm planning to cover SCD1 only for now, since \{{AutoCdcMergeFlow.schema}}
still throws \{{AUTOCDC_SCD2_NOT_SUPPORTED}}. SCD2's reserved columns can get
the same treatment when that path is wired up. Sound right?
For tests I'll extend \{{UserSpecifiedSchemaValidationSuite}}: omitting the
metadata column accepted, wrong data columns still rejected, declaring the
metadata column still works, plus a check that the created table actually ends
up with the column.
> Allow a user-specified AUTO CDC streaming-table schema to omit the reserved
> metadata column
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-58118
> URL: https://issues.apache.org/jira/browse/SPARK-58118
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Andreas Neumann
> Priority: Major
>
> **Description:**
> An AUTO CDC (SCD Type 1) flow appends a reserved metadata column named
> `__spark_autocdc_metadata` (a struct of `deleteSequence`/`upsertSequence`) to
> its output schema(`AutoCdcMergeFlow.schema`). Because
> `validateUserSpecifiedSchemas` requires the user-declared schema to *exactly
> equal* the inferred schema, a user who declares a schema on an AUTO CDC
> streaming table is currently forced to also declare this internal reserved
> column, e.g.:
> ```
> CREATE STREAMING TABLE t (
> id INT NOT NULL, name STRING, version BIGINT NOT NULL,
> __spark_autocdc_metadata
> STRUCT<deleteSequence:BIGINT,upsertSequence:BIGINT> NOT NULL
> ) FLOW AUTO CDC ...
> ```
> That column is an engine implementation detail; users should not need to know
> its name or shape. Declaring only the logical data columns should be
> accepted, with the engine owning the metadata column.
> Note: simply relaxing the validation is insufficient.
> `DatasetManager.materializeTable` uses `table.specifiedSchema` verbatim as
> the created-table schema, so if the user omits the metadata column the target
> table is created without it and the SCD1 MERGE fails at runtime with
> `UNRESOLVED_COLUMN ... __spark_autocdc_metadata` (verified empirically). The
> fix must both:
> 1. Relax `validateUserSpecifiedSchemas` to permit the declared schema to
> differ from the inferred schema *only* by the reserved AUTO CDC metadata
> column(s), and
> 2. Backfill the reserved metadata column into the effective output schema at
> materialization when a user schema is specified for an AUTO CDC target, so
> the created table actually contains it.
> This is a deliberate semantics decision ("the user schema describes the
> logical table; the engine owns reserved columns") and should be designed with
> the SDP schema-model owners. Subsumes the ergonomic wart where the combined
> form forces declaring the reserved column.
> Note: this came up during the review of SPARK-57402
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
