Anish Mahto created SPARK-60094:
-----------------------------------
Summary: Gate Ignore-null functionality
Key: SPARK-60094
URL: https://issues.apache.org/jira/browse/SPARK-60094
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Until AutoCDC ignore-null is fully built E2E for both SCD1 and SCD2, we should
explicitly gate its usage.
In ignore-null's current partial implementation state, if it runs then we may
produce partial results that get persisted against existing AutoCDC target/aux
tables.
Both the current (partially complete) implementation and API support for
ignore-null is merged into Spark master, and is planned to go out as of Spark
4.4. Once the implementation is complete, this gating can be removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
