Anish Mahto created SPARK-57356:
-----------------------------------
Summary: Implement SCD2 Batch Processor; Cleanup Delete Encoding
Rows Post-Reconciliation
Key: SPARK-57356
URL: https://issues.apache.org/jira/browse/SPARK-57356
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Anish Mahto
{*}Preamble{*}:
The SCD type 2 flow is a foreachBatch streaming query on an input
change-data-feed, and is responsible for reconciling the incoming change data
onto some target table that follows SCD2 replication semantics.
SCD2 flows also maintain an "auxiliary" table to keep track of early-arriving
out-of-order received events state. Each microbatch will need to reconcile
against this auxiliary table as well, and update the auxiliary table's state
appropriately for future microbatches.
*Cleanup Delete Encoding Rows Post-Reconciliation*
Reconciliation itself (re)assigns the start/end-ats of upsert representing
columns in the microbatch-affected row set, using the next neighbor event
(could be either another upsert that ends this upsert, or a delete event that
ends this upsert).
Once a delete event matches with an preceding row and reconciliation updates
that row's end-at accordingly, that upsert row (now closed) fully encodes the
delete event along with its upsert event. This means the delete event does not
need to have its own row, and the tombstone/decomposition tail can be dropped.
If a delete event does not match against a preceding upsert however, then it is
necessary to bookkeep as a tombstone in the aux table.
It is also possible that post-reconciliation, a decomposition tail survives as
a non-redundant delete encoding event. This decomposition tail should be
promoted to a tombstone and persisted in the aux to reconcile against future
microbatches.
As a trivial example image an existing closed upsert valid for start/end at
[T=10, T=20) gets bisected by a delete event at T=15. Then the closed upsert
gets decomposed into head and tail at [T=10, null) and (null, T=20],
reconciliation creates a new closed upsert at [T=10, T=15), and the
decomposition tail (null, T=20] should be promoted to a tombstone at instant
T=20.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
