[
https://issues.apache.org/jira/browse/SPARK-60097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Mahto updated SPARK-60097:
--------------------------------
Description:
Concretely define the contract for the _contents_ of the version map; the
format the keys are written in, which leaves get an entry, and what the stored
sequence represents when null or not null.
Then implement the logic to construct a version map column expression as per
the contract, given an upsert-representing row. Delete-representing rows need
no equivalent operation, as they always hold null version maps
was:
Concretely define the contract for the _contents_ of the version map; the
format must the keys be presented in, and what the stored sequence represents
when null or not null.
Then implement the logic to construct a version map column expression as per
the contract, given an upsert-representing row. Delete-representing rows do not
require any equivalent operation, as they will always hold null version maps.
> SCD1 Ignore-null support; define version map construction
> ---------------------------------------------------------
>
> Key: SPARK-60097
> URL: https://issues.apache.org/jira/browse/SPARK-60097
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Priority: Major
>
> Concretely define the contract for the _contents_ of the version map; the
> format the keys are written in, which leaves get an entry, and what the
> stored sequence represents when null or not null.
> Then implement the logic to construct a version map column expression as per
> the contract, given an upsert-representing row. Delete-representing rows need
> no equivalent operation, as they always hold null version maps
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
