hudi-agent commented on code in PR #18914:
URL: https://github.com/apache/hudi/pull/18914#discussion_r3669747852
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ErrorTableAwareChainedTransformer.java:
##########
@@ -54,9 +59,34 @@ public Dataset<Row> apply(JavaSparkContext jsc, SparkSession
sparkSession, Datas
dataset =
ErrorTableUtils.addNullValueErrorTableCorruptRecordColumn(dataset);
for (TransformerInfo transformerInfo : transformers) {
Transformer transformer = transformerInfo.getTransformer();
+
+ // Stash _corrupt_record values before the transformer can drop them
+ Dataset<Row> corruptRecordStash = null;
+ if (ErrorTableUtils.isErrorTableCorruptRecordColumnPresent(dataset)) {
+ corruptRecordStash = dataset.select(new
Column(ERROR_TABLE_CURRUPT_RECORD_COL_NAME));
+ corruptRecordStash.cache();
+ // Force materialization so the stash is computed and stored before
the transformer
+ // runs. Without this, both stash and transformed dataset recompute
the shared
+ // upstream lineage independently at zip time — if that lineage has
non-deterministic
+ // row ordering (e.g. shuffle/repartition), the zip silently misaligns
values.
+ corruptRecordStash.count();
Review Comment:
🤖 I think `corruptRecordStash.count()` only pins the *stash's* row order —
but the stash is `dataset.select(col)`, so caching it doesn't cache `dataset`
itself. The transformer still reads the uncached `dataset` and recomputes its
full lineage independently, so `transformed` can come out in a different
intra-partition order than the materialized stash. If any upstream step is
order-non-deterministic (a prior transformer's shuffle/repartition), the zip
still misaligns even though the comment says this prevents it. Would
caching+materializing the input `dataset` itself (before the select) so both
the stash and the transformer read identical rows be more robust — or better,
restoring via a monotonic join key? @nsivabalan
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
