yihua opened a new issue, #20260:
URL: https://github.com/apache/hudi/issues/20260
### Bug Description
**What happened:**
On a MOR table (table versions 6, 8 and 10) with
`hoodie.allow.operation.metadata.field=true`, upserts and deletes through the
Spark datasource, Spark SQL or the Java client lose every record routed to a
log file, and the write still reports success. These writers do not set a
`HoodieOperation` on their records, but
`HoodieAppendHandle.populateMetadataFields` calls
`hoodieRecord.getOperation().getName()` whenever the flag is on, so every
record fails with a `NullPointerException`. With `hoodie.write.ignore.failed`
at its default `true`, the failure is only recorded on the handle's write
status, and `close()` drops that status when no log block was appended. The
deltacommit completes without the records, and the next compaction finds no log
files. Flink is not affected because it always sets the operation.
**What you expected:**
The upsert lands in the log file and is visible in snapshot reads, or the
write fails.
**Steps to reproduce:**
1. In spark-shell, write two rows to a new MOR table with the operation
field enabled:
```scala
val opts = Map(
"hoodie.table.name" -> "t",
"hoodie.datasource.write.table.type" -> "MERGE_ON_READ",
"hoodie.datasource.write.operation" -> "upsert",
"hoodie.datasource.write.recordkey.field" -> "id",
"hoodie.datasource.write.partitionpath.field" -> "part",
"hoodie.table.ordering.fields" -> "ts",
"hoodie.allow.operation.metadata.field" -> "true")
Seq(("1", "a", 1L, "p1"), ("2", "b", 1L, "p1")).toDF("id", "name", "ts",
"part")
.write.format("hudi").options(opts).mode("overwrite").save(path)
```
2. Upsert one of them: `Seq(("1", "a2", 2L, "p1")).toDF("id", "name", "ts",
"part").write.format("hudi").options(opts).mode("append").save(path)`. The
write succeeds.
3. Read it back: `spark.read.format("hudi").load(path).select("id",
"name").show()` returns `1:a, 2:b` instead of `1:a2, 2:b`. With
`hoodie.allow.operation.metadata.field=false` it returns `1:a2, 2:b`.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT); the null dereference exists since
0.9.0
**Query engine:** Spark 3.5 (also the Java client)
**Relevant configs:** `hoodie.allow.operation.metadata.field=true`,
`hoodie.write.ignore.failed=true` (default), MOR table, table versions 6, 8 and
10
### Logs and Stack Trace
```
ERROR HoodieAppendHandle: Error writing record ...
java.lang.NullPointerException
at
org.apache.hudi.io.HoodieAppendHandle.populateMetadataFields(HoodieAppendHandle.java:453)
```
**Parent issue:** #20257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
