andygrove opened a new issue, #5837:
URL: https://github.com/apache/datafusion-comet/issues/5837
### Describe the bug
With native Iceberg writes enabled, compacting a partition of a table whose
spec has evolved rewrites the group into the wrong number of files and leaves
the dangling position delete file in place.
`TestRewriteDataFilesAction.testRemoveDangledPositionDeletesPartitionEvolution`
asserts `(addedDataFilesCount, rewrittenDataFilesCount,
removedDeleteFilesCount)` is `[1, 2, 1]`. It gets `[2, 2, 0]`: the two input
files were read, but they came out as two files rather than one, and the
dangled position delete was not removed.
```
org.opentest4j.AssertionFailedError: [Should rewrite 2 data files into 1 and
remove 1 dangled position delete file]
Expecting actual:
[2, 2, 0]
to contain exactly (and in same order):
[1, 2, 1]
```
The shape of the test: write 4 data files across two `c1` partitions, add a
position delete against one of them, evolve the spec by adding an identity
field on `c3`, write one more file, then `rewriteDataFiles().filter(c1 =
1).option(REWRITE_ALL, true).option(REMOVE_DANGLING_DELETES, true)`.
Only format version 2 fails. Versions 3 and 4, where the same test writes a
DV instead of a position delete file, both pass. That points at the
positional-delete path rather than at the writer's file rolling or at the
output partition spec, since the data files and the spec evolution are
identical across all versions.
Identical result on Iceberg 1.9.1, 1.10.0 and 1.11.0, so it is
deterministic. Iceberg 1.8.1 does not run this case.
### Steps to reproduce
Spark 4.1.3, Iceberg 1.11.0, with `spark.comet.iceberg.write.enabled=true`
and the Iceberg Spark SQL test setup from `dev/diffs/iceberg/1.11.0.diff`:
```
./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 -DflinkVersions=
-DkafkaVersions= \
:iceberg-spark:iceberg-spark-4.1_2.13:test \
--tests
'*TestRewriteDataFilesAction.testRemoveDangledPositionDeletesPartitionEvolution*'
\
-Pquick=true -x javadoc
```
### Expected behavior
Compaction produces the same counts through the native writer as through
iceberg-java: the file group is rewritten into a single data file and the
dangled position delete file is removed.
### Additional context
Found by turning the two Iceberg write flags on by default in #5677.
At the earlier baseline for that PR this case was grouped with #5690, the
target-file-size divergence, and the other `TestRewriteDataFilesAction` failure
in that group (`testBinPackCombineMediumFiles`) is green now that #5780 has
landed. This one is not, so it is a separate defect rather than residue of the
file-rolling work.
Two things worth checking first: how many tasks the rewrite job runs under
Comet (two output files for two rewritten files suggests the group was not
combined into one task), and what partition spec the added files carry.
Part of #5649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
