andygrove opened a new issue, #39: URL: https://github.com/apache/datafusion-iceberg/issues/39
### Describe the bug With fanout disabled, `TaskWriter` splits each batch with iceberg-rust's `RecordBatchPartitionSplitter::split` and hands the pieces to `ClusteredWriter` ([task_writer.rs#L129-L135](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/task_writer.rs#L129-L135), [task_writer.rs#L217-L237](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/task_writer.rs#L217-L237)). The splitter groups rows in a `HashMap`, so the partitions within a batch come out in arbitrary order. `ClusteredWriter` closes the current partition when it sees a different one, and rejects any partition it has already closed. Input that is correctly sorted by partition therefore fails whenever a partition spans a batch boundary. For example, batch N holds [A, B] but is emitted as B then A, and batch N+1 starts with B. ### To Reproduce Execute the plan returned by `IcebergTableProvider::insert_into` directly, so the `SortExec` is still present (#38 explains why the optimized plan loses it). Use a table partitioned by `identity(cat)` with fanout disabled, 200 rows with `cat = id % 10`, `batch_size = 8` and `target_partitions = 1`. 9 of 10 runs fail with: ``` The input is not sorted! Cannot write to partition that was previously closed ``` ### Expected behavior Input sorted by partition is written successfully. ### Additional context Fixing #38 alone won't make fanout-disabled writes work. The splitter lives in iceberg-rust (`crates/iceberg/src/arrow/record_batch_partition_splitter.rs`). Either it emits partitions in first-occurrence order, or `TaskWriter` orders each batch's pieces by row position before passing them to `ClusteredWriter`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
