peterxcli opened a new issue, #5790: URL: https://github.com/apache/datafusion-comet/issues/5790
### What is the problem the feature request solves? The native shuffle writer already knows every partition offset by the time it finishes a map task, but it hands them to the JVM through a temporary file rather than returning them directly. `LocalPartitionWriter::finish_all` creates a temp index file and writes `num_output_partitions + 1` little-endian i64 offsets into it. Back in the JVM, `CometNativeShuffleWriter` reads the whole file with `Files.readAllBytes`, converts the offsets to partition lengths, deletes the file, and passes the lengths to `IndexShuffleBlockResolver.writeMetadataFileAndCommit`, which writes Spark's real index file. The temp file exists only to move an array of longs across the JNI boundary, and it costs every map task a file create, write, read, and unlink on top of the index file Spark writes anyway. The parse is allocation-heavy too: `grouped(OFFSET_LENGTH)` allocates an intermediate array per partition and the `map` allocates a `ByteBuffer` per partition, so `2 * numPartitions` short-lived objects per map task. That part is already recorded as item 5 of #5198, but it goes away entirely if the file does. ### Describe the potential solution Return the offsets across the existing JNI boundary instead, as a `long[]` or a direct buffer filled by native code, and drop the temp index file. The JVM side then walks a primitive array once to turn offsets into lengths and commits as it does today. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
