viirya opened a new issue, #5900: URL: https://github.com/apache/datafusion-comet/issues/5900
Normalizing multi-entry `map<string,int>` batches currently slices each row's keys and invokes Arrow sorting, allocating temporary indices and prefix tuples for every map. The singleton optimization in #5887 avoids this work for one-entry maps, but multi-entry maps still pay these costs. Reuse an Arrow-compatible prefix-tuple scratch buffer across rows for multi-entry string-key/int-value maps, while preserving the singleton fast path, exact duplicate-key permutations, sliced offsets, null buffers, schema metadata, and unsupported-key errors. Also fill rebased offsets in bulk for all-empty visible batches. Matched release microbenchmarks show about 3x faster normalization for 2–10-entry forward maps and 33–39% faster normalization for 2–50-entry maps in independent paired measurements. Validation covers normalization, hashing, combined execution, nulls, mixed cardinalities, Unicode keys, and fallback types. Related to #5818 and the matched benchmark work in #5822. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
