peterxcli commented on PR #5809: URL: https://github.com/apache/datafusion-comet/pull/5809#issuecomment-5676938442
> Replying to https://github.com/apache/datafusion-comet/pull/5809#issuecomment-5656824227 Thanks for the comparison, and for #5909. You were right that the streaming loop is the better base, so this revision (a5290ca76) is built on it rather than stacked beside it: each block is walked message by message, the schema message is served from the cache on a byte match without being parsed, and bodies are read into exactly sized buffers. There is no whole-block materialization left to gate behind a threshold, so the "streaming below, in-place above" split turned out not to be needed. What remains of the earlier design: the cache is per thread and holds four schemas, so a reduce task that interleaves blocks from two shuffles does not thrash and no decoder handle has to be threaded through `ShuffleScanExec` or the JNI entry points; uncompressed blocks are walked in place, so their bodies are copied once without a zero fill; and the metadata scratch is reused across blocks. Re-measured against `main` on the idle host, base/new alternated, `None` and `Lz4Frame`, plain and dictionary-encoded strings. Table in the description. Against `main`, 5 col x 64 row is now -53% uncompressed and -30% under LZ4, and 50 col x 64 row -55% and -40%, in line with the -54% you measured for #5909's cached decoder on the uncompressed shape. The large-block gain from the earlier revision survives on uncompressed blocks (-40% at 50 col x 8192 rows, even on a cache miss, from copying the body without a zero fill) and is small under LZ4 (-2% to -5%) where decompression dominates. The dictionary case you flagged is covered by `dictionaries_are_scoped_to_their_block_under_a_cached_schema`: two blocks that share a schema but carry different dictionaries decode to their own values with the schema served from the cache, on every codec, validated and not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
