dwsmith1983 commented on PR #5565: URL: https://github.com/apache/datafusion-comet/pull/5565#issuecomment-5552530361
> This seems lot a lot of additional complexity for little performance gain? Fair question. the gain is on the encode side, 7 percent wall and 11 percent encode time at 10k partitions with zstd level 3 on the writer benchmark, and flat at large blocks. The decode side measured throughput neutral; its value is bounding retained memory. About half of the added lines are tests, and a good part of the surface came from review asks (the exec-side decode context, the retained-size cap, the spill boundary hook, RSS releasing per invocation). If the trade off looks wrong to you, the cleanest is to drop the decode side entirely and keep only writer-side context reuse. That removes the scan operator changes, the decode context type, and the thread-local, roughly half the diff, while keeping the measured win. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
