hubgeter opened a new pull request, #418: URL: https://github.com/apache/doris-thirdparty/pull/418
`BlockDecompressionStream` allocates and zeroes a block-sized `inputDataBuffer` for every stream as soon as it is constructed. `NextDecompress` only needs this buffer to stitch together a compressed chunk that spans input buffers, and Doris normally reads ORC data in 8 MiB ranges, so the buffer is usually never used. Every ZLIB, Snappy, LZ4 and ZSTD stream therefore holds a second block next to its output buffer. Since apache/doris-thirdparty#417, AArch64 ZLIB streams use this class too. With 256 KiB blocks, a wide scan over 100 nullable string columns (PRESENT, LENGTH and DATA streams) holds about 75 MiB per reader before any data is read. This PR starts the buffer empty. `NextDecompress` already grows it to `remainingLength` when a chunk has to be stitched, so contiguous chunks allocate nothing extra. Tests: `Compression.block_decompression_input_buffer_is_lazy` counts pool allocations for ZLIB, ZSTD, LZ4 and Snappy. - When the whole stream is in one input buffer, peak usage is exactly the output block. - When the chunk is split across 7-byte input buffers, the data still round-trips and the scratch buffer is only as large as the chunk. Both checks fail on the current code. The existing compression tests and the Doris ORC/Parquet reader unit tests pass. Requested in the review of apache/doris#68593. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
