airborne12 commented on code in PR #69770:
URL: https://github.com/apache/doris/pull/69770#discussion_r4400345246
##########
be/src/storage/index/inverted/snii/encoding/zstd_codec.cpp:
##########
@@ -49,4 +49,36 @@ Status zstd_decompress(Slice input, size_t
expected_uncomp_len, std::vector<uint
return Status::OK();
}
+ZstdDecompressor::~ZstdDecompressor() {
+ ZSTD_freeDCtx(static_cast<ZSTD_DCtx*>(_context));
+}
+
+Status ZstdDecompressor::decompress(Slice input, size_t expected_uncomp_len,
Slice* out) {
+ if (_context == nullptr) {
+ _context = ZSTD_createDCtx();
+ if (_context == nullptr) {
+ return Status::Error<ErrorCode::MEM_ALLOC_FAILED, false>(
+ "zstd decompress: cannot create a context");
+ }
+ }
+ if (expected_uncomp_len > _capacity) {
+ // Every byte is written by the decompression before it is read, so
the buffer is not
+ // zero-filled.
+ _buffer.reset(new uint8_t[expected_uncomp_len]);
Review Comment:
Fixed in 6fd35b186f. Reproduced first: with a task MemTracker attached,
decoding an 8 MiB frame through one `ZstdDecompressor` left the tracker's
consumption unchanged while the decompressor held the buffer
(`SniiZstdDecompressor.ChargesItsBufferToTheTaskMemTracker`, 0 against 8388608
bytes).
The retained buffer is now a `DorisUniqueBufferPtr<uint8_t>`
(`Allocator<false>`, no zero-fill), so it is consumed from the decoding task's
tracker for as long as the decompressor holds it and released when the source
is destroyed; the old buffer is released before a larger one is allocated. The
test now passes, and so do the 1739 SNII, inverted-index and index-file tests
on this branch.
The zstd context itself (about 96 KB, from `ZSTD_createDCtx`) stays with
zstd's allocator: it is fixed-size, one per source, and the same size as the
context every window used to create and free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
