Hi Robert, I mean #define CHUNK_SIZE is still in the patch. >
Oops, removed now. > > > With just that change, you can set > > > next_buf_len = LZ4F_HEADER_SIZE_MAX + mysink->output_buffer_bound -- > > > but that's also more than you need. You can instead do next_buf_len = > > > Min(LZ4F_HEADER_SIZE_MAX, mysink->output_buffer_bound). Now, you're > > > probably thinking that won't work, because bbsink_lz4_begin_archive() > > > could fill up the buffer partway, and then the first call to > > > bbsink_lz4_archive_contents() could overrun it. But that problem can > > > be solved by reversing the order of operations in > > > bbsink_lz4_archive_contents(): before you call LZ4F_compressUpdate(), > > > test whether you need to empty the buffer first, and if so, do it. > > > > I am still not able to get - how can we survive with a mere > > size of Min(LZ4F_HEADER_SIZE_MAX, mysink->output_buffer_bound). > > LZ4F_HEADER_SIZE_MAX is defined as 19 in lz4 library. With this > > proposal, it is almost guaranteed that the next buffer length will > > be always set to 19, which will result in failure of a call to > > LZ4F_compressUpdate() with the error LZ4F_ERROR_dstMaxSize_tooSmall, > > even if we had called bbsink_archive_contents() before. > > Sorry, should have been Max(), not Min(). > Ahh ok. I looked into the code of LZ4F_compressBound() and the header size is already included in the calculations, so when we compare LZ4F_HEADER_SIZE_MAX and mysink->output_buffer_bound, the latter will be always greater, and hence sufficient length for the output buffer. I made this change in the attached patch, and also cleared the buffer by calling bbsink_archive_contents() before calling LZ4_compressUpdate(). Similarly cleared the buffer before calling LZ4F_compressEnd(). > You mean the way gzip allows us to use our own alloc and free functions > > by means of providing the function pointers for them. Unfortunately, > > no, LZ4 does not have that kind of provision. Maybe that makes a > > good proposal for LZ4 library ;-). > > I cannot think of another solution to it right away. > > OK. Will give it some thought. I have started a thread[1] on LZ4 community for this, but so far no reply on that. Regards, Jeevan Ladhe [1] https://groups.google.com/g/lz4c/c/WnJkKwBWlcM/m/zszrla2mBQAJ?utm_medium=email&utm_source=footer
lz4_compress_v6.patch
Description: Binary data
