On Fri, Oct 15, 2021 at 7:54 AM Jeevan Ladhe <jeevan.la...@enterprisedb.com> wrote: > > The loop is gone, but CHUNK_SIZE itself seems to have evaded the > > executioner. > > I am sorry, but I did not really get it. Or it is what you have pointed > in the following paragraphs?
I mean #define CHUNK_SIZE is still in the patch. > > With just that change, you can set > > next_buf_len = LZ4F_HEADER_SIZE_MAX + mysink->output_buffer_bound -- > > but that's also more than you need. You can instead do next_buf_len = > > Min(LZ4F_HEADER_SIZE_MAX, mysink->output_buffer_bound). Now, you're > > probably thinking that won't work, because bbsink_lz4_begin_archive() > > could fill up the buffer partway, and then the first call to > > bbsink_lz4_archive_contents() could overrun it. But that problem can > > be solved by reversing the order of operations in > > bbsink_lz4_archive_contents(): before you call LZ4F_compressUpdate(), > > test whether you need to empty the buffer first, and if so, do it. > > I am still not able to get - how can we survive with a mere > size of Min(LZ4F_HEADER_SIZE_MAX, mysink->output_buffer_bound). > LZ4F_HEADER_SIZE_MAX is defined as 19 in lz4 library. With this > proposal, it is almost guaranteed that the next buffer length will > be always set to 19, which will result in failure of a call to > LZ4F_compressUpdate() with the error LZ4F_ERROR_dstMaxSize_tooSmall, > even if we had called bbsink_archive_contents() before. Sorry, should have been Max(), not Min(). > You mean the way gzip allows us to use our own alloc and free functions > by means of providing the function pointers for them. Unfortunately, > no, LZ4 does not have that kind of provision. Maybe that makes a > good proposal for LZ4 library ;-). > I cannot think of another solution to it right away. OK. Will give it some thought. -- Robert Haas EDB: http://www.enterprisedb.com
