On Thu, Mar 17, 2022 at 06:12:20AM -0500, Justin Pryzby wrote:
> I think this should use
>
> +#include "lz4frame.h"
>
> commit ba595d2322da095a1e6703171b3f1f2815cb
> Author: Michael Paquier
> Date: Fri Nov 5 11:33:25 2021 +0900
>
> Add support for LZ4 compression in pg_receivewal
Ye
On Sat, Feb 12, 2022 at 12:52:40PM +0900, Michael Paquier wrote:
> On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote:
> > Over in
> > http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com
> > I was noticing that CreateWalTarMethod doesn't support LZ4
> > c
On Fri, Feb 11, 2022 at 10:52 PM Michael Paquier wrote:
> > And, on a related note, Michael, do you plan to get something
> > committed here?
>
> Apart from f79962d, ba5 and 50e1441, I don't think that there was
> something left to do for this thread. Perhaps I am missing something?
Oh, my m
On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote:
> Over in
> http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com
> I was noticing that CreateWalTarMethod doesn't support LZ4
> compression. It would be nice if it did. I thought maybe the patch on
> this
On Thu, Nov 4, 2021 at 10:47 PM Michael Paquier wrote:
> Indeed. My rebase was a bit sloppy here.
Hi!
Over in
http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com
I was noticing that CreateWalTarMethod doesn't support LZ4
compression. It would be nice if it did
On Wed, Nov 24, 2021 at 10:55 AM Michael Paquier
wrote:
> On Mon, Nov 22, 2021 at 09:02:47AM -0500, Robert Haas wrote:
> > On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
> > wrote:
> >> Fair enough. But, still I have a doubt in mind what benefit would that
> >> really bring to us here, because we
On Mon, Nov 22, 2021 at 09:02:47AM -0500, Robert Haas wrote:
> On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
> wrote:
>> Fair enough. But, still I have a doubt in mind what benefit would that
>> really bring to us here, because we are immediately also freeing the
>> lz4buf without using it anywher
On Mon, Nov 22, 2021 at 12:46 AM Jeevan Ladhe
wrote:
> Fair enough. But, still I have a doubt in mind what benefit would that
> really bring to us here, because we are immediately also freeing the
> lz4buf without using it anywhere.
Yeah, I'm also doubtful about that. If we're freeng the compress
On Fri, Nov 19, 2021 at 7:37 AM Michael Paquier wrote:
> On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
> > In dir_open_for_write() I observe that we are writing the header
> > and then calling LZ4F_compressEnd() in case there is an error
> > while writing the buffer to the file, a
‐‐‐ Original Message ‐‐‐
On Friday, November 19th, 2021 at 3:07 AM, Michael Paquier
wrote:
> On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
>
> > In dir_open_for_write() I observe that we are writing the header
> > and then calling LZ4F_compressEnd() in case there is an e
On Thu, Nov 18, 2021 at 07:54:37PM +0530, Jeevan Ladhe wrote:
> In dir_open_for_write() I observe that we are writing the header
> and then calling LZ4F_compressEnd() in case there is an error
> while writing the buffer to the file, and the output buffer of
> LZ4F_compressEnd() is not written anywh
In dir_open_for_write() I observe that we are writing the header
and then calling LZ4F_compressEnd() in case there is an error
while writing the buffer to the file, and the output buffer of
LZ4F_compressEnd() is not written anywhere. Why should this be
necessary? To flush the contents of the intern
‐‐‐ Original Message ‐‐‐
On Friday, November 5th, 2021 at 3:47 AM, Michael Paquier
wrote:
>
> I have spent an extra couple of hours staring at the code, and the
> whole looked fine, so applied. While on it, I have tested the new TAP
> tests with all the possible combinations of --wit
On Thu, Nov 04, 2021 at 05:02:28PM +, gkokola...@pm.me wrote:
> Removed an extra condinional check while switching over compression_method.
Indeed. My rebase was a bit sloppy here.
> because compression_method is the global option exposed to the whereas
> wal_compression_method is the local
‐‐‐ Original Message ‐‐‐
On Thursday, November 4th, 2021 at 9:21 AM, Michael Paquier
wrote:
> > +#ifdef HAVE_LIBLZ4
> > +while (readp < readend)
> > +{
> > +size_tread_size = 1;
> > +size_tout_size = 1;
> > +
> > +status = LZ4F_decompress
‐‐‐ Original Message ‐‐‐
On Thursday, November 4th, 2021 at 9:21 AM, Michael Paquier
wrote:
> On Thu, Nov 04, 2021 at 04:31:48PM +0900, Michael Paquier wrote:
> Thanks. I have looked at 0001 today, and applied it after fixing a
> couple of issues.
Great! Thank you very much.
> Fro
On Thu, Nov 04, 2021 at 04:31:48PM +0900, Michael Paquier wrote:
> I would have expected read_size to be (readend - readp) to match with
> the remaining data in the read buffer that we still need to read.
> Shouldn't out_size also be LZ4_CHUNK_SZ to match with the size of the
> output buffer where
On Wed, Nov 03, 2021 at 09:11:24AM +, gkokola...@pm.me wrote:
> Please find v9 attached.
Thanks. I have looked at 0001 today, and applied it after fixing a
couple of issues. From memory:
- zlib.h was missing from pg_receivewal.c, issue that I noticed after
removing the redefinition of Z_DEFA
‐‐‐ Original Message ‐‐‐
On Wednesday, November 3rd, 2021 at 9:05 AM, wrote:
> ‐‐‐ Original Message ‐‐‐
>
> On Wednesday, November 3rd, 2021 at 12:23 AM, Michael Paquier
> mich...@paquier.xyz wrote:
> > On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
> > > On Tue,
‐‐‐ Original Message ‐‐‐
On Wednesday, November 3rd, 2021 at 12:23 AM, Michael Paquier
wrote:
> On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
>> On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander mag...@hagander.net wrote:
>>> I think for the end user, it is strictly bette
On Tue, Nov 02, 2021 at 12:31:47PM -0400, Robert Haas wrote:
> On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander wrote:
>> I think for the end user, it is strictly better to name it "gzip",
>> and given that the target of this option is the end user we should
>> do so. (It'd be different it we were t
On Tue, Nov 2, 2021 at 8:17 AM Magnus Hagander wrote:
> Um, why?
>
> That we are using zlib to provide the compression is an implementation
> detail. Whereas AFAIK "gzip" refers to both the program and the format. And
> we specifically use the gzxxx() functions in zlib, in order to produce gzip
On Tue, Nov 2, 2021 at 9:51 AM Michael Paquier wrote:
> On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
> > On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> > > Agreed.
> > >
> > > I have already started on v8 of the patch with that technique. I should
> > > be
‐‐‐ Original Message ‐‐‐
On Tuesday, November 2nd, 2021 at 9:51 AM, Michael Paquier
wrote:
> On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
> > On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> > > Agreed.
> > >
> > > I have already started on v8 of
On Tue, Nov 02, 2021 at 07:27:50AM +0900, Michael Paquier wrote:
> On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> > Agreed.
> >
> > I have already started on v8 of the patch with that technique. I should
> > be able to update the thread soon.
>
> Nice, thanks!
By the way, I
On Mon, Nov 01, 2021 at 08:39:59AM +, gkokola...@pm.me wrote:
> Agreed.
>
> I have already started on v8 of the patch with that technique. I should
> be able to update the thread soon.
Nice, thanks!
--
Michael
signature.asc
Description: PGP signature
‐‐‐ Original Message ‐‐‐
On Monday, November 1st, 2021 at 9:09 AM, Michael Paquier
wrote:
> On Fri, Oct 29, 2021 at 08:38:33PM +0900, Michael Paquier wrote:
>
> It would be good to test with many segments, but could we think about
> just relying on LZ4F_decompress() with a frame and
On Fri, Oct 29, 2021 at 08:38:33PM +0900, Michael Paquier wrote:
> Why would the header size change between the moment the segment is
> begun and it is finished? We could store it in memory and write it
> again when the segment is closed instead, even if it means to fseek()
> back to the beginning
On Fri, Oct 29, 2021 at 09:45:41AM +, gkokola...@pm.me wrote:
> On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier
> wrote:
>> We don't really care about contentSize as long as a segment is not
>> completed. Rather than filling contentSize all the time we write
>> something, we'd
‐‐‐ Original Message ‐‐‐
On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier
wrote:
> On Fri, Sep 17, 2021 at 08:12:42AM +, gkokola...@pm.me wrote:
>
> I have been digging into the issue I saw in the TAP tests when closing
> a segment, and found the problem. The way you
On Fri, Sep 17, 2021 at 08:12:42AM +, gkokola...@pm.me wrote:
> Yeah, I was considering it to split them over to a separate commit,
> then decided against it. Will do so in the future.
I have been digging into the issue I saw in the TAP tests when closing
a segment, and found the problem. The
‐‐‐ Original Message ‐‐‐
On Friday, September 17th, 2021 at 09:39, Michael Paquier
wrote:
> On Thu, Sep 16, 2021 at 03:17:15PM +, gkokola...@pm.me wrote:
>
> > Hopefully fixed.
>
> Thanks for the new version. I have put my hands on the patch, and
> began reviewing its internals w
On Thu, Sep 16, 2021 at 03:17:15PM +, gkokola...@pm.me wrote:
> Hopefully fixed.
Thanks for the new version. I have put my hands on the patch, and
began reviewing its internals with LZ4. I am not done with it yet,
and I have noticed some places that could be improved (error handling,
some us
‐‐‐ Original Message ‐‐‐
On Wednesday, September 15th, 2021 at 08:46, Michael Paquier
mich...@paquier.xyz wrote:
Hi,
thank you for the review.
> On Fri, Sep 10, 2021 at 08:21:51AM +, gkokola...@pm.me wrote:
>
> > Agreed. A default value of 5, which is in the middle point of options
On Fri, Sep 10, 2021 at 08:21:51AM +, gkokola...@pm.me wrote:
> Agreed. A default value of 5, which is in the middle point of options, has
> been
> defined and used.
>
> In addition, the tests have been adjusted to mimic the newly added gzip tests.
Looking at lz4frame.h, there is LZ4F_flush(
‐‐‐ Original Message ‐‐‐
On Saturday, September 11th, 2021 at 07:02, Jian Guo wrote:
Hi,
thank you for looking at the patch.
> - LZ4F_decompressionContext_t ctx = NULL;
> - snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir,
> dirent->d_name);
> -
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed
to be
* completed. For non-compressed segments just check the
on-disk size
-* and see if it matches a completed
‐‐‐ Original Message ‐‐‐
On Friday, July 9th, 2021 at 04:49, Michael Paquier wrote:
Hi,
please find v3 of the patch attached, rebased to the current head.
> > Michael Paquier wrote:
> >
>
> * http://www.zlib.org/rfc-gzip.html.
>
> - - For lz4 compressed segments
> */
>
On Mon, Jul 12, 2021 at 3:15 PM Magnus Hagander wrote:
>
> On Mon, Jul 12, 2021 at 11:33 AM wrote:
> >
> >
> >
> > ‐‐‐ Original Message ‐‐‐
> >
> > On Monday, July 12th, 2021 at 07:56, Michael Paquier
> > wrote:
> >
> > > On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
> >
On Mon, Jul 12, 2021 at 11:33 AM wrote:
>
>
>
> ‐‐‐ Original Message ‐‐‐
>
> On Monday, July 12th, 2021 at 07:56, Michael Paquier
> wrote:
>
> > On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
> >
> > > On Thu, Jul 8, 2021 at 7:48 PM gkokola...@pm.me wrote:
> > >
> > > > We
‐‐‐ Original Message ‐‐‐
On Monday, July 12th, 2021 at 07:56, Michael Paquier
wrote:
> On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
>
> > On Thu, Jul 8, 2021 at 7:48 PM gkokola...@pm.me wrote:
> >
> > > We can, though I am not in favour of doing so. There is seemingly
On Mon, Jul 12, 2021 at 11:10:24AM +0530, Dilip Kumar wrote:
> On Thu, Jul 8, 2021 at 7:48 PM wrote:
>> We can, though I am not in favour of doing so. There is seemingly
>> little benefit for added complexity.
>
> I am really not sure what complexity you are talking about, do you
> mean since wit
On Thu, Jul 8, 2021 at 7:48 PM wrote:
>
> Dilip Kumar wrote:
>
> > Wouldn't it be better to call it compression method instead of
> > compression program?
>
> Agreed. This is inline with the suggestions of other reviewers.
> Find the change in the attached patch.
Thanks, I will have a look.
> >
‐‐‐ Original Message ‐‐‐
On Friday, July 9th, 2021 at 04:49, Michael Paquier wrote:
> On Thu, Jul 08, 2021 at 02:18:40PM +, gkokola...@pm.me wrote:
>
> > please find v2 of the patch which tries to address the commends
> >
> > received so far.
>
> Thanks!
>
> > Michael Paquier wrot
On Thu, Jul 08, 2021 at 02:18:40PM +, gkokola...@pm.me wrote:
> please find v2 of the patch which tries to address the commends
> received so far.
Thanks!
> Michael Paquier wrote:
>> + system_or_bail('lz4', '-t', $lz4_wals[0]);
>> I think that you should just drop this part of the test. Th
Hi,
please find v2 of the patch which tries to address the commends received so far.
Thank you all for your comments.
Michael Paquier wrote:
> Documentation is missing from the patch.
>
It has now been added.
> + LZ4F_compressionContext_t ctx;
> + size_t outbufCapacity;
> + void
‐‐‐ Original Message ‐‐‐
On Friday, July 2nd, 2021 at 03:10, Michael Paquier wrote:
> On Thu, Jul 01, 2021 at 02:10:17PM +, gkokola...@pm.me wrote:
>
> > Micheal suggested on the same thread to move my entry in the help output so
> > that
> >
> > the output remains ordered. I wou
On Thu, Jul 01, 2021 at 02:10:17PM +, gkokola...@pm.me wrote:
> Micheal suggested on the same thread to move my entry in the help output so
> that
> the output remains ordered. I would like the options for the compression
> method and
> the already existing compression level to next to each o
‐‐‐ Original Message ‐‐‐
On Thursday, July 1st, 2021 at 15:58, Magnus Hagander
wrote:
> On Thu, Jul 1, 2021 at 3:39 PM gkokola...@pm.me wrote:
>
> > ‐‐‐ Original Message ‐‐‐
> >
> > On Thursday, July 1st, 2021 at 12:28, Magnus Hagander mag...@hagander.net
> > wrote:
> >
> >
On Thu, Jul 1, 2021 at 3:39 PM wrote:
>
>
>
> ‐‐‐ Original Message ‐‐‐
>
> On Thursday, July 1st, 2021 at 12:28, Magnus Hagander
> wrote:
>
> > On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbal...@gmail.com wrote:
> >
> > > On Tue, Jun 29, 2021 at 8:15 PM gkokola...@pm.me wrote:
> > >
‐‐‐ Original Message ‐‐‐
On Thursday, July 1st, 2021 at 12:28, Magnus Hagander
wrote:
> On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar dilipbal...@gmail.com wrote:
>
> > On Tue, Jun 29, 2021 at 8:15 PM gkokola...@pm.me wrote:
> >
> > > Hi,
> > >
> > > The program pg_receivewal can use g
On Wed, Jun 30, 2021 at 8:34 AM Dilip Kumar wrote:
>
> On Tue, Jun 29, 2021 at 8:15 PM wrote:
> >
> > Hi,
> >
> > The program pg_receivewal can use gzip compression to store the received
> > WAL.
> > This patch teaches it to be able to use lz4 compression if the binary is
> > build
> > using th
On Tue, Jun 29, 2021 at 8:15 PM wrote:
>
> Hi,
>
> The program pg_receivewal can use gzip compression to store the received WAL.
> This patch teaches it to be able to use lz4 compression if the binary is build
> using the -llz4 flag.
+1 for the idea
Some comments/suggestions on the patch
1.
@@
On Tue, Jun 29, 2021 at 02:45:17PM +, gkokola...@pm.me wrote:
> The program pg_receivewal can use gzip compression to store the received WAL.
> This patch teaches it to be able to use lz4 compression if the binary is build
> using the -llz4 flag.
Nice.
> Previously, the user had to use the op
54 matches
Mail list logo
