On Wed, Sep 07, 2022 at 07:02:07PM +0900, Michael Paquier wrote:
> Before posting my previous patch, I have considered a few options:
> - Extend errormsg_buf with an error code, but the frontend does not
> care about that.
> - Make RestoreBlockImage() a backend-only routine and provide a better
> e
On Wed, Sep 07, 2022 at 03:57:29AM -0500, Justin Pryzby wrote:
> But that's also what'll happen when attempting to replay WAL using a postgres
> build which doesn't support the necessary compression method. I ran into this
> while compiling postgres locally while reporting the recovery_prefetch bu
On Wed, Sep 07, 2022 at 03:29:08PM +0900, Michael Paquier wrote:
> At the end, I have not taken the approach to use errdetail() for this
> problem as errormsg_buf is designed to include an error string. So, I
> have finished by refining the error messages generated in
> RestoreBlockImage(), consum
On Tue, Sep 06, 2022 at 03:47:05PM +0900, Michael Paquier wrote:
> On Sun, Sep 04, 2022 at 07:23:20PM -0500, Justin Pryzby wrote:
>> page = BufferGetPage(*buf);
>> if (!RestoreBlockImage(record, block_id, page))
>> -elog(ERROR, "failed to restore block
On Sun, Sep 04, 2022 at 07:23:20PM -0500, Justin Pryzby wrote:
> That's also hitting an elog().
>
> 2022-09-04 15:56:29.916 CDT startup[2625] FATAL: XX000: failed to restore
> block image
> 2022-09-04 15:56:29.916 CDT startup[2625] DETAIL: image at 0/1D11CB8
> compressed with zstd not supporte
On Mon, Sep 05, 2022 at 02:45:57PM +0200, Matthias van de Meent wrote:
> Hi,
>
> I have a small question for those involved:
I suggest to "reply all"
> Context: I'm trying to get the smallest BKPIMAGE size possible
> regardless of CPU cost, which means I'm trying multiple compressions
> to get t
Hi,
I have a small question for those involved:
Context: I'm trying to get the smallest BKPIMAGE size possible
regardless of CPU cost, which means I'm trying multiple compressions
to get the smallest data possible with the options available. This
means ignoring the wal_compression GUC in XLogComp
On Fri, Sep 02, 2022 at 06:55:11AM -0500, Justin Pryzby wrote:
> On Sun, Jun 13, 2021 at 08:24:12PM -0500, Justin Pryzby wrote:
> > In this patch series, I added compression information to the errcontext from
> > xlog_block_info(), and allow specifying compression levels like zlib-2.
> > I'll
> >
On Sun, Jun 13, 2021 at 08:24:12PM -0500, Justin Pryzby wrote:
> In this patch series, I added compression information to the errcontext from
> xlog_block_info(), and allow specifying compression levels like zlib-2. I'll
> rearrange that commit earlier if we decide that's desirable to include.
40
> 29 июня 2021 г., в 08:41, Michael Paquier написал(а):
>
> Now that v15 is open for business, I have looked again at this stuff
> this morning and committed the LZ4 part
That's great, thanks Michael!
Best regards, Andrey Borodin.
On Tue, Jun 22, 2021 at 09:11:26AM +0900, Michael Paquier wrote:
> What about the following error then? Say:
> "image at %X/%X compressed with LZ4 not supported by build, block
> %d".
>
>> Also, maybe %s can be inlined to lz4 in this case.
>
> I think that's a remnant of the zstd part of the pat
> 28 июня 2021 г., в 12:36, Michael Paquier написал(а):
>
> Removing "allows_streaming => 1" in 011_crash_recovery.pl is enough to
> make the test fail on HEAD. And the test fails equally without or
> without any changes related to wal_compression, so adding or removing
> options to wal_compr
On Mon, Jun 28, 2021 at 04:36:42PM +0900, Michael Paquier wrote:
> Is this issue different than the XID problem not matching when using
> wal_level = minimal in test 011_crash_recovery.pl? I am not sure to
> understand if you are
(This paragraph has been cut in half)
referring to a different pro
On Sat, Jun 26, 2021 at 06:11:26PM -0500, Justin Pryzby wrote:
> Curious. I found that before a4d75c86bf, there was an issue without the
> "extra" patches.
Is this issue different than the XID problem not matching when using
wal_level = minimal in test 011_crash_recovery.pl? I am not sure to
und
On Tue, Jun 22, 2021 at 12:53:46PM +0900, Michael Paquier wrote:
> > So the patches that you say are unrelated still seem to me to be a
> > prerequisite .
>
> I may be missing something, of course, but I still don't see why
> that's necessary? We don't get any test failures on HEAD by switching
>
On Mon, Jun 21, 2021 at 10:13:58PM -0500, Justin Pryzby wrote:
> @Michael: I assume that if you merge this patch, you'd set your animals to use
> wal_compression=lz4, and then they would fail the recovery tests.
Yes, I'd like to do that on my animal dangomushi.
> So the patches that you say are u
On Tue, May 25, 2021 at 12:05:19PM +0530, Dilip Kumar wrote:
> +++ b/src/test/recovery/t/011_crash_recovery.pl
> @@ -14,7 +14,7 @@ use Config;
> plan tests => 3;
>
> my $node = get_new_node('primary');
> -$node->init(allows_streaming => 1);
> +$node->init();
> $node->start;
>
> How this change
On Mon, Jun 21, 2021 at 07:19:27PM -0500, Justin Pryzby wrote:
> The two similar, existing messages are:
>
> +#define NO_LZ4_SUPPORT() \
> + ereport(ERROR, \
> + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \
> +errmsg("unsupported LZ4 compression me
On Tue, Jun 22, 2021 at 09:11:26AM +0900, Michael Paquier wrote:
> On Sun, Jun 20, 2021 at 11:15:08PM +0500, Andrey Borodin wrote:
> > I have some small questions.
> >
> > 1.
> > + report_invalid_record(record, "image at %X/%X
> > compressed with %s not supported, block %d",
> >
On Sun, Jun 20, 2021 at 11:15:08PM +0500, Andrey Borodin wrote:
> I have some small questions.
>
> 1.
> + report_invalid_record(record, "image at %X/%X
> compressed with %s not supported, block %d",
> + (uint32)
>
> 17 июня 2021 г., в 11:44, Michael Paquier написал(а):
>
> I have worked more on that today and finished with two patches
I have some small questions.
1.
+ report_invalid_record(record, "image at %X/%X
compressed with %s not supported, block %d",
+
> 17 июня 2021 г., в 11:57, Michael Paquier написал(а):
>
> On Thu, Jun 17, 2021 at 11:45:37AM +0500, Andrey Borodin wrote:
>> Konstantin, Daniil and Justin are working on compressing libpq
>> [0]. That would make walsender compress WAL automatically.
>> And we (at least I and Dan) are incline
On 17/06/2021 04:12, Michael Paquier wrote:
On Wed, Jun 16, 2021 at 11:49:51AM +0300, Heikki Linnakangas wrote:
Hmm, do we currently compress each block in a WAL record separately, for
records that contain multiple full-page images? That could make a big
difference e.g. for GiST index build that
On Thu, Jun 17, 2021 at 03:44:26PM +0900, Michael Paquier wrote:
> I have worked more on that today and finished with two patches:
> - 0001 is the mininal patch to add support for LZ4. This is in a
> rather committable shape. I noticed that we checked for an incorrect
> error code in the compress
On Thu, Jun 17, 2021 at 11:45:37AM +0500, Andrey Borodin wrote:
> Konstantin, Daniil and Justin are working on compressing libpq
> [0]. That would make walsender compress WAL automatically.
> And we (at least I and Dan) are inclined to work on compressing
> on-disk WAL as soon as libpq compression
> 17 июня 2021 г., в 06:19, Michael Paquier написал(а):
>
> On Wed, Jun 16, 2021 at 01:17:26PM +0500, Andrey Borodin wrote:
>> I agree that allowing just lz4 - is already a huge step ahead.
>
> Yeah, I am tempted to just add LZ4 as a first step as the patch
> footprint would be minimal, and w
On Thu, Jun 17, 2021 at 10:19:47AM +0900, Michael Paquier wrote:
> Yeah, I am tempted to just add LZ4 as a first step as the patch
> footprint would be minimal, and we could come back to zstd once we
> have more feedback from the field, if that's necessary. As said
> upthread, we have more flexibi
On Wed, Jun 16, 2021 at 01:17:26PM +0500, Andrey Borodin wrote:
> I agree that allowing just lz4 - is already a huge step ahead.
Yeah, I am tempted to just add LZ4 as a first step as the patch
footprint would be minimal, and we could come back to zstd once we
have more feedback from the field, if
On Wed, Jun 16, 2021 at 11:49:51AM +0300, Heikki Linnakangas wrote:
> Hmm, do we currently compress each block in a WAL record separately, for
> records that contain multiple full-page images? That could make a big
> difference e.g. for GiST index build that WAL-logs 32 pages in each record.
> If i
> 16 июня 2021 г., в 13:49, Heikki Linnakangas написал(а):
>
> Hmm, do we currently compress each block in a WAL record separately, for
> records that contain multiple full-page images? That could make a big
> difference e.g. for GiST index build that WAL-logs 32 pages in each record.
> If
On 16/06/2021 11:17, Andrey Borodin wrote:
16 июня 2021 г., в 12:18, Michael Paquier написал(а):
Among the
remaining two I would be tempted to choose LZ4. That's consistent
with what toast can use now.
I agree that allowing just lz4 - is already a huge step ahead.
But I'd suggest supporting
> 16 июня 2021 г., в 12:18, Michael Paquier написал(а):
> Among the
> remaining two I would be tempted to choose LZ4. That's consistent
> with what toast can use now.
I agree that allowing just lz4 - is already a huge step ahead.
But I'd suggest supporting zstd as well. Currently we only com
On Wed, Jun 16, 2021 at 09:39:57AM +0900, Michael Paquier wrote:
> From I'd like us to finish with here is one new algorithm method, able
> to cover a large range of cases as mentioned upthread, from
> low-CPU/low-compression to high-CPU/high-compression. It does not
> seem like a good idea to be
On Tue, Jun 15, 2021 at 11:14:56AM -0500, Justin Pryzby wrote:
> You're right, I hadn't though this through all the way.
> There's precedent if the default is non-static (wal_sync_method).
>
> But I think yes/on/true/1 should be a compatibility alias for a static thing,
> and then the only option
On Tue, Jun 15, 2021 at 07:53:26AM +0200, Peter Eisentraut wrote:
> On 15.06.21 07:37, Michael Paquier wrote:
> > > > Actually, I was just thinking that default yes/no/on/off stuff maybe
> > > > should be
> > > > defined to mean "lz4" rather than meaning pglz for "backwards
> > > > compatibility"
On 15.06.21 07:37, Michael Paquier wrote:
Actually, I was just thinking that default yes/no/on/off stuff maybe should be
defined to mean "lz4" rather than meaning pglz for "backwards compatibility".
+1
I am not sure that we have any reasons to be that aggressive about
this one either, and this
On Tue, Jun 15, 2021 at 08:08:54AM +0300, Heikki Linnakangas wrote:
> On 15/06/2021 06:42, Justin Pryzby wrote:
>> Why ? This is WAL, not table data. WAL depends on the major version, so
>> I think wal_compression could provide a different set of compression methods
>> at
>> every major release?
On 15/06/2021 06:42, Justin Pryzby wrote:
On Tue, Jun 15, 2021 at 11:39:24AM +0900, Michael Paquier wrote:
On Mon, Jun 14, 2021 at 08:42:08PM -0500, Justin Pryzby wrote:
It's USERSET following your own suggestion (which is a good suggestion):
+ {"wal_compression_method", PGC_SIGHUP, WAL_
On Tue, Jun 15, 2021 at 11:39:24AM +0900, Michael Paquier wrote:
> On Mon, Jun 14, 2021 at 08:42:08PM -0500, Justin Pryzby wrote:
It's USERSET following your own suggestion (which is a good suggestion):
> >> + {"wal_compression_method", PGC_SIGHUP, WAL_SETTINGS,
> >> + gettext_
On Mon, Jun 14, 2021 at 08:42:08PM -0500, Justin Pryzby wrote:
> On Tue, Jun 15, 2021 at 09:50:41AM +0900, Michael Paquier wrote:
>> + {"wal_compression_method", PGC_SIGHUP, WAL_SETTINGS,
>> + gettext_noop("Set the method used to compress full page images
>> in the WAL."),
>> +
On Tue, Jun 15, 2021 at 09:50:41AM +0900, Michael Paquier wrote:
> On Sun, Jun 13, 2021 at 08:24:12PM -0500, Justin Pryzby wrote:
> > I think it's more nuanced than just finding the algorithm with the least CPU
> > use. The GUC is PGC_USERSET, and it's possible that a data-loading process
> > migh
Hi,
On 2021-06-15 09:50:41 +0900, Michael Paquier wrote:
> On Sun, Jun 13, 2021 at 08:24:12PM -0500, Justin Pryzby wrote:
> > I think it's more nuanced than just finding the algorithm with the least CPU
> > use. The GUC is PGC_USERSET, and it's possible that a data-loading process
> > might want
On Sun, Jun 13, 2021 at 08:24:12PM -0500, Justin Pryzby wrote:
> I think it's more nuanced than just finding the algorithm with the least CPU
> use. The GUC is PGC_USERSET, and it's possible that a data-loading process
> might want to use zlib for better compress ratio, but an interactive OLTP
> p
Thanks for benchmarks, Justin!
> 14 июня 2021 г., в 06:24, Justin Pryzby написал(а):
>
> The GUC is PGC_USERSET
Oh, wow, that's neat. I did not realize that we can tune this for each
individual client connection. Cool!
> pglz writes ~half as much, but takes twice as long as uncompressed:
> |T
On Tue, Jun 01, 2021 at 11:06:53AM +0900, Michael Paquier wrote:
> - Speed and CPU usage. We should worry about that for CPU-bounded
> environments.
> - Compression ratio, which is just monitoring the difference in WAL.
> - Effect of the level of compression perhaps?
> - Use a fixed amount of WAL
On Mon, May 31, 2021 at 12:33:44PM +0500, Andrey Borodin wrote:
> Would it make sense to run our own benchmarks?
Yes, I think that it could be a good idea to run some custom-made
benchmarks as that could mean different bottlenecks found when it
comes to PG.
There are a couple of factors that matt
> 25 мая 2021 г., в 12:26, Michael Paquier написал(а):
>
> On Mon, May 24, 2021 at 11:44:45PM -0500, Justin Pryzby wrote:
>> The goal is to support 2+ "methods" (including "none"), which takes 4 bits,
>> so
>> may as well support 3 methods.
>>
>> - uncompressed
>> - pglz
>> - lz4
>> - zlib o
On Mon, May 24, 2021 at 11:44:45PM -0500, Justin Pryzby wrote:
> The goal is to support 2+ "methods" (including "none"), which takes 4 bits, so
> may as well support 3 methods.
>
> - uncompressed
> - pglz
> - lz4
> - zlib or zstd or ??
Let's make a proper study of all that and make a choice, the
On Tue, May 25, 2021 at 10:14 AM Justin Pryzby wrote:
Some comment.
+#define BKPIMAGE_COMPRESS_METHOD1 0x04 /* bits to encode compression method */
+#define BKPIMAGE_COMPRESS_METHOD2 0x08 /* 0=none, 1=pglz, 2=zlib */
Instead of using METHOD1, METHOD2, can we use the direct method name,
that wi
On Wed, May 19, 2021 at 06:31:15PM +0900, Michael Paquier wrote:
> I still don't understand why XID consistency has anything to do with
> the compression of FPIs. There is nothing preventing the testing of
> compression of FPIs, and plese note this argument:
> https://www.postgresql.org/message-id
On Tue, May 18, 2021 at 10:06:18PM -0500, Justin Pryzby wrote:
> On Mon, May 17, 2021 at 04:44:11PM +0900, Michael Paquier wrote:
>> On Sun, Mar 21, 2021 at 02:30:04PM -0500, Justin Pryzby wrote:
>>
>> For this patch, this is going to require a bit more in terms of library
>> linking as the block
On Mon, May 17, 2021 at 04:44:11PM +0900, Michael Paquier wrote:
> On Sun, Mar 21, 2021 at 02:30:04PM -0500, Justin Pryzby wrote:
>
> For this patch, this is going to require a bit more in terms of library
> linking as the block decompression is done in xlogreader.c, so that's one
> thing to worry
On Sun, Mar 21, 2021 at 02:30:04PM -0500, Justin Pryzby wrote:
> rebased to keep cfbot happy. This will run with default=zlib.
I have been looking at bit at this patch set, and I think that we
should do something here for 15~.
First, I think that we should make more tests and pick up one
compres
rebased to keep cfbot happy. This will run with default=zlib.
On Mon, Mar 15, 2021 at 01:09:18PM -0500, Justin Pryzby wrote:
> On Sun, Mar 14, 2021 at 07:31:35PM -0500, Justin Pryzby wrote:
> > On Sat, Mar 13, 2021 at 08:48:33PM +0500, Andrey Borodin wrote:
> > > > 13 марта 2021 г., в 06:28, Just
On Sun, Mar 14, 2021 at 07:31:35PM -0500, Justin Pryzby wrote:
> On Sat, Mar 13, 2021 at 08:48:33PM +0500, Andrey Borodin wrote:
> > > 13 марта 2021 г., в 06:28, Justin Pryzby
> > > написал(а):
> > > Updated patch with a minor fix to configure.ac to avoid warnings on OSX.
> > > And 2ndary patches
@cfbot: Resending without duplicate patches
>From 581884bb12f666a1eb6a7f4bd769b1edeba4c563 Mon Sep 17 00:00:00 2001
From: Andrey Borodin
Date: Sat, 27 Feb 2021 09:03:50 +0500
Subject: [PATCH 01/10] Allow alternate compression methods for wal_compression
TODO: bump XLOG_PAGE_MAGIC
---
doc/src/sgm
On Sat, Mar 13, 2021 at 08:48:33PM +0500, Andrey Borodin wrote:
> > 13 марта 2021 г., в 06:28, Justin Pryzby написал(а):
> > Updated patch with a minor fix to configure.ac to avoid warnings on OSX.
> > And 2ndary patches from another thread to allow passing recovery tests.
> > Renamed to WAL_COMPR
> 13 марта 2021 г., в 06:28, Justin Pryzby написал(а):
>>
> Updated patch with a minor fix to configure.ac to avoid warnings on OSX.
> And 2ndary patches from another thread to allow passing recovery tests.
> Renamed to WAL_COMPRESSION_*
> Split LZ4 support to a separate patch and support zstd
On Fri, Mar 12, 2021 at 01:45:47AM -0600, Justin Pryzby wrote:
> On Sat, Mar 06, 2021 at 12:29:14PM +0500, Andrey Borodin wrote:
> > > 1 марта 2021 г., в 10:03, Justin Pryzby написал(а):
> >
> > Justin, Michael, thanks for comments!
> >
> > As far as I understood TODO list for the patch looks as
On Sat, Mar 06, 2021 at 12:29:14PM +0500, Andrey Borodin wrote:
> > 1 марта 2021 г., в 10:03, Justin Pryzby написал(а):
>
> Justin, Michael, thanks for comments!
>
> As far as I understood TODO list for the patch looks as follows:
Your patch can be simplified some, and then the only ifdef are i
> 1 марта 2021 г., в 10:03, Justin Pryzby написал(а):
Justin, Michael, thanks for comments!
As far as I understood TODO list for the patch looks as follows:
1. Reuse compression API of other patches. But which one? Or should I invent
new one? "compressamapi.h" from "custom compression metho
On Mon, Mar 01, 2021 at 01:57:12PM +0900, Michael Paquier wrote:
> On Sun, Feb 28, 2021 at 05:08:17PM -0600, Justin Pryzby wrote:
> > Does this need to patch ./configure{,.ac} and Solution.pm for HAVE_LIBLZ4 ?
> > I suggest to also include an 0002 patch (not for commit) which changes to
> > use a
On Sun, Feb 28, 2021 at 05:08:17PM -0600, Justin Pryzby wrote:
> Does this need to patch ./configure{,.ac} and Solution.pm for HAVE_LIBLZ4 ?
> I suggest to also include an 0002 patch (not for commit) which changes to use
> a
> non-default compression, to exercise this on the CIs - linux and bsd
>
On Sat, Feb 27, 2021 at 12:43:52PM +0500, Andrey Borodin wrote:
> So I think it worth to propose a patch to make wal_compression_method =
> {"pglz", "lz4", "zlib"}. Probably, "zstd" can be added to the list.
> Attached is a draft taking CompressionId from "custom compression methods"
> patch and
Hi!
There is a lot of different compression threads nearby. And that's great!
Every few bytes going to IO still deserve to be compressed.
Currently, we have a pglz compression for WAL full page images. As shown in [0]
this leads to high CPU usage in pglz when wal_compression is on. Swapping pglz
65 matches
Mail list logo
