Re: pglz performance

Hi Petr! > 4 авг. 2019 г., в 05:41, Petr Jelinek написал(а): > > Just so that we don't idly talk, what do you think about the attached? > It: > - adds new GUC compression_algorithm with possible values of pglz (default) > and lz4 (if lz4 is compiled in), requires SIGHUP > - adds --with-lz4 conf

Re: pglz performance

On Fri, Nov 29, 2019 at 10:18:18AM +0500, Andrey Borodin wrote: >> 29 нояб. 2019 г., в 3:43, Tomas Vondra >> написал(а): >> >> OK, pushed, with some minor cosmetic tweaks on the comments (essentially >> using the formatting trick pointed out by Alvaro), and removing one >> unnecessary change in p

Re: pglz performance

> 29 нояб. 2019 г., в 3:43, Tomas Vondra > написал(а): > > OK, pushed, with some minor cosmetic tweaks on the comments (essentially > using the formatting trick pointed out by Alvaro), and removing one > unnecessary change in pglz_maximum_compressed_size. Cool, thanks! > pglz_maximum_compre

Re: pglz performance

On Wed, Nov 27, 2019 at 11:27:49PM +0500, Andrey Borodin wrote: 27 нояб. 2019 г., в 20:28, Tomas Vondra написал(а): 6) I'm pretty sure the comment in the 'while (off < len)' branch will be badly mangled by pgindent. I think I can just write it without line limit and then run pgindent. W

Re: pglz performance

On Wed, Nov 27, 2019 at 04:28:18PM +0100, Tomas Vondra wrote: > Yes. I was considering using the test_pglz extension first, but in the > end I decided an end-to-end test is easier to do and more relevant. I actually got something in this area in one of my trees: https://github.com/michaelpq/pg_plu

Re: pglz performance

On 2019-Nov-27, Andrey Borodin wrote: > > > > 27 нояб. 2019 г., в 20:28, Tomas Vondra > > написал(а): > > > >>> > >>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be > >>> badly mangled by pgindent. > >> > >> I think I can just write it without line limit and then r

Re: pglz performance

> 27 нояб. 2019 г., в 20:28, Tomas Vondra > написал(а): > >>> >>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be >>> badly mangled by pgindent. >> >> I think I can just write it without line limit and then run pgindent. >> Will try to do it this evening. Also, I wil

Re: pglz performance

On Wed, Nov 27, 2019 at 05:47:25PM +0500, Andrey Borodin wrote: Hi Tomas! Thanks for benchmarking this! 26 нояб. 2019 г., в 14:43, Tomas Vondra написал(а): On Mon, Nov 25, 2019 at 05:29:40PM +0900, Michael Paquier wrote: On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote: I th

Re: pglz performance

Hi Tomas! Thanks for benchmarking this! > 26 нояб. 2019 г., в 14:43, Tomas Vondra > написал(а): > > On Mon, Nov 25, 2019 at 05:29:40PM +0900, Michael Paquier wrote: >> On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote: >>> I think status Needs Review describes what is going on bet

Re: pglz performance

On Wed, Nov 27, 2019 at 05:01:47PM +0900, Michael Paquier wrote: On Tue, Nov 26, 2019 at 09:05:59PM +0100, Tomas Vondra wrote: Yeah, although the difference is minimal. We could probably construct a benchmark where #2 wins, but I think these queries are fairly realistic. So I'd just go with #1.

Re: pglz performance

On Tue, Nov 26, 2019 at 09:05:59PM +0100, Tomas Vondra wrote: > Yeah, although the difference is minimal. We could probably construct a > benchmark where #2 wins, but I think these queries are fairly realistic. > So I'd just go with #1. Nice results. Using your benchmarks it indeed looks like pat

Re: pglz performance

On Tue, Nov 26, 2019 at 08:17:13PM +0100, Peter Eisentraut wrote: On 2019-11-26 10:43, Tomas Vondra wrote: In general, I think the results for both patches seem clearly a win, but maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd probably go with that one. Patch 1 is als

Re: pglz performance

On 2019-11-26 10:43, Tomas Vondra wrote: In general, I think the results for both patches seem clearly a win, but maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd probably go with that one. Patch 1 is also the simpler patch, so it seems clearly preferable. -- Peter Eise

Re: pglz performance

On Mon, Nov 25, 2019 at 05:29:40PM +0900, Michael Paquier wrote: On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote: I think status Needs Review describes what is going on better. It's not like something is awaited from my side. Indeed. You are right so I have moved the patch inst

Re: pglz performance

On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote: > I think status Needs Review describes what is going on better. It's > not like something is awaited from my side. Indeed. You are right so I have moved the patch instead, with "Needs review". The patch status was actually incorrec

Re: pglz performance

> 25 нояб. 2019 г., в 13:03, Michael Paquier написал(а): > > On Wed, Nov 06, 2019 at 09:04:25AM +0100, Peter Eisentraut wrote: >> OK, waiting on some independent verification of benchmark numbers. > > Still waiting for these after 19 days, so the patch has been marked as > returned with feedb

Re: pglz performance

On Wed, Nov 06, 2019 at 09:04:25AM +0100, Peter Eisentraut wrote: > OK, waiting on some independent verification of benchmark numbers. Still waiting for these after 19 days, so the patch has been marked as returned with feedback. -- Michael signature.asc Description: PGP signature

Re: pglz performance

On 2019-11-01 17:59, Tomas Vondra wrote: I'd try running the benchmarks to verify the numbers, and maybe do some additional tests, but it's not clear to me which patches should I use. I think the last patches with 'hacked' and 'hacked8' in the name are a couple of months old, and the recent post

Re: pglz performance

On 2019-11-01 16:48, Alvaro Herrera wrote: As I understand that report, in these results "less is better", so the hacked8 variant shows better performance (33.8) than current (42.5). The "hacked" variant shows worse performance (48.2) that the current code. Which appears to be the opposite of,

Re: pglz performance

Hi Tels! Thanks for your interest in fast decompression. > 3 нояб. 2019 г., в 12:24, Tels написал(а): > > I wonder if you agree and what would happen if you try this variant on your > corpus tests. I've tried some different optimization for literals. For example loop unrolling[0] and literals

Re: pglz performance

Hello Andrey, On 2019-11-02 12:30, Andrey Borodin wrote: 1 нояб. 2019 г., в 18:48, Alvaro Herrera написал(а): PFA two patches: v4-0001-Use-memcpy-in-pglz-decompression.patch (known as 'hacked' in test_pglz extension) v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch (known as 'ha

Re: pglz performance

> 1 нояб. 2019 г., в 18:48, Alvaro Herrera > написал(а): > > On 2019-Nov-01, Peter Eisentraut wrote: > >> On 2019-10-25 07:05, Andrey Borodin wrote: 21 окт. 2019 г., в 14:09, Andrey Borodin написал(а): With Silesian corpus pglz_decompress_hacked is actually decreasing p

Re: pglz performance

On Fri, Nov 01, 2019 at 12:48:28PM -0300, Alvaro Herrera wrote: On 2019-Nov-01, Peter Eisentraut wrote: On 2019-10-25 07:05, Andrey Borodin wrote: > > 21 окт. 2019 г., в 14:09, Andrey Borodin написал(а): > > > > With Silesian corpus pglz_decompress_hacked is actually decreasing performance on

Re: pglz performance

On 2019-Nov-01, Peter Eisentraut wrote: > On 2019-10-25 07:05, Andrey Borodin wrote: > > > 21 окт. 2019 г., в 14:09, Andrey Borodin > > > написал(а): > > > > > > With Silesian corpus pglz_decompress_hacked is actually decreasing > > > performance on high-entropy data. > > > Meanwhile pglz_deco

Re: pglz performance

On 2019-10-25 07:05, Andrey Borodin wrote: 21 окт. 2019 г., в 14:09, Andrey Borodin написал(а): With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data. Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress. In spite of this be

Re: pglz performance

> 21 окт. 2019 г., в 14:09, Andrey Borodin написал(а): > > With Silesian corpus pglz_decompress_hacked is actually decreasing > performance on high-entropy data. > Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress. > In spite of this benchmarks, I think that pglz_dec

Re: pglz performance

> 28 сент. 2019 г., в 10:29, Andrey Borodin написал(а): > > I hope to benchmark decompression on Silesian corpus soon. I've done it. And results are quite controversial. Dataset adds 12 payloads to our 5. Payloads have relatively high entropy. In many cases pglz cannot compress them at all,

Re: pglz performance

Oleg, Peter, thanks for looking into this! I hope to benchmark decompression on Silesian corpus soon. PFA v2 with better comments. > 27 сент. 2019 г., в 14:41, Peter Eisentraut > написал(а): > > After reviewing this thread and more testing, I think > 0001-Use-memcpy-in-pglz-decompression.patc

Re: pglz performance

On 2019-09-04 14:45, Andrey Borodin wrote: >> On 2019-09-04 11:22, Andrey Borodin wrote: What about the two patches? Which one is better? >>> On our observations pglz_decompress_hacked.patch is best for most of tested >>> platforms. >>> Difference is that pglz_decompress_hacked8.patch will n

Re: pglz performance

Hi, On 05/08/2019 09:26, Andres Freund wrote: Hi, On 2019-08-05 00:19:28 +0200, Petr Jelinek wrote: It carries that information inside the compressed value, like I said in the other reply, that's why the extra byte. I'm not convinced that that is a good plan - imo the reference to the compre

Re: pglz performance

On Wed, Sep 4, 2019 at 12:22 PM Andrey Borodin wrote: > > Hi, Peter! Thanks for looking into this. > > > 4 сент. 2019 г., в 14:09, Peter Eisentraut > > написал(а): > > > > On 2019-06-24 10:44, Andrey Borodin wrote: > >>> 18 мая 2019 г., в 11:44, Andrey Borodin написал(а): > >>> > >> Hi! > >> He

Re: pglz performance

I just noticed we had two CF items pointing to this thread, https://commitfest.postgresql.org/24/2119/ https://commitfest.postgresql.org/24/2180/ so I marked the newer one as withdrawn. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DB

Re: pglz performance

> 4 сент. 2019 г., в 17:40, Peter Eisentraut > написал(а): > > On 2019-09-04 11:22, Andrey Borodin wrote: >>> What about the two patches? Which one is better? >> On our observations pglz_decompress_hacked.patch is best for most of tested >> platforms. >> Difference is that pglz_decompress_h

Re: pglz performance

On 2019-09-04 11:22, Andrey Borodin wrote: >> What about the two patches? Which one is better? > On our observations pglz_decompress_hacked.patch is best for most of tested > platforms. > Difference is that pglz_decompress_hacked8.patch will not appply optimization > if decompressed match is not

Re: pglz performance

Hi, Peter! Thanks for looking into this. > 4 сент. 2019 г., в 14:09, Peter Eisentraut > написал(а): > > On 2019-06-24 10:44, Andrey Borodin wrote: >>> 18 мая 2019 г., в 11:44, Andrey Borodin написал(а): >>> >> Hi! >> Here's rebased version of patches. >> >> Best regards, Andrey Borodin. >

Re: pglz performance

On 2019-06-24 10:44, Andrey Borodin wrote: >> 18 мая 2019 г., в 11:44, Andrey Borodin написал(а): >> > Hi! > Here's rebased version of patches. > > Best regards, Andrey Borodin. I think this is the most recent patch for the CF entry . What about the

Re: pglz performance

Hi, On 2019-08-05 16:04:46 +0900, Michael Paquier wrote: > On Fri, Aug 02, 2019 at 07:52:39PM +0200, Tomas Vondra wrote: > > On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote: > >> Why would they be stuck continuing to *compress* with pglz? As we > >> fully retoast on write anyway we c

Re: pglz performance

Hi, On 2019-08-05 00:19:28 +0200, Petr Jelinek wrote: > It carries that information inside the compressed value, like I said in the > other reply, that's why the extra byte. I'm not convinced that that is a good plan - imo the reference to the compressed data should carry that information. I.e.

Re: pglz performance

On Fri, Aug 02, 2019 at 07:52:39PM +0200, Tomas Vondra wrote: > On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote: >> Why would they be stuck continuing to *compress* with pglz? As we >> fully retoast on write anyway we can just gradually switch over to the >> better algorithm. Decompre

Re: pglz performance

Hi, On 05/08/2019 00:15, Andres Freund wrote: Hi, On 2019-08-04 17:53:26 +0200, Petr Jelinek wrote: 5) I wonder why compression_algorithm is defined as PGC_SIGHUP. Why not to allow users to set it per session? I suppose we might have a separate option for WAL compression_algorithm. Yeah I w

Re: pglz performance

Hi, On 2019-08-04 17:53:26 +0200, Petr Jelinek wrote: > > 5) I wonder why compression_algorithm is defined as PGC_SIGHUP. Why not > > to allow users to set it per session? I suppose we might have a separate > > option for WAL compression_algorithm. > > > > Yeah I was thinking we might want to ch

Re: pglz performance

Hi, On 04/08/2019 21:20, Andres Freund wrote: On 2019-08-04 02:41:24 +0200, Petr Jelinek wrote: Same here. Just so that we don't idly talk, what do you think about the attached? Cool! It: - adds new GUC compression_algorithm with possible values of pglz (default) and lz4 (if lz4 is compile

Re: pglz performance

Hi, On 2019-08-04 02:41:24 +0200, Petr Jelinek wrote: > Same here. > > Just so that we don't idly talk, what do you think about the attached? Cool! > It: > - adds new GUC compression_algorithm with possible values of pglz (default) > and lz4 (if lz4 is compiled in), requires SIGHUP As Tomas re

Re: pglz performance

On Sun, Aug 04, 2019 at 05:53:26PM +0200, Petr Jelinek wrote: ... 4) I did a simple test with physical replication, with lz4 enabled on both sides (well, can't build without lz4 anyway, per previous point). It immediately failed like this: FATAL:  failed to restore block image CONTEXT:  WAL

Re: pglz performance

Hi, On 04/08/2019 11:57, Andrey Borodin wrote: 2 авг. 2019 г., в 21:39, Andres Freund написал(а): On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF. I don't understand why it's a goo

Re: pglz performance

Hi, On 04/08/2019 13:52, Tomas Vondra wrote: On Sun, Aug 04, 2019 at 02:41:24AM +0200, Petr Jelinek wrote: Hi, On 02/08/2019 21:48, Tomas Vondra wrote: On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote: Another question is whether we'd actually want to include the code in core

Re: pglz performance

On Sun, Aug 04, 2019 at 02:41:24AM +0200, Petr Jelinek wrote: Hi, On 02/08/2019 21:48, Tomas Vondra wrote: On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote: Another question is whether we'd actually want to include the code in core directly, or use system libraries (and if some

Re: pglz performance

> 2 авг. 2019 г., в 21:39, Andres Freund написал(а): > > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: >> We have some kind of "roadmap" of "extensible pglz". We plan to provide >> implementation on Novembers CF. > > I don't understand why it's a good idea to improve the compression si

Re: pglz performance

Hi, On 02/08/2019 21:48, Tomas Vondra wrote: On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote: Another question is whether we'd actually want to include the code in core directly, or use system libraries (and if some packagers might decide to disable that, for whatever reason).

Re: pglz performance

On 02.08.2019 21:20, Andres Freund wrote: Another question is whether we'd actually want to include the code in core directly, or use system libraries (and if some packagers might decide to disable that, for whatever reason). I'd personally say we should have an included version, and a --wit

Re: pglz performance

On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote: Hi, On 2019-08-02 19:52:39 +0200, Tomas Vondra wrote: Hmmm, I don't remember the details of those patches so I didn't realize it allows incremental recompression. If that's possible, that would mean existing systems can start using

Re: pglz performance

Hi, On 2019-08-02 19:52:39 +0200, Tomas Vondra wrote: > Hmmm, I don't remember the details of those patches so I didn't realize > it allows incremental recompression. If that's possible, that would mean > existing systems can start using it. Which is good. That depends on what do you mean by "inc

Re: pglz performance

On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote: Hi, On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote: On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote: > Hi, > > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: > > We have some kind of "roadmap" of "extensible pgl

Re: pglz performance

Hi, On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote: > On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote: > > Hi, > > > > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: > > > We have some kind of "roadmap" of "extensible pglz". We plan to > > > provide implementation on November

Re: pglz performance

On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote: Hi, On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF. I don't understand why it's a good idea to improve the compression sid

Re: pglz performance

Hi, On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: > We have some kind of "roadmap" of "extensible pglz". We plan to provide > implementation on Novembers CF. I don't understand why it's a good idea to improve the compression side of pglz. There's plenty other people that spent a lot of tim

Re: pglz performance

Thanks for looking into this! > 2 авг. 2019 г., в 19:43, Tomas Vondra > написал(а): > > On Fri, Aug 02, 2019 at 04:45:43PM +0300, Konstantin Knizhnik wrote: >> >> It takes me some time to understand that your memcpy optimization is >> correct;) Seems that comments are not explanatory enough..

Re: pglz performance

On Fri, Aug 02, 2019 at 04:45:43PM +0300, Konstantin Knizhnik wrote: On 27.06.2019 21:33, Andrey Borodin wrote: 13 мая 2019 г., в 12:14, Michael Paquier написал(а): Decompression can matter a lot for mostly-read workloads and compression can become a bottleneck for heavy-insert loads, so i

Re: pglz performance

On 27.06.2019 21:33, Andrey Borodin wrote: 13 мая 2019 г., в 12:14, Michael Paquier написал(а): Decompression can matter a lot for mostly-read workloads and compression can become a bottleneck for heavy-insert loads, so improving compression or decompression should be two separate problems

Re: pglz performance

> 13 мая 2019 г., в 12:14, Michael Paquier написал(а): > > Decompression can matter a lot for mostly-read workloads and > compression can become a bottleneck for heavy-insert loads, so > improving compression or decompression should be two separate > problems, not two problems linked. Any impr

Re: pglz performance

> 18 мая 2019 г., в 11:44, Andrey Borodin написал(а): > Hi! Here's rebased version of patches. Best regards, Andrey Borodin. 0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch Description: Binary data 0001-Use-memcpy-in-pglz-decompression.patch Description: Binary data

Re: pglz performance

Hi hackers! I am a student participating in GSoC 2019. I am looking forward to working with you all and learning from you. My project would aim to provide the ability to de-TOAST a fully TOAST'd and compressed field using an iterator. For more details, please take a look at my proposal[0]. Any sugg

Re: pglz performance

> 17 мая 2019 г., в 18:40, Gasper Zejn написал(а): > > I've tested according to instructions at the test repo > https://github.com/x4m/test_pglz > > Test_pglz is at a97f63b and postgres at 6ba500. > > Hardware is desktop AMD Ryzen 5 2600, 32GB RAM > > Decompressor score (summ of all times):

Re: pglz performance

On 16. 05. 19 19:13, Andrey Borodin wrote: > >> 15 мая 2019 г., в 15:06, Andrey Borodin написал(а): >> >> Owners of AMD and ARM devices are welcome. I've tested according to instructions at the test repo https://github.com/x4m/test_pglz Test_pglz is at a97f63b and postgres at 6ba500. Hardware

Re: pglz performance

> 17 мая 2019 г., в 6:44, Michael Paquier написал(а): > > That's nice. > > From the numbers you are presenting here, all of them are much better > than the original, and there is not much difference between any of the > patched versions. Having a 20%~30% improvement with a patch is very > nic

Re: pglz performance

On Thu, May 16, 2019 at 10:13:22PM +0500, Andrey Borodin wrote: > Meanwhile I made some enhancements to test suit: > Intel Server > NOTICE: 0: Decompressor pglz_decompress_hacked result 10.346763 > NOTICE: 0: Decompressor pglz_decompress_hacked8 result 11.192078 > NOTICE: 0: Decompre

Re: pglz performance

> 15 мая 2019 г., в 15:06, Andrey Borodin написал(а): > > Owners of AMD and ARM devices are welcome. Yandex hardware RND guys gave me ARM server and Power9 server. They are looking for AMD and some new Intel boxes. Meanwhile I made some enhancements to test suit: 1. I've added Shakespeare pa

Re: pglz performance

> 13 мая 2019 г., в 12:14, Michael Paquier написал(а): > >> Currently we test mostly decompression improvements against two WALs >> and one data file taken from pgbench-generated database. Any >> suggestion on more relevant data payloads are very welcome. > > Text strings made of random data

Re: pglz performance

On Mon, May 13, 2019 at 07:45:59AM +0500, Andrey Borodin wrote: > I was reviewing Paul Ramsey's TOAST patch[0] and noticed that there > is a big room for improvement in performance of pglz compression and > decompression. Yes, I believe so too. pglz is a huge CPU-consumer when it comes to compila