On Tue, May 21, 2024 at 12:08 PM Jacob Burroughs
wrote:
> I think I thought I was writing about something else when I wrote that
> :sigh:. I think what I really should have written was a version of
> the part below, which is that we use streaming decompression, only
> decompress 8kb at a time, an
On Tue, May 21, 2024 at 1:24 PM Jacob Champion
wrote:
>
> We absolutely have to document the risks and allow clients to be
> written safely. But I think server-side controls on risky behavior
> have proven to be generally more valuable, because the server
> administrator is often in a better spot
On Tue, May 21, 2024 at 1:39 PM Jacob Champion
wrote:
>
> If the server doesn't reject compressed packets pre-authentication,
> then case 2 isn't mitigated. (I haven't proven how risky that case is
> yet, to be clear.) In other words: if the threat model is that a
> client can attack us, we should
On Tue, May 21, 2024 at 9:14 AM Jacob Burroughs
wrote:
> On Tue, May 21, 2024 at 10:43 AM Jelte Fennema-Nio wrote:
> > To help get everyone on the same page I wanted to list all the
> > security concerns in one place:
> >
> > 1. Triggering excessive CPU usage before authentication, by asking for
On Tue, May 21, 2024 at 8:23 AM Jacob Burroughs
wrote:
> As currently implemented, the compression only applies to
> CopyData/DataRow/Query messages, none of which should be involved in
> authentication, unless I've really missed something in my
> understanding.
Right, but Robert has argued that
On Tue, May 21, 2024 at 10:43 AM Jelte Fennema-Nio wrote:
>
> To help get everyone on the same page I wanted to list all the
> security concerns in one place:
>
> 1. Triggering excessive CPU usage before authentication, by asking for
> very high compression levels
> 2. Triggering excessive memory/
On Mon, 20 May 2024 at 21:42, Jacob Champion
wrote:
> As Andrey points out, there was prior work done that started to take
> this into account. I haven't reviewed it to see how good it is -- and
> I think there are probably many use cases in which queries and tables
> contain both private and atta
On Mon, May 20, 2024 at 2:42 PM Jacob Champion
wrote:
>
> I mean... you said it, not me. I'm trying not to rain on the parade
> too much, because compression is clearly very valuable. But it makes
> me really uncomfortable that we're reintroducing the compression
> oracle (especially over the auth
On Mon, May 20, 2024 at 4:12 PM Magnus Hagander wrote:
> That used to be the case in HTTP/1. But header compression was one of the
> headline features of HTTP/2, which isn't exactly new anymore. But there's a
> special algorithm, HPACK, for it. And then http/3 uses QPACK. Cloudflare has
> a pre
On Mon, May 20, 2024 at 9:09 PM Andrey M. Borodin
wrote:
>
>
>
> > On 20 May 2024, at 23:37, Robert Haas wrote:
> >
> > But, does this mean that we should just refuse to offer compression as
> > a feature?
>
> No, absolutely, we need the feature.
>
> > I guess I don't understand why TLS removed
On Mon, May 20, 2024 at 11:37 AM Robert Haas wrote:
> But if that's a practical
> attack, preventing compression prior to the authentication exchange
> probably isn't good enough
I mean... you said it, not me. I'm trying not to rain on the parade
too much, because compression is clearly very valu
On Mon, May 20, 2024 at 11:05 AM Andrey M. Borodin wrote:
> > So, the data would be compressed first, with framing around that, and
> > then transport encryption would happen afterwards. I don't see how
> > that would leak your password, but I have a feeling that might be a
> > sign that I'm about
> On 20 May 2024, at 23:37, Robert Haas wrote:
>
> But if that's a practical
> attack, preventing compression prior to the authentication exchange
> probably isn't good enough: the user could also try to guess what
> queries are being sent on behalf of other users through the same
> pooled c
On Mon, May 20, 2024 at 2:05 PM Andrey M. Borodin wrote:
> Compression defeats encryption. That's why it's not in TLS anymore.
> The thing is compression codecs use data self correlation. And if you mix
> secret data with user's data, user might guess how correlated they are.
Yeah, I'm aware tha
> On 20 May 2024, at 22:48, Robert Haas wrote:
>
> On Mon, May 20, 2024 at 1:23 PM Jacob Champion
> wrote:
>> On Mon, May 20, 2024 at 10:01 AM Robert Haas wrote:
>>> I really hope that you can't poke big enough holes to kill the feature
>>> entirely, though. Because that sounds sad.
>>
>> E
On Mon, May 20, 2024 at 1:23 PM Jacob Champion
wrote:
> On Mon, May 20, 2024 at 10:01 AM Robert Haas wrote:
> > I really hope that you can't poke big enough holes to kill the feature
> > entirely, though. Because that sounds sad.
>
> Even if there are holes, I don't think the situation's going to
On Mon, May 20, 2024 at 10:01 AM Robert Haas wrote:
> I really hope that you can't poke big enough holes to kill the feature
> entirely, though. Because that sounds sad.
Even if there are holes, I don't think the situation's going to be bad
enough to tank everything; otherwise no one would be abl
On Mon, May 20, 2024 at 12:49 PM Jacob Champion
wrote:
> ...and my response was that, no, the proposal doesn't seem to be
> requiring that authentication take place before compression is done.
> (As evidenced by your email. :D) If the claim is that there are no
> security problems with letting una
On Mon, May 20, 2024 at 8:29 AM Robert Haas wrote:
> It does occur to me that if some compression algorithm has a buffer
> overrun bug, restricting its use until after authentication might
> reduce the score of the resulting CVE, because now you have to be able
> to authenticate to make an exploit
On Mon, May 20, 2024 at 10:15 AM Jacob Champion
wrote:
> This is just restating the "you can already do bad things" argument. I
> understand that if your query gets executed, it's going to consume
> resources on the thing that's executing it (for the record, though,
> there are people working on c
On Sat, May 18, 2024 at 1:18 AM Jacob Burroughs
wrote:
> I like this more than what I proposed, and will update the patches to
> reflect this proposal. (I've gotten them locally back into a state of
> applying cleanly and dealing with the changes needed to support direct
> SSL connections, so refa
On Fri, May 17, 2024 at 4:03 PM Jelte Fennema-Nio wrote:
>
> On Fri, 17 May 2024 at 23:10, Jacob Champion
> wrote:
> > We're talking about a transport-level option, though -- I thought the
> > proposal enabled compression before authentication completed? Or has
> > that changed?
>
> I think it wo
On Fri, May 17, 2024 at 11:40 AM Robert Haas wrote:
>
> To be clear, I am not arguing that it should be the receiver's choice.
> I'm arguing it should be the client's choice, which means the client
> decides what it sends and also tells the server what to send to it.
> I'm open to counter-argument
On Fri, 17 May 2024 at 23:10, Jacob Champion
wrote:
> We're talking about a transport-level option, though -- I thought the
> proposal enabled compression before authentication completed? Or has
> that changed?
I think it would make sense to only compress messages after
authentication has complet
On Fri, 17 May 2024 at 23:40, Robert Haas wrote:
> To be clear, I am not arguing that it should be the receiver's choice.
> I'm arguing it should be the client's choice, which means the client
> decides what it sends and also tells the server what to send to it.
> I'm open to counter-arguments, bu
On Fri, May 17, 2024 at 4:53 PM Jacob Burroughs
wrote:
> I think I was more thinking that trying to let both parties control
> the parameter seemed like a recipe for confusion and sadness, and so
> the choice that felt most natural to me was to let the sender control
> it, but I'm definitely open
On Fri, May 17, 2024 at 1:53 PM Jacob Burroughs
wrote:
> New proposal, predicated on the assumption that if you enable
> compression you are ok with the client picking whatever level they
> want. At least with the currently enabled algorithms I don't think
> any of them are so insane that they wo
On Thu, May 16, 2024 at 3:28 AM Robert Haas wrote:
>
> Well, I mean, I don't really know what the right answer is here, but
> right now I can say pg_dump --compress=gzip to compress the dump with
> gzip, or pg_dump --compress=gzip:9 to compress with gzip level 9. Now,
> say that instead of compres
On Wed, May 15, 2024 at 12:50 PM Jacob Burroughs
wrote:
> I think I would agree with that. That said, I don't think the client
> should be in the business of specifying what configuration of the
> compression algorithm the server should use. The server administrator
> (or really most of the time
On Wed, May 15, 2024 at 11:31 AM Robert Haas wrote:
> From my point of view, it's the client who knows what it wants to do
> with the connection. If the client plans to read a lot of data, it
> might want the server to compress that data, especially if it knows
> that it's on a slow link. If the c
On Wed, May 15, 2024 at 12:24 PM Jacob Burroughs
wrote:
> > But now I'm wondering whether these options should really be symmetric
> > on the client and server sides? Isn't it for the server just to
> > specify a list of acceptable algorithms, and the client to set the
> > compression options? If
On Wed, May 15, 2024 at 8:38 AM Robert Haas wrote:
>
> I agree with that goal, but I'm somewhat confused by how your proposal
> achieves it. You had
> libpq_compression=lzma:client_to_server=off;gzip:server_to_client=off,
> so how do we parse that? Is that two completely separate
> specifications,
On Tue, May 14, 2024 at 5:21 PM Jacob Burroughs
wrote:
> The reason for both the semicolons and for not doing this is related
> to using the same specification structure as here:
> https://www.postgresql.org/docs/current/app-pgbasebackup.html
> (specifically the --compress argument).
I agree with
On Tue, May 14, 2024 at 3:22 PM Jacob Burroughs
wrote:
> What if we went with:
> Server side:
> * `libpq_compression=on` (I just want everything the server supports
> available; probably the most common case)
> * `libpq_compression=off` (I don't want any compression ever with this server)
> * `lib
On Tue, May 14, 2024 at 1:35 PM Robert Haas wrote:
>
> Well, in my last response before the thread died, I complained that
> libpq_compression=gzip:compress=off was confusing, and I stand by
> that, because "compress" is used both in the name of the parameter and
> as an option within the value of
On Tue, May 14, 2024 at 12:30 PM Jacob Burroughs
wrote:
> I've withdrawn this patch from the commitfest. I had been waiting for
> some resolution on "Add new protocol message to change GUCs for usage
> with future protocol-only GUCs" before I rebased/refactored this one,
> because this would be i
On Tue, May 14, 2024 at 11:08 AM Robert Haas wrote:
>
> According to https://commitfest.postgresql.org/48/4746/ this patch set
> needs review, but:
>
> 1. Considering that there have been no updates for 5 months, maybe
> it's actually dead?
I've withdrawn this patch from the commitfest. I had be
On Fri, Jan 12, 2024 at 4:11 PM Robert Haas wrote:
> I think that would definitely be better than "compress" and
> "decompress," but I was worried that it might be unclear to the user
> whether the parameter that they specified was from the point of view
> of the client or the server. Perhaps that
On Fri, Jan 12, 2024 at 4:02 PM Jacob Burroughs
wrote:
> > I wonder if we could use "upstream" and "downstream" to be clearer? Or
> > some other terminology?
>
> What about `send` and `receive`?
I think that would definitely be better than "compress" and
"decompress," but I was worried that it mi
> I wonder if we could use "upstream" and "downstream" to be clearer? Or
> some other terminology?
What about `send` and `receive`?
On Sun, Dec 31, 2023 at 2:32 AM Jacob Burroughs
wrote:
> I ended up reworking this to use a version of this option in place of
> the `none` hackery, but naming the parameters `compress` and
> `decompress, so to disable compression but allow decompression you
> would specify `libpq_compression=gzip
On Fri, 29 Dec 2023 at 11:02, Andrey M. Borodin wrote:
> This patchset allows sending CompressionMethod message, which allows to set
> another codec\level picked from the set of negotiated codec sets (during
> startup).
Did you mean to attach a patchset? I don't see the CompressionMethod
messag
> On 21 Dec 2023, at 05:30, Jelte Fennema-Nio wrote:
>
> One thing I'm wondering: should it be possible for the client to change the
> compression it wants mid-connection?
This patchset allows sending CompressionMethod message, which allows to set
another codec\level picked from the set of
Thanks for working on this!
One thing I'm wondering: should it be possible for the client to change the
compression it wants mid-connection? I can think of some scenarios where
that would be useful to connection poolers: if a pooler does plain
forwarding of the compressed messages, then it would n
> I'm having difficulty understanding the details of this handshaking
> algorithm from this description. It seems good that the handshake
> proceeds in each direction somewhat separately from the other, but I
> don't quite understand how the whole thing fits together. If the
> client tells the serv
On Tue, Dec 19, 2023 at 11:41 AM Jacob Burroughs
wrote:
> The compression "handshaking" process now looks as follows:
> 1. Client sends startup packet with `_pq_.libpq_compression = alg1;alg2`
> 2. At this point, the server can immediately begin compressing packets
> to the client with any of the
> I believe this patch series is ready for detailed review/testing, with one
> caveat: as can be seen here https://cirrus-ci.com/build/6732518292979712 ,
> the build is passing on all platforms and all tests except for the primary
> SSL test on Windows.
One correction: I apparently missed a ker
> On 10 Aug 2023, at 19:47, Jonah H. Harris wrote:
>
> Pinging to see if anyone has continued to work on this behind-the-scenes or
> whether this is the latest patch set there is.
It's still on my TODO list, but I haven't done much review cycles yet. And the
patch series already needs heavy
Pinging to see if anyone has continued to work on this behind-the-scenes or
whether this is the latest patch set there is.
--
Jonah H. Harris
On 18.11.22 02:07, Andrey Borodin wrote:
2. This literal
{no_compression_name}
should be replaced by explicit form
{no_compression_name, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL}
That doesn't seem better.
On Thu, Nov 17, 2022 at 7:09 AM Peter Eisentraut
wrote:
>
> Note that the above code was just changed in dce92e59b1.
Thanks!
> I don't know
> how that affects this patch set.
With dce92e59b1 it would be much easier to find a bug in the compression patch.
Some more notes about the patch. (sorry f
On 17.11.22 01:27, Andrey Borodin wrote:
Also I've found one more TODO item for the patch. Currently in
fe-connect.c patch is doing buffer reset:
conn->inStart = conn->inCursor = conn->inEnd = 0;
This effectively consumes butes up tu current cursor. However, packet
inspection is continued. The pa
On Tue, Nov 15, 2022 at 7:17 PM Justin Pryzby wrote:
>
Also I've found one more TODO item for the patch. Currently in
fe-connect.c patch is doing buffer reset:
conn->inStart = conn->inCursor = conn->inEnd = 0;
This effectively consumes butes up tu current cursor. However, packet
inspection is con
On Mon, Nov 14, 2022 at 07:44:24PM -0800, Andrey Borodin wrote:
> patchset needs a heavy rebase. If no one shows up to fix it I'll do
Despite what its git timestamp says, this is based on the most recent
patch from January, which I've had floating around since then. It
needed to be rebased over a
On Sat, Nov 12, 2022 at 8:04 PM Andrey Borodin wrote:
>
> While testing patch some more I observe unpleasant segfaults:
>
> #27 0x0b08fda2 in lz4_decompress (d_stream=0x18cf82a0,
> src=0x7feae4fa505d, src_size=92,
> (gdb) select-frame 27
> (gdb) info locals
> ds = 0x18cf82a0
> decPtr = 0x1
On Sat, Nov 12, 2022 at 1:47 PM Andrey Borodin wrote:
>
> I've tried the patch, it works as advertised.
While testing patch some more I observe unpleasant segfaults:
#26 0x7fecafa1e058 in __memcpy_ssse3_back () from target:/lib64/libc.so.6
#27 0x0b08fda2 in lz4_decompress (d_stream=0
On Tue, Aug 2, 2022 at 1:53 PM Jacob Champion wrote:
>
> and changing the status to "Needs Review"
I've tried the patch, it works as advertised. The code generally is
OK, maybe some functions require comments (because at least their
neighbours have some).
Some linkers complain about zs_is_valid_i
This entry has been waiting on author input for a while (our current
threshold is roughly two weeks), so I've marked it Returned with
Feedback.
Once you think the patchset is ready for review again, you (or any
interested party) can resurrect the patch entry by visiting
https://commitfest.pos
Ok, thanks
> On 3 Mar 2022, at 02:33, Justin Pryzby wrote:
>
> If there's no objection, I'd like to move this to the next CF for
> consideration
> in PG16.
>
> On Mon, Jan 17, 2022 at 10:39:19PM -0600, Justin Pryzby wrote:
>> On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
If there's no objection, I'd like to move this to the next CF for consideration
in PG16.
On Mon, Jan 17, 2022 at 10:39:19PM -0600, Justin Pryzby wrote:
> On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
> > > => Since March, errmsg doesn't need extra parenthesis around it (e3a87b
On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
> > => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
> I’ve resolved the stuck tests and added zlib support for CI Windows builds to
> patch 0003-*. Thanks
> for the suggestion, all tests seem to be OK n
On Fri, Jan 14, 2022 at 02:12:17AM +0500, Daniil Zakhlystov wrote:
> Hi, Justin!
>
> First of all, thanks for the detailed review. I’ve applied your patches to
> the current version.
Note that my message had other comments that weren't addressed in this patch.
Your 0003 patch has a couple "nois
zlib still causes check-world to get stuck. I first mentioned this last March:
20210319062800.gi11...@telsasoft.com
Actually all the compression methods seems to get stuck with
time make check -C src/bin/pg_rewind
time make check -C src/test/isolation
For CI purposes, there should be an 0003 pat
> 8 янв. 2022 г., в 01:56, Stephen Frost написал(а):
>>
>> It's discussed in last year's thread. The thinking is that there tends to be
>> *fewer* exploitable opportunities between application->DB than between
>> browser->app.
>
> Yes, this was discussed previously and addressed.
What else
Greetings,
* Bruce Momjian (br...@momjian.us) wrote:
> On Fri, Jan 7, 2022 at 03:56:39PM -0500, Stephen Frost wrote:
> > As for the details of how we allow control over it, I suppose there's a
> > number of options. Having it in the HBA doesn't seem terrible, though I
> > suspect most will just
On Fri, Jan 7, 2022 at 03:56:39PM -0500, Stephen Frost wrote:
> As for the details of how we allow control over it, I suppose there's a
> number of options. Having it in the HBA doesn't seem terrible, though I
> suspect most will just want to enable it across the board and having to
> have "compr
Greetings,
* Justin Pryzby (pry...@telsasoft.com) wrote:
> On Fri, Jan 07, 2022 at 01:46:00PM -0500, Bruce Momjian wrote:
> > On Sat, Jan 1, 2022 at 11:25:05AM -0600, Justin Pryzby wrote:
> > > Thanks for working on this. The patch looks to be in good shape - I hope
> > > more
> > > people will
On Fri, Jan 07, 2022 at 01:46:00PM -0500, Bruce Momjian wrote:
> On Sat, Jan 1, 2022 at 11:25:05AM -0600, Justin Pryzby wrote:
> > Thanks for working on this. The patch looks to be in good shape - I hope
> > more
> > people will help to review and test it. I took the liberty of creating a
> >
On Sat, Jan 1, 2022 at 11:25:05AM -0600, Justin Pryzby wrote:
> Thanks for working on this. The patch looks to be in good shape - I hope more
> people will help to review and test it. I took the liberty of creating a new
> CF entry. http://cfbot.cputube.org/daniil-zakhlystov.html
>
> +zpq_shoul
> Maybe you should reset the streams between each compression message (even if
> it's using the same compression algorithm). This might allow better
> compression.
AFAIK on the contrary - longer data sequence usually compresses better. The
codec can use knowledge about prior data to better comp
On Fri, Oct 01, 2021 at 11:20:09PM +0200, Daniel Gustafsson wrote:
> To keep this thread from stalling, attached is a rebased patchset with the
> above mentioned fix to try and get this working on Windows.
This patch has been waiting on author for two months now, so I have
marked it as RwF in the
Hi, thanks for your fix! I am currently working on implementing the lz4 support for libpq compression. Looking forward to post an update soon.—Daniil ZakhlystovOn 2 Oct 2021, at 02:20, Daniel Gustafsson wrote:On 2 Sep 2021, at 00:29, Daniel Gustafsson wrote:On 29 Jul 2021, at 16:57, Daniil Zakhly
> On 29 Jul 2021, at 16:57, Daniil Zakhlystov wrote:
>
> Forgot to attach the updated patch :)
This fails to build on Windows due to the use of strcasecmp:
+ if (strcasecmp(supported_algorithms[zpq->compressors[i].impl],
"zstd") ==
Was that meant to be pg_strcasecmp?
--
Danie
Hi!
I made some noticeable changes to the patch and fixed the previously mentioned
issues.
On Fri, Mar 19, 2021 at 16:28 AM Justin Pryzby wrote:
> Previously, I suggested that the server should have a "policy" GUC defining
> which compression methods are allowed. Possibly including compression
On Wed, Jul 14, 2021 at 6:31 PM Daniil Zakhlystov
wrote:
>
> **sorry for the noise, but I need to re-send the message because one of the
> recipients is blocked on the pgsql-hackers for some reason**
>
> Hi!
>
> Done, the patch should apply to the current master now.
>
> Actually, I have an almos
One little fix to 0001-Add-zlib-and-zstd-streaming-compression patch for
configure.ac
@@ -1455,6 +1456,7 @@ fi
if test "$with_lz4" = yes; then
AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required
for LZ4])])
+fi
Otherwise autoconf generat
All, thanks!
On Wed, Apr 21, 2021 at 10:47 AM Michael Paquier
wrote:
> Patch reviews had better be posted on the community lists. This way,
> if the patch is left dead by the authors (things happen in life), then
> somebody else could move on with the patch without having to worry
> about this
On Wed, Apr 21, 2021 at 09:08:09AM +, Ian Zagorskikh wrote:
> I took a look at proposed patches and found several typos/mistakes. Where
> should I send my comments? In this thread or directly to the authors?
Patch reviews had better be posted on the community lists. This way,
if the patch is
On Wed, Apr 21, 2021 at 2:38 PM Ian Zagorskikh
wrote:
>
> Hi all!
>
> I took a look at proposed patches and found several typos/mistakes. Where
> should I send my comments? In this thread or directly to the authors?
>
I feel it is good to send comments here. This is what we normally do
for all t
Hi all!
I took a look at proposed patches and found several typos/mistakes. Where
should I send my comments? In this thread or directly to the authors?
Thanks!
On Wed, Apr 21, 2021 at 9:03 AM Daniil Zakhlystov
wrote:
> Hi, thanks for your review!
>
> > On Mar 19, 2021, at 11:28 AM, Justin Pry
On Thu, Mar 18, 2021 at 08:02:32PM -0500, Justin Pryzby wrote:
> On Thu, Mar 18, 2021 at 07:30:09PM +, Daniil Zakhlystov wrote:
> > The new status of this patch is: Ready for Committer
>
> The CF bot is failing , because the last patch sent to the list is from
> January:
> | Latest attachment
On Thu, Mar 18, 2021 at 07:30:09PM +, Daniil Zakhlystov wrote:
> The new status of this patch is: Ready for Committer
The CF bot is failing , because the last patch sent to the list is from January:
| Latest attachment (libpq-compression-31.patch) at 2021-01-12 14:05:22 from
Konstantin Knizhn
The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation:tested, passed
Hi,
I've compared the different libpq compression approaches
> 26 февр. 2021 г., в 02:18, Daniil Zakhlystov
> написал(а):
>
> Yes, there is still no possibility for compressed traffic pass-through for
> poolers,
> but do we actually need it?
> I don’t see any solution except starting a new compression context for
> each message in order to make it work
Hi, thanks for your review,
> On Feb 22, 2021, at 10:38 AM, Craig Ringer
> wrote:
>
>
> On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov,
> wrote::
>
> 3. Chunked compression allows to compress only well compressible messages and
> save the CPU cycles by not compressing the others
> 4. Chunked
On 22.02.2021 08:38, Craig Ringer wrote:
On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov,
mailto:usernam...@yandex-team.ru>> wrote::
3. Chunked compression allows to compress only well compressible
messages and save the CPU cycles by not compressing the others
4. Chunked compress
On 22.02.2021 23:44, Tom Lane wrote:
Robert Haas writes:
So at the end of the day I'm not really quite sure what is best here.
I agree with all of Craig's points about the advantages of
packet-level compression, so I'd really prefer to make that approach
work if we can. However, it also seem
> On 22 Feb 2021, at 22:21, Andres Freund wrote:
>
> Hi,
>
> On 2021-02-22 15:44:30 -0500, Tom Lane wrote:
>> Robert Haas writes:
>>> So at the end of the day I'm not really quite sure what is best here.
>>> I agree with all of Craig's points about the advantages of
>>> packet-level compression
Hi,
On 2021-02-22 15:44:30 -0500, Tom Lane wrote:
> Robert Haas writes:
> > So at the end of the day I'm not really quite sure what is best here.
> > I agree with all of Craig's points about the advantages of
> > packet-level compression, so I'd really prefer to make that approach
> > work if we
Robert Haas writes:
> So at the end of the day I'm not really quite sure what is best here.
> I agree with all of Craig's points about the advantages of
> packet-level compression, so I'd really prefer to make that approach
> work if we can. However, it also seems to me that there's room to be
> f
Hi,
On 2021-02-22 14:48:25 -0500, Robert Haas wrote:
> So, if I read these results correctly, on the "pg_restore of IMDB
> database" test, we get 88% of the RX bytes reduction and 99.8% of the
> TX bytes reduction for 90% of the CPU cost. On the "pgbench" test,
> which probably has much smaller pa
On Thu, Feb 11, 2021 at 8:09 AM Daniil Zakhlystov
wrote:
> [ benchmark results ]
So, if I read these results correctly, on the "pg_restore of IMDB
database" test, we get 88% of the RX bytes reduction and 99.8% of the
TX bytes reduction for 90% of the CPU cost. On the "pgbench" test,
which probabl
On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov,
wrote::
>
> 3. Chunked compression allows to compress only well compressible messages
> and save the CPU cycles by not compressing the others
> 4. Chunked compression introduces some traffic overhead compared to the
> permanent (1.2810G vs 1.2761G TX
On 11.02.2021 16:09, Daniil Zakhlystov wrote:
Hi!
On 09.02.2021 09:06, Konstantin Knizhnik wrote:
Sorry, but my interpretation of your results is completely different:
permanent compression is faster than chunked compression (2m15 vs. 2m27)
and consumes less CPU (44 vs 48 sec).
Size of RX d
Hi!
> On 09.02.2021 09:06, Konstantin Knizhnik wrote:
>
> Sorry, but my interpretation of your results is completely different:
> permanent compression is faster than chunked compression (2m15 vs. 2m27)
> and consumes less CPU (44 vs 48 sec).
> Size of RX data is slightly larger - 0.5Mb but TX s
On 08.02.2021 22:23, Daniil Zakhlystov wrote:
Hi everyone,
I’ve been making some experiments with an on-the-fly compression switch lately
and have some updates.
...
pg_restore of IMDB database test results
Chunked compression with only CopyData or DataRow compression (second approach):
tim
Hi everyone,
I’ve been making some experiments with an on-the-fly compression switch lately
and have some updates.
> On Dec 22, 2020, at 10:42 PM, Robert Haas wrote:
>
>
> Hmm, I assumed that if the compression buffers were flushed on the
> sending side, and if all the data produced on the
> 12 янв. 2021 г., в 20:47, Konstantin Knizhnik
> написал(а):
>
>> I think we should come up with an minimal, prelimininary 0001 patch which is
>> common between the 3 compression patches (or at least the two using zstd).
>> The
>> ./configure changes and a compressionlibs struct would also
On 12.01.2021 18:38, Justin Pryzby wrote:
On Tue, Jan 12, 2021 at 08:44:43AM +0300, Konstantin Knizhnik wrote:
On 11.01.2021 20:38, Tomas Vondra wrote:
1) Fixes the MSVC makefile. The list of files is sorted alphabetically,
so I've added the file at the end.
Thank you
This is still failing
On Tue, Jan 12, 2021 at 08:44:43AM +0300, Konstantin Knizhnik wrote:
> On 11.01.2021 20:38, Tomas Vondra wrote:
> > 1) Fixes the MSVC makefile. The list of files is sorted alphabetically,
> > so I've added the file at the end.
> Thank you
This is still failing the windows build.
I think you need
1 - 100 of 270 matches
Mail list logo
