On Tue, Dec 8, 2020 at 9:42 AM Daniil Zakhlystov <usernam...@yandex-team.ru> wrote: > I proposed a slightly different handshake (three-phase): > > 1. At first, the client sends _pq_.compression parameter in startup packet > 2. Server replies with CompressionAck and following it with > SetCompressionMethod message. > These two might be combined but I left them like this for symmetry reasons. > In most cases they > will arrive as one piece without any additional delay. > 3. Client replies with SetCompressionMethod message.
I think that's pretty similar to what I proposed, actually, except I think that SetCompressionMethod in one direction should be decoupled from SetCompressionMethod in the other direction, so that those things don't have to be synchronized with respect to each other. Each affects its own direction only. > Yes, there is actually some amount of complexity involved in implementing the > switchable on-the-fly compression. > Currently, compression itself operates on a different level, independently of > libpq protocol. By allowing > the compression to be switchable on the fly, we need to solve these tasks: > > 1. When the new portion of bytes comes to the decompressor from the > socket.read() call, there may be > a situation when the first part of these bytes is a compressed fragment and > the other is part is uncompressed, or worse, > in a single portion of new bytes, there may be the end of some ZLIB > compressed message and the beginning of the ZSTD compressed message. > The problem is that we don’t know the exact end of the ZLIB compressed > message before decompressing the entire chunk of new bytes > and reading the SetCompressionMethod message. Moreover, streaming compression > by itself may involve some internal buffering, > which also complexifies this problem. > > 2. When sending the new portion of bytes, it may be not sufficient to keep > track of only the current compression method. > There may be a situation when there could be multiple SetCompressionMessages > in PqSendBuffer (backend) or conn->outBuffer (frontend). > It means that it is not enough to simply track the current compression method > but also keep track of all compression method > switches in PqSendBuffer or conn->outBuffer. Also, same as for decompression, > internal buffering of streaming compression makes the situation more complex > in this case too. Good points. I guess you need to arrange to "flush" at the compression layer as well as the libpq layer so that you don't end up with data stuck in the compression buffers. Another idea is that you could have a new message type that says "hey, the payload of this is 1 or more compressed messages." It uses the most-recently set compression method. This would make switching compression methods easier since the SetCompressionMethod message itself could always be sent uncompressed and/or not take effect until the next compressed message. It also allows for a prudential decision not to bother compressing messages that are short anyway, which might be useful. On the downside it adds a little bit of overhead. Andres was telling me on a call that he liked this approach; I'm not sure if it's actually best, but have you considered this sort of approach? > I personally think that this approach is the most practical one. For example: > > In the server’s postgresql.conf: > > compress_algorithms = ‘uncompressed' // means that the server forbids any > server-to-client compression > decompress_algorithms = 'zstd:7,8;uncompressed' // means that the server can > only decompress zstd with compression ratio 7 and 8 or communicate with > uncompressed messages > > In the client connection string: > > “… compression=zlib:1,3,5;zstd:6,7,8;uncompressed …” // means that the client > is able to compress/decompress zlib, zstd, or communicate with uncompressed > messages > > For the sake of simplicity, the client’s “compression” parameter in the > connection string is basically an analog of the server’s compress_algorithms > and decompress_algorithms. > So the negotiation process for the above example would look like this: > > 1. Client sends startup packet with > “algorithms=zlib:1,3,5;zstd:6,7,8;uncompressed;” > Since there is no compression mode specified, assume that the client wants > permanent compression. > In future versions, the client can turn request the switchable compression > after the ‘;’ at the end of the message > > 2. Server replies with two messages: > - CompressionAck message containing “algorithms=zstd:7,8;uncompressed;” > Where the algorithms section basically matches the “decompress_algorithms” > server GUC parameter. > In future versions, the server can specify the chosen compression mode after > the ‘;’ at the end of the message > > - Following SetCompressionMethod message containing “alg_idx=1;level_idx=1” > which > essentially means that the server chose zstd with compression level 7 for > server-to-client compression. Every next message from the server is now > compressed with zstd > > 3. Client replies with SetCompressionMethod message containing “alg_idx=0” > which means that the client chose the uncompressed > client-to-server messaging. Actually, the client had no other options, > because the “uncompressed” was the only option left after the intersection of > compression algorithms from the connection string and algorithms received > from the server in the CompressionAck message. > Every next message from the client is now being sent uncompressed. I still think this is excessively baroque and basically useless. Nobody wants to allow compression levels 1, 3, and 5 but disallow 2 and 4. At the very most, somebody might want to start a maximum or minimum level. But even that I think is pretty pointless. Check out the "Decompression Time" and "Decompression Speed" sections from this link: https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/ This shows that decompression time and speed is basically independent of compression method for all three of these compressors; to the extent that there is a difference, higher compression levels are generally slightly faster to decompress. I don't really see the argument for letting either side be proscriptive here. Deciding with algorithms you're willing to accept is totally reasonable since different things may be supported, security concerns, etc. but deciding you're only willing to accept certain levels seems unuseful. It's also unenforceable, I think, since the receiving side has no way of knowing what the sender actually did. -- Robert Haas EDB: http://www.enterprisedb.com
