On 09.02.2019 1:38, Tomas Vondra wrote:
On 2/8/19 11:10 PM, Konstantin Knizhnik wrote:
On 08.02.2019 21:57, Andres Freund wrote:
On 2019-02-08 12:15:58 +0300, Konstantin Knizhnik wrote:
Frankly speaking, I do not think that such flexibility in choosing
compression algorithms is really needed.
I do not expect that there will be many situations where old client
has to
communicate with new server or visa versa.
In most cases both client and server belongs to the same postgres
distributive and so implements the same compression algorithm.
As far as we are compressing only temporary data (traffic), the
problem of
providing backward compatibility seems to be not so important.
I think we should outright reject any patch without compression type
negotiation.
Does it mean that it is necessary to support multiple compression
algorithms and make it possible to perform switch between them at
runtime?
IMHO the negotiation should happen at connection time, i.e. the server
should support connections compressed by different algorithms. Not sure
if that's what you mean by runtime.
AFAICS this is quite close to how negotiation of encryption algorithms
works, in TLS and so on. Client specifies supported algorithms, server
compares that to list of supported algorithms, deduces the encryption
algorithm and notifies the client.
To allow fall-back to uncompressed connection, use "none" as algorithm.
If there's no common algorithm, fail.
It is good analogue with SSL.
Yes, SSL protocol provides several ways of authentication, encryption,...
And there are several different libraries implementing SSL.
But Postgres is using only one of them: OpenSSL.
If I want to use some other library (for example to make it possible to
serialize and pass SSL session state to other
process), then there is no way to achieve it.
Actually zstd also includes implementations of several compression
algorithms and it choose one of them best fitting particular data
stream. As in case of SSL, choice of algorithm is performed internally
inside zstd - not at libpq level.
Sorry, if my explanation about static and dynamic (at runtime) choice
were not correct.
This is how compression is toggled now:
#if HAVE_LIBZSTD
ZpqStream*
zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
{
...
}
#endif
So if Postgres was configured with zstd, then this implementation is
included inclient and server Postgres libraries.
If postgres is configures with zlib, them zlib implementation will be used.
This is similar with using compression and most of other configurable
features in Postgres.
If we want to provide dynamic choice at runtime, then we need to have
array with available compression algorithms:
#if HAVE_LIBZSTD
static ZpqStream*
zstd_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
{
...
}
#endif
ZpqCompressorImpl compressorImpl[] =
{
#if HAVE_LIBZSTD
{zstd_create, zstd_read,zstd_write,...},
#endif
#if HAVE_ZLIB
{zlib_create, zlib_read,zslib_write,...},
#endif
...
}
And the most interesting case is that if we load library dynamically.
Each implementation is generated in separate library (for example
libpztd.so).
In this case we need to somehow specify available libraries.
For example by placing them in separate directory, or specifying list of
libraries in postgresql.conf.
Then we try to load this library using dlopen. Such library has
external dependencies of correspondent compressor library (for example
-lz). The library can be successfully loaded if there correspond
compressor implementation was install at the system.
This is most flexible approach allowing to provide custom implementation
of compressors.
Compression implementation can be organized as Postgres extension and
its PG_init function registers this implementation in some list.
This is what I am asking about.
Right now approach 1) is implemented: compression algorithm is defined
by configure.
It is no so difficult to extend it to support multiple algorithms.
And the most flexible but more sophisticated is to load libraries
dynamically.
Right now compression algorithm is linked statically.
Negotiation of compression type is currently performed but it only
checks that server and client are implementing the same algorithm and
disables compression if it is not true.
I don't think we should automatically fall-back to disabled compression,
when a client specifies compression algorithm.
Compression is disabled only when client and server were configured with
different compression algorithms (i.e. zstd and zlib).
If we are going to support multiple compression algorithms, do we need
dynamic loading of correspondent compression libraries or static linking
is ok? In case of dynamic linking we need to somehow specify information
about available compression algorithms.
Some special subdirectory for them so that I can traverse this directory
and try to load correspondent libraries?
Only I find it too complicated for the addressed problem?
I don't think we need dynamic algorithms v1, but IMHO it'd be pretty
simple to do - just add a shared_preload_library which registers it in a
list in memory.
I do not think that it is necessary to include such libraries in
preload_shared_libraries list.
It can be done lazily only of compression is requested by client.
Also please notice that we need to load compression library both at
server and client sides.
preload_shared_libraries works only for postmaster.
