On Tue, Jul 16, 2024 at 7:31 PM Tomas Vondra <tomas.von...@enterprisedb.com> wrote: > > On 7/16/24 14:52, Amit Kapila wrote: > > On Tue, Jul 16, 2024 at 12:58 AM Tomas Vondra > > <tomas.von...@enterprisedb.com> wrote: > >> > >> FWIW I'd expect that to be handled at the libpq level - there's already > >> a patch for that, but I haven't checked if it would handle this. But > >> maybe more importantly, I think compressing streamed data might need to > >> handle some sort of negotiation of the compression algorithm, which > >> seems fairly complex. > >> > >> To conclude, I'd leave this out of scope for this patch. > >> > > > > Your point sounds reasonable to me. OTOH, if we want to support > > compression for spill case then shouldn't there be a question how > > frequent such an option would be required? Users currently have an > > option to stream large transactions for parallel apply or otherwise in > > which case no spilling is required. I feel sooner or later we will > > make such behavior (streaming=parallel) as default, and then spilling > > should happen in very few cases. Is it worth adding this new option > > and GUC if that is true? > > > > I don't know, but streaming is 'off' by default, and I'm not aware of > any proposals to change this, so when you suggest "sooner or later" > we'll change this, I'd probably bet on "later or never". > > I haven't been following the discussions about parallel apply very > closely, but my impression from dealing with similar stuff in other > tools is that it's rather easy to run into issues with some workloads, > which just makes me more skeptical about "streamin=parallel" by default. > But as I said, I'm out of the loop so I may be wrong ... >
It is difficult to say whether enabling it by default will have issues or not but till now we haven't seen many reports for the streaming = 'parallel' option. It could be due to the reason that not many people enable it in their workloads. We can probably find out by enabling it by default. > As for whether the GUC is needed, I don't know. I guess we might do the > same thing we do for streaming - we don't have a GUC to enable this, but > we default to 'off' and the client has to request that when opening the > replication connection. So it'd be specified at the subscription level, > more or less. > > But then how would we specify compression for cases that invoke decoding > directly by pg_logical_slot_get_changes()? Through options? > If we decide to go with this then yeah that is one way, another possibility is to make it a slot's property, so we can allow to take a new parameter in pg_create_logical_replication_slot(). We can even think of inventing a new API to alter the slot's properties if we decide to go this route. > BTW if we specify this at subscription level, will it be possible to > change the compression method? > This needs analysis but offhand I can't see the problems with it. -- With Regards, Amit Kapila.
