On Mon, May 10, 2021 at 7:30 PM Justin Pryzby <pry...@telsasoft.com> wrote: > > Can these be merged: > Allow logical replication to stream long transactions to standbys (Dilip > Kumar, Tomas Vondra, Amit Kapila, Nikhil Sontakke) > Improve the logical replication API to allow streaming large in-progress > transactions (Tomas Vondra, Dilip Kumar, Amit Kapila) >
I think we should merge the below two: Add support for streaming to built-in logical replication (Vondra, Dilip Kumar, Amit Kapila, Ajin Cherian, Nikhil Sontakke, Stas Kelvich) Allow logical replication to stream long transactions to standbys (Dilip Kumar, Tomas Vondra, Amit Kapila, Nikhil Sontakke) and say something like: Allow logical replication to stream long in-progress transactions to standbys Previously transactions that exceeded logical_decoding_work_mem were written to disk until the transaction completed. Also, the commits that lead to the above feature is: 2020-09-03 [464824323] Add support for streaming to built-in logical replicatio 2020-08-08 [7259736a6] Implement streaming mode in ReorderBuffer. * Improve the logical replication API to allow streaming large in-progress transactions (Tomas Vondra, Dilip Kumar, Amit Kapila) The output functions begin with "stream". test_decoding also supports these. In the above, I think it is better to say Enhance instead of Improve. * Add system view pg_stat_replication_slots to report replication slot activity (Sawada Masahiko, Amit Kapila) I think for this item, we can give Vignesh C as author credit because he helped a lot enhance/stabilize this feature after the initial commit. See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=f5fc2f5b23d1b1dff60f8ca5dc211161df47eda4 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=cca57c1d9bf7eeba5b81115e0b82651cf3d8e4ea https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=592f00f8dec68038301467a904ac514eddabf6cd * Allow multiple xacts during table sync in logical replication (Peter Smith, Amit Kapila, and Takamichi Osumi) IMPORTANT? Yes, this is a big enough improvement in the initial table sync mechanism of logical replication that it is worth mentioning in Release Notes. One of the main user-visible effects this commit can have is that it will reduce the time for which we need to hold WAL on the server during the initial table sync phase. Then it can avoid large copies in case there is any error during the time when apply-worker and tablesync-worker synchronizes the apply location. * Enable logical replication to handle two phase commits (Ajin Cherian) This is controlled via pg_create_logical_replication_slot(). We still haven't supported two-phase commits for built-in logical replication but we have exposed APIs for decoding plugins to achieve that. Using these APIs, we are planning to support it for built-in logical replication but that will be probably done in PG-15. So, I suggest we can modify it to something like: * Enhance logical decoding APIs to handle two-phase commits This is controlled via pg_create_logical_replication_slot(). Most of the work for this feature is done via commits: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a271a1b50e9bec07e2ef3a05e38e7285113e4ce6 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=0aa8a01d04c8fe200b7a106878eebc3d0af9105c So, I think the author credit should be given to Ajin Cherian, Amit Kapila, Nikhil Sontakke, and Stas Kelvich * I think we should mention about commit: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d7eb52d7181d83cf2363570f7a205b8eb1008dbc This allows Logical decoding to work efficiently in presence of a large amount of DDLs. This was a complaint from many users for long time. Apart from the discussion thread, see some of the earlier complaints: https://www.postgresql.org/message-id/flat/20f3de7675f83176253f607b5e199b228406c21c.ca...@cybertec.at https://www.postgresql.org/message-id/flat/CAHoiPjzea6N0zuCi=+f9v_j94nfsy6y8SU7-=bp4=7qw6_i=r...@mail.gmail.com and most recently at: https://www.postgresql.org/message-id/DB8PR10MB36126BC0DC3570637FDFED12EB4F9%40DB8PR10MB3612.EURPRD10.PROD.OUTLOOK.COM * Speed truncation of small tables on large shared buffer servers (Kirk Jamison) This optimization applies during recovery. Shouldn't we mention that in some way? -- With Regards, Amit Kapila.
