Re: Postgres Logical Replication - how to see what subscriber is doing with received data?

2024-09-02 Thread Shaheed Haque
Hi Muhammad, On Mon, 2 Sep 2024, 07:08 Muhammad Ikram, wrote: > Hi Shaheed, > > Maybe these considerations could help you or give any hint to the problem ? > > > Check if wal_receiver_timeout being set to 0 could potentially cause > issues, like not detecting network issues quickly enough. Consi

Re: Postgres Logical Replication - how to see what subscriber is doing with received data?

2024-09-02 Thread Muhammad Ikram
Hi Shaheed, I think you must have already analyzed the outcome of queries on pg_replication_slots, pg_current_wal_lsn(), pg_stat_subscription etc. I could find a query SELECT pg_size_pretty(pg_wal_lsn_diff('', '')); As a side note if you want to see what has been applied to subscribers vs what ex

Re: Postgres Logical Replication - how to see what subscriber is doing with received data?

2024-09-02 Thread Shaheed Haque
Hi Muhammad, On Mon, 2 Sep 2024, 09:45 Muhammad Ikram, wrote: > Hi Shaheed, > I think you must have already analyzed the outcome of queries > on pg_replication_slots, pg_current_wal_lsn(), pg_stat_subscription etc. I > could find a query SELECT > pg_size_pretty(pg_wal_lsn_diff('', > '')); > Ye

Re: Partitioning and unique key

2024-09-02 Thread Laurenz Albe
On Sun, 2024-09-01 at 01:32 +0530, veem v wrote: > due to postgres limitations we are unable to have this unique constraint or > primary key > only on the transaction_id column, we have to include transaction_timestamp > with it as > a composite key. So I want to understand from experts if there

Re: PG17 optimizations to vacuum

2024-09-02 Thread Peter Geoghegan
On Mon, Sep 2, 2024 at 4:58 PM Peter Geoghegan wrote: > On Mon, Sep 2, 2024 at 4:35 PM Pavel Luzanov wrote: > > If it helps, without creating index on id column, the numbers will be > > much closer: > > Yes, avoiding all index vacuuming seems useful. It makes the test case > cleaner, since we don

Re: PG17 optimizations to vacuum

2024-09-02 Thread Heikki Linnakangas
On 03/09/2024 00:11, Heikki Linnakangas wrote: On 03/09/2024 00:01, Peter Geoghegan wrote: On Mon, Sep 2, 2024 at 4:58 PM Heikki Linnakangas wrote: Do you have any non-default settings? "select name, current_setting(name), source  from pg_settings where setting <> boot_val;" would show that.

Re: Partitioning and unique key

2024-09-02 Thread veem v
On Mon, 2 Sept 2024 at 19:13, Laurenz Albe wrote: > On Sun, 2024-09-01 at 01:32 +0530, veem v wrote: > > due to postgres limitations we are unable to have this unique constraint > or primary key > > only on the transaction_id column, we have to include > transaction_timestamp with it as > > a com

Re: PG17 optimizations to vacuum

2024-09-02 Thread Melanie Plageman
On Sun, Sep 1, 2024 at 6:00 PM Peter Geoghegan wrote: > > On Sun, Sep 1, 2024 at 5:44 PM Pavel Luzanov wrote: > > I see a perfectly working TID-store optimization. > > With reduced maintenance_work_mem it used only one 'vacuuming indexes' > > phase instead of 21 in v16. > > But I also expected to

Re: PG17 optimizations to vacuum

2024-09-02 Thread Peter Geoghegan
On Mon, Sep 2, 2024 at 1:29 PM Melanie Plageman wrote: > I'll investigate more tomorrow, but based on my initial investigation, > there appears to be some interaction related to how much of the > relation is in shared buffers after creating the table and updating > it. If you set shared_buffers su

Re: PG17 optimizations to vacuum

2024-09-02 Thread Melanie Plageman
On Mon, Sep 2, 2024 at 1:47 PM Peter Geoghegan wrote: > > On Mon, Sep 2, 2024 at 1:29 PM Melanie Plageman > wrote: > > I'll investigate more tomorrow, but based on my initial investigation, > > there appears to be some interaction related to how much of the > > relation is in shared buffers after

Re: PG17 optimizations to vacuum

2024-09-02 Thread Peter Geoghegan
On Mon, Sep 2, 2024 at 3:23 PM Melanie Plageman wrote: > This is roughly what I get for records by vacuum. Note that I prefixed > VACUUM with BTREE on master to indicate those records are from index > vacuuming. By default the headesc routine for records emitted by index > vacuuming prints just VA

Re: Partitioning and unique key

2024-09-02 Thread Laurenz Albe
On Mon, 2024-09-02 at 21:39 +0530, veem v wrote: > On Mon, 2 Sept 2024 at 19:13, Laurenz Albe wrote: > > On Sun, 2024-09-01 at 01:32 +0530, veem v wrote: > > > due to postgres limitations we are unable to have this unique constraint > > > or primary key > > > only on the transaction_id column, we

Re: PG17 optimizations to vacuum

2024-09-02 Thread Pavel Luzanov
On 02.09.2024 22:23, Melanie Plageman wrote: For some reason I stopped being able to reproduce Pavel's case. I repeated the test on another computer, but compared master with v15. The results are the same. The test can be simplified as follows: CREATE TABLE t(id integer) WITH (autovacuum_enabl

Re: PG17 optimizations to vacuum

2024-09-02 Thread Heikki Linnakangas
On 02/09/2024 23:35, Pavel Luzanov wrote: On 02.09.2024 22:23, Melanie Plageman wrote: For some reason I stopped being able to reproduce Pavel's case. I also cannot reproduce this. I repeated the test on another computer, but compared master with v15. The results are the same. The test can b

Re: PG17 optimizations to vacuum

2024-09-02 Thread Peter Geoghegan
On Mon, Sep 2, 2024 at 4:35 PM Pavel Luzanov wrote: > If it helps, without creating index on id column, the numbers will be > much closer: Yes, avoiding all index vacuuming seems useful. It makes the test case cleaner, since we don't have to think about the variability from the TIDStore work (and

Re: PG17 optimizations to vacuum

2024-09-02 Thread Peter Geoghegan
On Mon, Sep 2, 2024 at 4:58 PM Heikki Linnakangas wrote: > Do you have any non-default settings? "select name, > current_setting(name), source from pg_settings where setting <> > boot_val;" would show that. What about page checksums? One simple explanation is that we're writing extra FPIs to se

Re: PG17 optimizations to vacuum

2024-09-02 Thread Heikki Linnakangas
On 03/09/2024 00:01, Peter Geoghegan wrote: On Mon, Sep 2, 2024 at 4:58 PM Heikki Linnakangas wrote: Do you have any non-default settings? "select name, current_setting(name), source from pg_settings where setting <> boot_val;" would show that. What about page checksums? One simple explanat

Re: Partitioning and unique key

2024-09-02 Thread veem v
On Tue, 3 Sept 2024 at 01:14, Laurenz Albe wrote: > > You can keep the primary key defined on both columns if it is good enough > for you. > But it will give you lower guarantees of uniqueness: with that primary > key, there could > be two rows with a different timestamp, but the same "txn_id", a

Re: Partitioning and unique key

2024-09-02 Thread Laurenz Albe
On Tue, 2024-09-03 at 10:39 +0530, veem v wrote: > As you rightly said "they will make it more difficult to detach a partition." > , > we are really seeing a longer time when detaching parent table partitions. > It runs forever sometimes. So do you mean it's because we have primary key > defined t