Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:20 AM Avinash Kumar wrote: > I can share any detail that would help here. I would like to know what you see when you run a slightly modified version of the same amcheck query. The same query as before, but with the call to bt_index_parent_check() replaced with a call to

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:08 AM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would run towards a bug in WAL log c

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
On Tue, Mar 16, 2021 at 3:08 PM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would run towards a bug in WAL log c

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Tom Lane
Peter Geoghegan writes: > ... It's hard to believe that the problem is > squarely with _bt_swap_posting(). IIUC, the problem is seen on a replica server but not the primary? In that case, my thoughts would run towards a bug in WAL log creation or replay, causing the index contents to be different

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 9:50 AM Avinash Kumar wrote: > Yes, it was on the failover-over server where the issue is currently seen. > Took a snapshot of the data directory so that the issue can be analyzed. I would be very cautious when using LVM snapshots with a Postgres data directory, or VM-bas

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
Hi, On Tue, Mar 16, 2021 at 1:44 PM Peter Geoghegan wrote: > On Tue, Mar 16, 2021 at 5:01 AM Avinash Kumar > wrote: > > I am afraid that it looks to me like a deduplication bug but not sure > how this can be pin-pointed. If there is something I could do to determine > that, I would be more tha

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 5:01 AM Avinash Kumar wrote: > I am afraid that it looks to me like a deduplication bug but not sure how > this can be pin-pointed. If there is something I could do to determine that, > I would be more than happy. That cannot be ruled out, but I don't consider it to be t

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
On Mon, Mar 15, 2021 at 3:21 PM Avinash Kumar wrote: > Hi, > > On Mon, Mar 15, 2021 at 1:18 PM Peter Geoghegan wrote: > >> On Mon, Mar 15, 2021 at 6:56 AM Avinash Kumar >> wrote: >> > psql:amchecksql.sql:17: DEBUG: leaf block 1043751 of index >> "idx_id_mtime" has no first data item >> >> That

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Avinash Kumar
Hi, On Mon, Mar 15, 2021 at 1:18 PM Peter Geoghegan wrote: > On Mon, Mar 15, 2021 at 6:56 AM Avinash Kumar > wrote: > > psql:amchecksql.sql:17: DEBUG: leaf block 1043751 of index > "idx_id_mtime" has no first data item > > That one is harmless. > > > And one error as follows. > > > > psql:amch

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Peter Geoghegan
On Mon, Mar 15, 2021 at 6:56 AM Avinash Kumar wrote: > psql:amchecksql.sql:17: DEBUG: leaf block 1043751 of index "idx_id_mtime" > has no first data item That one is harmless. > And one error as follows. > > psql:amchecksql.sql:17: ERROR: down-link lower bound invariant violated for > index

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Avinash Kumar
Hi, On Sun, Mar 14, 2021 at 11:24 PM Peter Geoghegan wrote: > On Sun, Mar 14, 2021 at 6:54 PM Avinash Kumar > wrote: > > Following may be helpful to understand what I meant. > > > > I have renamed the table and index names before adding it here. > > It should be possible to run amcheck on your

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Avinash Kumar
Hi Thomas, On Sun, Mar 14, 2021 at 10:01 PM Avinash Kumar wrote: > Hi Thomas, > > On Sun, Mar 14, 2021 at 9:40 PM Thomas Munro > wrote: > >> On Mon, Mar 15, 2021 at 1:29 PM Avinash Kumar >> wrote: >> > Is this expected when replication is happening between PostgreSQL >> databases hosted on dif

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Avinash Kumar
Hi Thomas, On Sun, Mar 14, 2021 at 9:40 PM Thomas Munro wrote: > On Mon, Mar 15, 2021 at 1:29 PM Avinash Kumar > wrote: > > Is this expected when replication is happening between PostgreSQL > databases hosted on different OS versions like Ubuntu 16 and Ubuntu 20 ? > Or, do we think this is some

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Thomas Munro
On Mon, Mar 15, 2021 at 1:29 PM Avinash Kumar wrote: > Is this expected when replication is happening between PostgreSQL databases > hosted on different OS versions like Ubuntu 16 and Ubuntu 20 ? Or, do we > think this is some sort of corruption ? Is this index on a text datatype, and using a c

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-14 Thread Peter Geoghegan
On Sun, Mar 14, 2021 at 6:54 PM Avinash Kumar wrote: > Following may be helpful to understand what I meant. > > I have renamed the table and index names before adding it here. It should be possible to run amcheck on your database, which will detect corrupt posting list tuples on Postgres 13. It's

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-14 Thread Avinash Kumar
Hi, On Sun, Mar 14, 2021 at 10:17 PM Thomas Munro wrote: > [Dropping pgsql-general@ from the CC, because cross-posting triggers > moderation; sorry I didn't notice that on my first reply] > > On Mon, Mar 15, 2021 at 2:05 PM Avinash Kumar > wrote: > > On Sun, Mar 14, 2021 at 10:01 PM Avinash Kum

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-14 Thread Thomas Munro
[Dropping pgsql-general@ from the CC, because cross-posting triggers moderation; sorry I didn't notice that on my first reply] On Mon, Mar 15, 2021 at 2:05 PM Avinash Kumar wrote: > On Sun, Mar 14, 2021 at 10:01 PM Avinash Kumar > wrote: >> Also the datatype is bigint Ok. Collation changes ar

Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-14 Thread Avinash Kumar
Hi , In one of the environments, using pg_upgrade with hard links, PostgreSQL 12 has been upgraded to PostgreSQL 13.1. The OS was Ubuntu 16.04.7 LTS (Xenial Xerus). pg_repack was used to rebuild all the tables across the database right after the upgrade to PG 13. A new server with Ubuntu 20.04.1