Re: AWS forcing PG upgrade from v9.6 a disaster

2021-05-28 Thread Dean Gibson (DB Administrator)
On 2021-05-28 16:51, Ron wrote: On 5/28/21 5:06 PM, Dean Gibson (DB Administrator) wrote: On 2021-05-28 12:38, Ron wrote: On 5/28/21 1:40 PM, Dean Gibson (DB Administrator) wrote: On 2021-05-28 08:12, Adrian Klaver wrote: On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: I started to

Re: AWS forcing PG upgrade from v9.6 a disaster

2021-05-28 Thread Ron
On 5/28/21 5:06 PM, Dean Gibson (DB Administrator) wrote: On 2021-05-28 12:38, Ron wrote: On 5/28/21 1:40 PM, Dean Gibson (DB Administrator) wrote: On 2021-05-28 08:12, Adrian Klaver wrote: On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: I started to use PostgreSQL v7.3 in 2003 on my

Re: AWS forcing PG upgrade from v9.6 a disaster

2021-05-28 Thread Dean Gibson (DB Administrator)
On 2021-05-28 12:38, Ron wrote: On 5/28/21 1:40 PM, Dean Gibson (DB Administrator) wrote: On 2021-05-28 08:12, Adrian Klaver wrote: On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: I started to use PostgreSQL v7.3 in 2003 on my home Linux systems (4 at one point), gradually moving to

Re: WAL accumulating, Logical Replication pg 13

2021-05-28 Thread Vijaykumar Jain
I am not too sure with 9.3 i tried an upgrade from 9.6 to 11 using logical replication (pg_logical extension) one thing to note. logical replication initiates a copy from a snapshot, then changes from then on. I had a very high insert rate on my source tables (v9.6) and the destination (v11) coul

WAL accumulating, Logical Replication pg 13

2021-05-28 Thread Willy-Bas Loos
Hi , I'm upgrading a 1.5TB database from postgres 9.3 to postgres 13 on Debian 10. This is now in an Acceptance stage (DTAP). I have encountered a problem: the WAL is not being deleted. I now have 1.4 TB of WAL in pg_wal and my disks are getting full. The oldest WAL file is 18 days old. I use Logic

Re: AWS forcing PG upgrade from v9.6 a disaster

2021-05-28 Thread Ron
On 5/28/21 1:40 PM, Dean Gibson (DB Administrator) wrote: On 2021-05-28 08:12, Adrian Klaver wrote: On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: I started to use PostgreSQL v7.3 in 2003 on my home Linux systems (4 at one point), gradually moving to v9.0 w/ replication in 2010.  In

Re: AWS forcing PG upgrade from v9.6 a disaster

2021-05-28 Thread Dean Gibson (DB Administrator)
On 2021-05-28 08:12, Adrian Klaver wrote: On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: I started to use PostgreSQL v7.3 in 2003 on my home Linux systems (4 at one point), gradually moving to v9.0 w/ replication in 2010.  In 2017 I moved my 20GB database to AWS/RDS, gradually upgrad

Re: Modelling versioning in Postgres

2021-05-28 Thread Laura Smith
Thanks both for the interesting idea of using tsrange, but also for introducing me to EXCLUDE USING GIST, I had never heard of it before. Have a good weekend ‐‐‐ Original Message ‐‐‐ On Friday, 28 May 2021 14:13, Michael van der Kolff wrote: > One thing you could consider is a range t

Re: How different is AWS-RDS postgres?

2021-05-28 Thread Rob Sargent
Found it.  Though there are indeed separate transactions for the bulk copy  v. target update, I had missed that there is an over-arching transaction in play and of course when things go south, /all/ things go south. I get away with this, for now by bumping the tomcat idle time to an hour, whi

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Tom Lane
I wrote: > However ... what seems odd is that we only get an inval message when > there's an index involved. In principle such messages need to be > issued against the table as well. I suspect that we're optimizing > away the message for the table, either on the grounds that it's temp > or that i

Re: How long to get a password reset ???

2021-05-28 Thread Ron
On 5/28/21 10:02 AM, Magnus Hagander wrote: On Fri, May 28, 2021 at 5:42 AM Dean Gibson (DB Administrator) wrote: It's pretty simple: Not having used this mailing list for a while, I went to https://lists.postgresql.org/ to make sure my settings were as I wanted them. I attempted to log in wi

Re: WARNING: oldest xmin is far in the past

2021-05-28 Thread Vijaykumar Jain
If the replication slot is still inactive, It will prevent vacuum to do the cleanup no matter how much vacuum is run manually. did the slot show as active after the restart of the collector ? If it is active then may be increase maintenance_work_mem to a aggresive value and lower nap time for aut

WARNING: oldest xmin is far in the past

2021-05-28 Thread Alban Hertroys
Good day, We have a PG 11.11 instance here that serves as a data-warehouse for us. This morning I was investigating an issue with our ETL's and discovered this error in the logs, that keeps repeating: 2021-05-28 15:01:54.094 CEST [20164]   WARNING:  oldest xmin is far in the past 2021-05-28

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Tom Lane
"Nick Muerdter" writes: > I've been seeing what looks like unbounded memory growth (until the OOM > killer kicks in and kills the postgres process) when running a pl/pgsql > function that performs TRUNCATE statements against various temporary tables > in a loop. I think I've been able to come u

Re: AWS forcing PG upgrade from v9.6 a disaster

2021-05-28 Thread Adrian Klaver
On 5/27/21 8:41 PM, Dean Gibson (DB Administrator) wrote: I started to use PostgreSQL v7.3 in 2003 on my home Linux systems (4 at one point), gradually moving to v9.0 w/ replication in 2010.  In 2017 I moved my 20GB database to AWS/RDS, gradually upgrading to v9.6, & was entirely satisfied with

Re: How long to get a password reset ???

2021-05-28 Thread Magnus Hagander
On Fri, May 28, 2021 at 5:42 AM Dean Gibson (DB Administrator) wrote: > > It's pretty simple: > > Not having used this mailing list for a while, I went to > https://lists.postgresql.org/ to make sure my settings were as I wanted them. > I attempted to log in with the above eMail address, which is

Re: How long to get a password reset ???

2021-05-28 Thread Adrian Klaver
On 5/27/21 8:42 PM, Dean Gibson (DB Administrator) wrote: It's pretty simple: 1. Not having used this mailing list for a while, I went to https://lists.postgresql.org/ to make sure my settings were as I wanted them. 2. I attempted to log in with the above eMail address, which is ob

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Nick Muerdter
On Fri, May 28, 2021, at 7:22 AM, Tom Lane wrote: > The text column would cause the table to have an associated toast table [1], > which in turn would have an index. Both of those would be reallocated as > new files on-disk during TRUNCATE, just like the table proper. > > A plausible theory here

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Vijaykumar Jain
i tried to reproduce tracking mem allocation. demo=# DO $$ DECLAREi bigint; BEGIN CREATE TEMPORARY TABLE pg_temp.foo (id int) with ( AUTOVACUUM_ENABLED = 0, TOAST.AUTOVACUUM_ENABLED = 0 ); FOR i IN 1..2 LOOP TRUNCA

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Tom Lane
Vijaykumar Jain writes: > I too see growth when text type is used, but not when int or even fixed > size char(10) is used. > ... > but then i still do not understand how a col type *text* which is dynamic > results in mem growth (coz there are no rows inserted, i understand for > long strings db d

Re: Modelling versioning in Postgres

2021-05-28 Thread Michael van der Kolff
One thing you could consider is a range type for your "versionTS" field instead of a single point in time. So that would be: CREATE TABLE objects ( objectID uuid, versionID uuid, validRange tsrange, objectData text, ); See https://www.postgresql.org/docs/12.5/rangetypes.html for more inf

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Ravi Krishna
I am not sure about that "It creates a new empty table , followed by rename of the existing table to the new empty table and finally dropping of the old table." You mean table is re-created with new oid? I don't think oid changes, but the file relnode on the disk changes. So let me rephrase i

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Ahmet Demir
Hi Ravi, I am not sure about that "It creates a new empty table , followed by rename of the existing table to the new empty table and finally dropping of the old table." You mean table is re-created with new oid? thanks Ahmet On Fri, 28 May 2021 at 15:10, Ravi Krishna wrote: > Truncate is not

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Ravi Krishna
Truncate is not delete + vaccum. It creates a new empty table , followed by rename of the existing table to the new empty table and finally dropping of the old table. On May 28, 2021 at 7:05 AM, Vijaykumar Jain wrote: Yes, I too see growth when text type is used, but not when int or even

Re: TRUNCATE memory leak with temporary tables?

2021-05-28 Thread Vijaykumar Jain
Yes, I too see growth when text type is used, but not when int or even fixed size char(10) is used. I always thought truncate was similar to delete + vacuum full, but checking for your scenarios, I did not see an update on pg_stat_user_table on truncate for vacuums. then i checked PostgreSQL Inte

Modelling versioning in Postgres

2021-05-28 Thread Laura Smith
Hi I was wondering what the current thinking is on ways to model versioning in Postgres. The overall premise is that the latest version is the current version unless a rollback has occurred, in which case versions get tracked from the rollback point (forking ?). My initial naïve starting poin

Re: Size on disk of INT and BIGINT - not sure I'm getting it?

2021-05-28 Thread Francisco Olarte
Pól: On Fri, May 28, 2021 at 11:36 AM Pól Ua Laoínecháin wrote: > BIGINTEGER and INTEGER tables the same size? Comparing tables of 100M records. ... > test=# CREATE TABLE big_int (x BIGINT NOT NULL); > test=# CREATE TABLE int_32 (y INT); Bear in mind a row has more data than just the fields ( thi

Size on disk of INT and BIGINT - not sure I'm getting it?

2021-05-28 Thread Pól Ua Laoínecháin
Hi all, I did the following (Asus laptop, Intel core i5, SATA SSD): BIGINTEGER and INTEGER tables the same size? Comparing tables of 100M records. Create tables: = test=# CREATE TABLE big_int (x BIGINT NOT NULL); CREATE TABLE test=# CREATE TABL