Re: autovacuum locking question

2019-12-05 Thread Jeff Janes
On Thu, Dec 5, 2019 at 5:26 PM Mike Schanne wrote: > Hi, > > I am investigating a performance problem in our application and am seeing > something unexpected in the postgres logs regarding the autovacuum. > > > > 2019-12-01 13:05:39.029 > UTC,"wb","postgres",6966,"127.0.0.1:53976",5ddbd990.1b36,

Re: autovacuum locking question

2019-12-05 Thread Tom Lane
Mike Schanne writes: > I am investigating a performance problem in our application and am seeing > something unexpected in the postgres logs regarding the autovacuum. > 2019-12-01 13:05:39.029 > UTC,"wb","postgres",6966,"127.0.0.1:53976",5ddbd990.1b36,17099,"INSERT > waiting",2019-11-25 13:39:

RE: autovacuum locking question

2019-12-05 Thread Mike Schanne
In this particular case autovacuum_count is 0. n_live_tup is 659,631 and n_dead_tup is 3,400,347. We are using the default vacuum parameters. From: Michael Lewis [mailto:mle...@entrata.com] Sent: Thursday, December 05, 2019 5:49 PM To: Mike Schanne Cc: pgsql-performa...@postgresql.org Subject:

Re: autovacuum locking question

2019-12-05 Thread Michael Lewis
On Thu, Dec 5, 2019 at 3:26 PM Mike Schanne wrote: > I am concerned that if the autovacuum is constantly canceled, then the > table never gets cleaned and its performance will continue to degrade over > time. Is it expected for the vacuum to be canceled by an insert in this > way? > > > > We are

autovacuum locking question

2019-12-05 Thread Mike Schanne
Hi, I am investigating a performance problem in our application and am seeing something unexpected in the postgres logs regarding the autovacuum. 2019-12-01 13:05:39.029 UTC,"wb","postgres",6966,"127.0.0.1:53976",5ddbd990.1b36,17099,"INSERT waiting",2019-11-25 13:39:28 UTC,12/1884256,12615023,L

Re: Postgres backup tool recommendations for multi-terabyte database in Google Cloud

2019-12-05 Thread Nikolay Samokhvalov
Consider WAL-G. It works well with GCS nowadays. We have a good fresh experience of using it on GCP, for multi-TB databases. BTW, what is your opinion on using GCE snapshots in this context? You mention that you've consider them. Thoughts? Thanks, Nik On Thu, Dec 5, 2019 at 09:48 Craig Jackson

Re: Postgres backup tool recommendations for multi-terabyte database in Google Cloud

2019-12-05 Thread Craig Jackson
Thanks, I'll check it out. On Thu, Dec 5, 2019 at 12:51 PM Craig James wrote: > On Thu, Dec 5, 2019 at 9:48 AM Craig Jackson > wrote: > >> Hi, >> >> We are in the process of migrating an oracle database to postgres in >> Google Cloud and are investigating backup/recovery tools. The database is

Re: Postgres backup tool recommendations for multi-terabyte database in Google Cloud

2019-12-05 Thread Craig James
On Thu, Dec 5, 2019 at 9:48 AM Craig Jackson wrote: > Hi, > > We are in the process of migrating an oracle database to postgres in > Google Cloud and are investigating backup/recovery tools. The database is > size is > 20TB. We have an SLA that requires us to be able to complete a > full restore

Postgres backup tool recommendations for multi-terabyte database in Google Cloud

2019-12-05 Thread Craig Jackson
Hi, We are in the process of migrating an oracle database to postgres in Google Cloud and are investigating backup/recovery tools. The database is size is > 20TB. We have an SLA that requires us to be able to complete a full restore of the database within 24 hours. We have been testing pgbackreset

Re: How to run in parallel in Postgres

2019-12-05 Thread Laurenz Albe
On Thu, 2019-12-05 at 12:10 +, Lars Aksel Opsahl wrote: > have a function that prepares data, so the big job can be run it in parallel. > > Today I have solved this by using "Gnu parallel" like this. > psql testdb -c"\! psql -t -q -o /tmp/run_cmd.sql testdb -c\"SELECT > find_overlap_gap_make

How to run in parallel in Postgres

2019-12-05 Thread Lars Aksel Opsahl
Hi I have a function that prepares data, so the big job can be run it in parallel. Today I have solved this by using "Gnu parallel" like this. psql testdb -c"\! psql -t -q -o /tmp/run_cmd.sql testdb -c\"SELECT find_overlap_gap_make_run_cmd('sl_lop.overlap_gap_input_t1','geom',4258,'sl_lop.overla