Re: Need help identifying a periodic performance issue.

2021-11-29 Thread Robert Creager
> On Nov 24, 2021, at 4:15 PM, Justin Pryzby wrote: > > This message originated outside your organization. > > On Wed, Nov 24, 2021 at 10:44:12PM +, Robert Creager wrote: >> I forgot, I had reloaded postgres, but had not re-started our app, so the >> connections wouldn’t have that plan sett

Re: Need help identifying a periodic performance issue.

2021-11-24 Thread Justin Pryzby
On Wed, Nov 24, 2021 at 10:44:12PM +, Robert Creager wrote: > I forgot, I had reloaded postgres, but had not re-started our app, so the > connections wouldn’t have that plan setting on them. Re-doing now. Are you sure? GUC changes should be applied for existing sessions, right ? Would you s

Re: Need help identifying a periodic performance issue.

2021-11-24 Thread Robert Creager
I forgot, I had reloaded postgres, but had not re-started our app, so the connections wouldn’t have that plan setting on them. Re-doing now. On Nov 24, 2021, at 2:13 PM, Robert Creager mailto:robe...@spectralogic.com>> wrote: On Nov 19, 2021, at 11:47 AM, Robert Creager mailto:robe...@spectr

Re: Need help identifying a periodic performance issue.

2021-11-24 Thread Robert Creager
On Nov 19, 2021, at 11:47 AM, Robert Creager mailto:robe...@spectralogic.com>> wrote: On Nov 18, 2021, at 2:42 PM, Tom Lane mailto:t...@sss.pgh.pa.us>> wrote: This message originated outside your organization. Thomas Munro mailto:thomas.mu...@gmail.com>> writes: This is all guesswork thou

Re: Need help identifying a periodic performance issue.

2021-11-19 Thread Robert Creager
On Nov 18, 2021, at 2:42 PM, Tom Lane mailto:t...@sss.pgh.pa.us>> wrote: This message originated outside your organization. Thomas Munro mailto:thomas.mu...@gmail.com>> writes: This is all guesswork though, since we don't know quite what's happening on Robert's system. It might be worth sett

Re: Need help identifying a periodic performance issue.

2021-11-18 Thread Robert Creager
> On Nov 18, 2021, at 2:42 PM, Tom Lane wrote: > > This is all guesswork though, since we don't know quite what's > happening on Robert's system. It might be worth setting > "log_autovacuum_min_duration = 0" (either globally, or as a > reloption on the relevant tables), and seeing if there see

Re: Need help identifying a periodic performance issue.

2021-11-18 Thread Robert Creager
On Nov 18, 2021, at 2:08 PM, Thomas Munro mailto:thomas.mu...@gmail.com>> wrote: This message originated outside your organization. On Fri, Nov 19, 2021 at 6:03 AM Robert Creager mailto:robe...@spectralogic.com>> wrote: Which would be better? Discard plans or forcing custom plans? Seems lik

Re: Need help identifying a periodic performance issue.

2021-11-18 Thread Tom Lane
Thomas Munro writes: > I'm wondering if the thing that changed between 9.6 and 13 might be > the heuristics for when auto vacuum's background ANALYZE is triggered, > creating the unlucky timing required to get your system to this state > occasionally. > For a while now I have been wondering how w

Re: Need help identifying a periodic performance issue.

2021-11-18 Thread Thomas Munro
On Fri, Nov 19, 2021 at 6:03 AM Robert Creager wrote: > Which would be better? Discard plans or forcing custom plans? Seems like > wrapping a copy might be better than the Postgres.conf change as that would > affect all statements. What kind of performance hit would we be taking with > that

Re: Need help identifying a periodic performance issue.

2021-11-18 Thread Robert Creager
> On Nov 17, 2021, at 10:42 PM, Justin Pryzby wrote: > > This message originated outside your organization. > > On Thu, Nov 18, 2021 at 04:39:42PM +1300, Thomas Munro wrote: >> On Thu, Nov 18, 2021 at 1:18 PM Robert Creager >> wrote: >>> So, how do I go about capturing more information for

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Justin Pryzby
On Thu, Nov 18, 2021 at 04:39:42PM +1300, Thomas Munro wrote: > On Thu, Nov 18, 2021 at 1:18 PM Robert Creager > wrote: > > So, how do I go about capturing more information for the big brains (you > > guys) to help figure this out? I have all our resources at mine (and hence > > your) disposal

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Thomas Munro
On Thu, Nov 18, 2021 at 1:18 PM Robert Creager wrote: > So, how do I go about capturing more information for the big brains (you > guys) to help figure this out? I have all our resources at mine (and hence > your) disposal. As a workaround, does it help if you issue DISCARD PLANS before your C

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Robert Creager
> On Nov 17, 2021, at 4:18 PM, Justin Pryzby wrote: > > This message originated outside your organization. > > On Wed, Nov 17, 2021 at 09:54:14PM +, Robert Creager wrote: > > We are able to move up to Postgres 13.5, in our ports tree, if that would > > help. We used pg_upgrade to get from

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 09:54:14PM +, Robert Creager wrote: > We are able to move up to Postgres 13.5, in our ports tree, if that would > help. We used pg_upgrade to get from 9.6 to 13.3, so that should work fine > going instead to 13.5. We’re almost branching/releasing our code, so it’s >

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Thomas Munro
On Thu, Nov 18, 2021 at 8:28 AM Tom Lane wrote: > Justin Pryzby writes: > > It shows that the process is running FK triggers. > > Indeed, and doing a seqscan therein. Normally I'd suppose that > this reflects a lack of an index, but RI_FKey_check should always > be doing something that matches t

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Robert Creager
On Nov 17, 2021, at 12:00 PM, Justin Pryzby mailto:pry...@telsasoft.com>> wrote: This message originated outside your organization. On Wed, Nov 17, 2021 at 05:51:05PM +, Robert Creager wrote: postgres`HeapTupleSatisfiesVisibility+0x42 postgres`heapgetpage+0x237

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Tom Lane
Justin Pryzby writes: > It shows that the process is running FK triggers. Indeed, and doing a seqscan therein. Normally I'd suppose that this reflects a lack of an index, but RI_FKey_check should always be doing something that matches the referenced table's unique constraint, so why isn't it usi

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 05:51:05PM +, Robert Creager wrote: > postgres`HeapTupleSatisfiesVisibility+0x42 > postgres`heapgetpage+0x237 > postgres`heapgettup_pagemode+0x5ad > postgres`heap_getnextslot+0x52 > postgres`SeqNext+0x

Re: Need help identifying a periodic performance issue.

2021-11-17 Thread Robert Creager
On Nov 17, 2021, at 10:51 AM, Robert Creager mailto:robe...@spectralogic.com>> wrote: On Nov 15, 2021, at 10:50 PM, Thomas Munro mailto:thomas.mu...@gmail.com>> wrote: This message originated outside your organization. On Tue, Nov 16, 2021 at 5:43 PM Robert Creager mailto:robe...@spectral

Re: Need help identifying a periodic performance issue.

2021-11-16 Thread Thomas Munro
On Wed, Nov 17, 2021 at 11:51 AM Thomas Munro wrote: > It's the right output format, but isn't /pid == '$PID'/ only going to > match one single process called "postgres"? Maybe /execname == > "postgres"/ to catch them all? Oh, duh, it's the top CPU one. Makes sense. Never mind :-)

Re: Need help identifying a periodic performance issue.

2021-11-16 Thread Thomas Munro
On Wed, Nov 17, 2021 at 11:40 AM Robert Creager wrote: > Presuming this is the type of output you are expecting: > > CPU IDFUNCTION:NAME > 0 58709:tick-10s > > > postgres`AtEOXact_LargeObject+0x11 > postgres`CommitTrans

Re: Need help identifying a periodic performance issue.

2021-11-16 Thread Robert Creager
On Nov 15, 2021, at 10:50 PM, Thomas Munro mailto:thomas.mu...@gmail.com>> wrote: This message originated outside your organization. On Tue, Nov 16, 2021 at 5:43 PM Robert Creager mailto:robe...@spectralogic.com>> wrote: One CPU is pegged, the data has been sent over STDIN, so Postgres is not

Re: Need help identifying a periodic performance issue.

2021-11-16 Thread Robert Creager
On Nov 15, 2021, at 10:29 PM, Justin Pryzby mailto:pry...@telsasoft.com>> wrote: This message originated outside your organization. On Tue, Nov 16, 2021 at 04:43:25AM +, Robert Creager wrote: > We’re executing the following copy to fill a table with approximately 5k > records, then repeat

Re: Need help identifying a periodic performance issue.

2021-11-15 Thread Thomas Munro
On Tue, Nov 16, 2021 at 5:43 PM Robert Creager wrote: > One CPU is pegged, the data has been sent over STDIN, so Postgres is not > waiting for more, there are no other queries running using this select: So PostgreSQL is eating 100% CPU, with no value shown in wait_event_type, and small numbers o

Re: Need help identifying a periodic performance issue.

2021-11-15 Thread Justin Pryzby
On Tue, Nov 16, 2021 at 04:43:25AM +, Robert Creager wrote: > We’re executing the following copy to fill a table with approximately 5k > records, then repeating for a total of 250k records. Normally, this copy > executes < 1 second, with the entire set taking a couple of minutes. The > prob