Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Andres Freund
Hi, On 2021-04-13 23:04:50 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2021-04-13 19:16:46 -0400, Tom Lane wrote: > >> Like this? This passes check-world, modulo the one very-unsurprising > >> regression test change. I've not tried to do any performance testing. > > > I wonder if the

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Tom Lane
Andres Freund writes: > On 2021-04-13 19:16:46 -0400, Tom Lane wrote: >> Like this? This passes check-world, modulo the one very-unsurprising >> regression test change. I've not tried to do any performance testing. > I wonder if there's a realistic chance it could create additional > deadlocks

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Andres Freund
Hi, On 2021-04-13 19:16:46 -0400, Tom Lane wrote: > > Maybe the same thing we do with user tables, ie not give up the lock > > when we close a toast rel? As long as the internal lock counters > > are 64-bit, we'd not have to worry about overflowing them. Well, I was assuming we'd not want to do

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Tom Lane
I wrote: > Andres Freund writes: >> Cool. And damn: I can't immediately think of a way to optimize this to >> not require this kind of hack in the future. > Maybe the same thing we do with user tables, ie not give up the lock > when we close a toast rel? As long as the internal lock counters > a

RE: LWLocks by LockManager slowing large DB

2021-04-13 Thread Paul Friedman
> For toast tables we do not keep locks held for the duration of the transaction, > but release the lock as soon as one access is done. ... > The ability to lock a toast table? Yea, it might be worth doing that. I seem to > recall this being discussed not too long ago... Actually, the requested im

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Tom Lane
Andres Freund writes: > On 2021-04-13 11:47:06 -0700, Paul Friedman wrote: >> YES!!! This completely alleviates the bottleneck and I'm able to run the >> queries full-throttle. Thank you SO much for your help+insight. > Cool. And damn: I can't immediately think of a way to optimize this to > no

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Alvaro Herrera
On 2021-Apr-13, Andres Freund wrote: > > Sounds like I should file this as a requested improvement? > > The ability to lock a toast table? Yea, it might be worth doing that. I > seem to recall this being discussed not too long ago... Yep, commit 59ab4ac32460 reverted by eeda7f633809. There were

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Andres Freund
Hi, On 2021-04-13 11:47:06 -0700, Paul Friedman wrote: > YES!!! This completely alleviates the bottleneck and I'm able to run the > queries full-throttle. Thank you SO much for your help+insight. Cool. And damn: I can't immediately think of a way to optimize this to not require this kind of hac

RE: LWLocks by LockManager slowing large DB

2021-04-13 Thread Paul Friedman
YES!!! This completely alleviates the bottleneck and I'm able to run the queries full-throttle. Thank you SO much for your help+insight. Interestingly, "lock pg_toast.pg_toast_2233612264 in ACCESS SHARE MODE;" which should do the same thing returns an error " ERROR: "pg_toast_2233612264" is not

Re: LWLocks by LockManager slowing large DB

2021-04-13 Thread Andres Freund
Hi, On 2021-04-13 09:33:48 -0700, Paul Friedman wrote: > I've attached the 2 perf reports. From the 2nd one, I can see lots of > time waiting for TOAST table locks on the geometry column, but I > definitely don't fully understand the implications or why LockManager > would be struggling here. Oh

RE: LWLocks by LockManager slowing large DB

2021-04-13 Thread Paul Friedman
Thanks for this – these tools (and the raw selects on pg_stat_activity and pg_locks) are all showing wait events being created by LockManager waiting on an LWLock. ---Paul Paul Friedman CTO 677 Harrison St | San Francisco, CA 94107 *M:* (650) 270-7676 *E-mail:* paul.fried...@streetlig

RE: LWLocks by LockManager slowing large DB

2021-04-13 Thread Paul Friedman
Thanks for this, I read too quickly! I've attached the 2 perf reports. From the 2nd one, I can see lots of time waiting for TOAST table locks on the geometry column, but I definitely don't fully understand the implications or why LockManager would be struggling here. Thanks for the continued hel