Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-25 Thread Tom Lane
Jeff Frost writes: > On May 24, 2012, at 3:35 PM, Tom Lane wrote: >> Jeff Frost writes: >>> BTW, when I connected to it this time, I had a really long time before my >>> psql was able to send a query, so it seems to be still broken at least. >> Yeah, I was afraid that re-initdb would be at best

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-24 Thread Jeff Frost
On May 24, 2012, at 3:35 PM, Tom Lane wrote: > Jeff Frost writes: >> On May 24, 2012, at 3:13 PM, Tom Lane wrote: >>> Huh. A bit bigger, but not by that much. It doesn't seem like this >>> would be enough to make seqscan performance fall off a cliff, as it >>> apparently did. Unless maybe the

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-24 Thread Tom Lane
Jeff Frost writes: > On May 24, 2012, at 3:13 PM, Tom Lane wrote: >> Huh. A bit bigger, but not by that much. It doesn't seem like this >> would be enough to make seqscan performance fall off a cliff, as it >> apparently did. Unless maybe the slightly-bloated catalogs were just a >> bit too lar

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-24 Thread Jeff Frost
On May 24, 2012, at 3:13 PM, Tom Lane wrote: > Jeff Frost writes: >> On 05/24/12 12:21, Tom Lane wrote: > > Huh. A bit bigger, but not by that much. It doesn't seem like this > would be enough to make seqscan performance fall off a cliff, as it > apparently did. Unless maybe the slightly-blo

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-24 Thread Tom Lane
Jeff Frost writes: > On 05/24/12 12:21, Tom Lane wrote: >> How big is pg_attribute normally in this database? I'm suspicious that >> what triggered this is some major bloating of pg_attribute (and maybe >> some of the other catalogs too). > So, the current working system's pg_attribute looks lik

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-24 Thread Jeff Frost
On 05/24/12 12:21, Tom Lane wrote: > Jeff Frost writes: >> A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 >> x86_64 >> get stuck in 'startup' mode. By that I mean the set_ps_output mode. Postgres >> is installed via Martin Pitt's packages. > I took another look at your o

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-05-24 Thread Tom Lane
Jeff Frost writes: > A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 x86_64 > get stuck in 'startup' mode. By that I mean the set_ps_output mode. Postgres > is installed via Martin Pitt's packages. I took another look at your original report here, and noticed something th

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-30 Thread Jeff Frost
On 04/28/12 17:17, Jeff Frost wrote: > Since I had a theory that it's probably stalling on pg_catalog access, one of > the guys wrote a test harness that makes several connections and creates and > drops lots of temp tables. That does seem to allow us to reproduce the issue > in the production D

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-28 Thread Jeff Frost
On Apr 27, 2012, at 6:15 PM, Tom Lane wrote: > Jeff Frost writes: >> Oh, good idea! Looks like pg_buffercache is installed in this DB. Customer >> reports that it has been installed since the server has existed (and on the >> previous server) but is not currently being used, though the issue w

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On Apr 27, 2012, at 6:34 PM, Jeff Frost wrote: > On 04/27/12 18:27, Jeff Frost wrote: >> To make it more interesting, today is a >> slow day. > > And since it's a slow day..one more question..any further logging we could do > to help find the culprit? FYI, reindexed the system tables and red

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 18:27, Jeff Frost wrote: > To make it more interesting, today is a > slow day. And since it's a slow day..one more question..any further logging we could do to help find the culprit? -- Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org) To make changes to your subscription:

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 18:15, Tom Lane wrote: > No, it shouldn't. Perhaps we should start asking questions around the > idea of "so what changed this morning?". There has to have been some > triggering cause, a configuration change or application change or > something. Oh, we've been asking those questions

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Tom Lane
Jeff Frost writes: > Oh, good idea! Looks like pg_buffercache is installed in this DB. Customer > reports that it has been installed since the server has existed (and on the > previous server) but is not currently being used, though the issue with the > hanging startups did not start until this

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 17:45, Jeff Frost wrote: > Oh, good idea! Looks like pg_buffercache is installed in this DB. Customer > reports that it has been installed since the server has existed (and on the > previous server) but is not currently being used, though the issue with the > hanging startups did not

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 17:30, Tom Lane wrote: > Jeff Frost writes: >> and I've got 81 more that do not contain bufmgr.c and are also not block on >> LWLockAcquire. > Hm ... no smoking gun in what you showed so far. I also took another > look through 9.1 bufmgr.c, and I'm darned if I can see any code path >

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Tom Lane
Jeff Frost writes: > and I've got 81 more that do not contain bufmgr.c and are also not block on > LWLockAcquire. Hm ... no smoking gun in what you showed so far. I also took another look through 9.1 bufmgr.c, and I'm darned if I can see any code path there that holds a buffer mapping lock for a

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
It's tough to catch the thing in the act on the new server as they always eventually clear..though they still sit in startup for up to 30 seconds on occasion. A few here that look interesting: #0 0x7f62b6150583 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:82 #1 0x7f62b8

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
> I think you can probably skip all that are blocked in LWLockAcquire > called from bufmgr.c:531, at least for a first pass. Calls from > elsewhere in bufmgr.c might be more interesting, and anything that's not > blocked at an LWLockAcquire at all might be even more interesting. A few more that i

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 12:27, Tom Lane wrote: > Jeff Frost writes: >> Any idea what I should be looking for in the backtraces? >> I would imagine I can ignore any that are in InitPostgres, but that still >> leaves quite a few to look through. > I think you can probably skip all that are blocked in LWLockAcqu

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Tom Lane
Jeff Frost writes: > Any idea what I should be looking for in the backtraces? > I would imagine I can ignore any that are in InitPostgres, but that still > leaves quite a few to look through. I think you can probably skip all that are blocked in LWLockAcquire called from bufmgr.c:531, at least f

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 12:17, Tom Lane wrote: > Jeff Frost writes: >> Alright, found one that's a little different (at least it wasn't in >> InitPostgres): > It's still blocking at bufmgr.c:531 though ... so AFAICS this is another > victim of somebody monopolizing a buffer mapping lock. > > Any idea what I

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Tom Lane
Jeff Frost writes: > Alright, found one that's a little different (at least it wasn't in > InitPostgres): It's still blocking at bufmgr.c:531 though ... so AFAICS this is another victim of somebody monopolizing a buffer mapping lock. regards, tom lane -- Sent via pgsql

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 11:54, Jeff Frost wrote: > On 04/27/12 10:14, Tom Lane wrote: >> Jeff Frost writes: >>> A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 >>> x86_64 >>> get stuck in 'startup' mode. >> Well, the one you backtraced seems to be waiting for somebody else to >> relea

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 10:14, Tom Lane wrote: > Jeff Frost writes: >> A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 >> x86_64 >> get stuck in 'startup' mode. > Well, the one you backtraced seems to be waiting for somebody else to > release one of the buffer mapping locks ... which

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Tom Lane
Jeff Frost writes: > A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 x86_64 > get stuck in 'startup' mode. Well, the one you backtraced seems to be waiting for somebody else to release one of the buffer mapping locks ... which is not a lock I'd expect to get held long, eve

Re: [BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
On 04/27/12 09:07, Jeff Frost wrote: > A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 x86_64 > get stuck in 'startup' mode. By that I mean the set_ps_output mode. Postgres > is installed via Martin Pitt's packages. quick followup on this..when it happens, you can connect

[BUGS] 9.1.3 backends getting stuck in 'startup'

2012-04-27 Thread Jeff Frost
A few times today, we've seen postgresql 9.1.3 backends on Ubuntu 11.10 x86_64 get stuck in 'startup' mode. By that I mean the set_ps_output mode. Postgres is installed via Martin Pitt's packages. It manifests like this: Server has been humming along fine, then suddenly many backends get stuck i