Re: walreceiver termination

2020-05-04 Thread Justin King
ystem in common -- particularly ZFS (or btrfs, in the bottom case). Is there anything more we can do here to help narrow down this issue? I'm happy to help, but I honestly wouldn't even know where to begin. Thanks- Justin King flightaware.com On Thu, Apr 23, 2020 at 4:40 PM Justin King

Re: walreceiver termination

2020-04-23 Thread Justin King
On Thu, Apr 23, 2020 at 3:06 PM Tom Lane wrote: > > Justin King writes: > > I assume it would be related to the following: > > LOG: incorrect resource manager data checksum in record at 2D6/C259AB90 > > since the walreceiver terminates just after this - but I'm un

Re: walreceiver termination

2020-04-23 Thread Justin King
seemingly random times. Also, just to clarify, this will only happen on a single replica at a time. On Thu, Apr 23, 2020 at 2:46 PM Justin King wrote: > > On Thu, Apr 23, 2020 at 12:47 PM Tom Lane wrote: > > > > Justin King writes: > > > We've seen unexpect

Re: walreceiver termination

2020-04-23 Thread Justin King
On Thu, Apr 23, 2020 at 12:47 PM Tom Lane wrote: > > Justin King writes: > > We've seen unexpected termination of the WAL receiver process. This > > stops streaming replication, but the replica stays available -- > > restarting the server resumes streaming

walreceiver termination

2020-04-23 Thread Justin King
We've seen unexpected termination of the WAL receiver process. This stops streaming replication, but the replica stays available -- restarting the server resumes streaming replication where it left off. We've seen this across nearly every recent version of PG, (9.4, 9.5, 11.x, 12.x) -- anything om

Re: PG12 autovac issues

2020-03-27 Thread Justin King
On Fri, Mar 27, 2020 at 12:12 AM Michael Paquier wrote: > > On Thu, Mar 26, 2020 at 09:46:47AM -0500, Justin King wrote: > > Nope, it was just these tables that were looping over and over while > > nothing else was getting autovac'd. I'm happy to share the full log &

Re: PG12 autovac issues

2020-03-26 Thread Justin King
On Wed, Mar 25, 2020 at 8:43 PM Michael Paquier wrote: > > On Wed, Mar 25, 2020 at 10:39:17AM -0500, Justin King wrote: > > This started happening again. DEBUG1 is enabled: > > Thanks for enabling DEBUG1 logs while this happened. > > > Mar 25 14:48:26 cowtn postgres[3

Re: PG12 autovac issues

2020-03-25 Thread Justin King
All- This started happening again. DEBUG1 is enabled: Mar 25 14:48:03 cowtn postgres[39720]: [35294-1] 2020-03-25 14:48:03.972 GMT [39720] DEBUG: autovacuum: processing database "template0" Mar 25 14:48:06 cowtn postgres[39735]: [35294-1] 2020-03-25 14:48:06.545 GMT [39735] DEBUG: autovacuum:

Re: PG12 autovac issues

2020-03-23 Thread Justin King
On Mon, Mar 23, 2020 at 4:31 PM Justin King wrote: > > On Mon, Mar 23, 2020 at 3:00 PM Andres Freund wrote: > > > > Hi, > > > > On 2020-03-23 20:47:25 +0100, Julien Rouhaud wrote: > > > > - relfrozenxid, age(relfrozenxid) for the oldest table in the old

Re: PG12 autovac issues

2020-03-23 Thread Justin King
On Mon, Mar 23, 2020 at 3:00 PM Andres Freund wrote: > > Hi, > > On 2020-03-23 20:47:25 +0100, Julien Rouhaud wrote: > > > - relfrozenxid, age(relfrozenxid) for the oldest table in the oldest > > > database > > > SELECT oid::regclass, age(relfrozenxid), relfrozenxid FROM pg_class > > > WHERE r

Re: PG12 autovac issues

2020-03-20 Thread Justin King
On Thu, Mar 19, 2020 at 6:56 PM Andres Freund wrote: > > Hi, > > On 2020-03-19 18:07:14 -0500, Justin King wrote: > > On Thu, Mar 19, 2020 at 5:35 PM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2020-03-19 10:23:48 -0500, Justin King

Re: PG12 autovac issues

2020-03-19 Thread Justin King
On Thu, Mar 19, 2020 at 5:35 PM Andres Freund wrote: > > Hi, > > On 2020-03-19 10:23:48 -0500, Justin King wrote: > > > From a single stats snapshot we can't actually understand the actual xid > > > consumption - is it actually the xid usage that triggers the

Re: Fwd: PG12 autovac issues

2020-03-19 Thread Justin King
On Thu, Mar 19, 2020 at 11:02 AM Michael Lewis wrote: > > On Thu, Mar 19, 2020 at 9:31 AM Justin King wrote: >> >> On Wed, Mar 18, 2020 at 1:40 PM Michael Lewis wrote: >> > >> > Do you have default fillfactor set on this table? If not, I would wonder &

Re: Fwd: PG12 autovac issues

2020-03-19 Thread Justin King
On Wed, Mar 18, 2020 at 1:40 PM Michael Lewis wrote: > > Do you have default fillfactor set on this table? If not, I would wonder if > reducing it to 50% or even 20% would allow many more HOT updates that would > reduce bloat. I don't believe we have a default fillfactor, but I'm still trying t

Re: Re: PG12 autovac issues

2020-03-19 Thread Justin King
On Wed, Mar 18, 2020 at 10:13 AM Adrian Klaver wrote: > > On 3/18/20 6:57 AM, Justin King wrote: > Please reply to list also > Ccing list > > > >>> Here are the settings, these are the only ones that are not set to > >>> default with the exception

RE: PG12 autovac issues

2020-03-19 Thread Justin King
Hi Andres- Thanks for the reply, answers below. On Tue, Mar 17, 2020 at 8:19 PM Andres Freund wrote: > > Hi, > > On 2020-03-17 17:18:57 -0500, Justin King wrote: > > As you can see in this table, there are only ~80K rows, but billions > > of updates. What we have observ

Re: Fwd: PG12 autovac issues

2020-03-17 Thread Justin King
On Tue, Mar 17, 2020 at 5:39 PM Adrian Klaver wrote: > > On 3/17/20 3:22 PM, Justin King wrote: > > Apologies, I accidentally sent this to the pgsql-admin list initially > > but intended it go here: > > > > We have a database that isn't overly large (~20G

Fwd: PG12 autovac issues

2020-03-17 Thread Justin King
and why it suddenly started when we moved from PG10 > PG12. The configs and workload are essentially the same between versions. We realize we could simply increase the autovacuum_freeze_max_age, but that doesn't seem to actually resolve anything -- it just pushes the problem out. Has anyone seen anything similar to this? Thanks very much for the consideration. Justin King http://flightaware.com/