Re: [GENERAL] Increase in max_connections

2014-04-08 Thread Kevin Grittner
Kevin Grittner wrote: > Please post the output of this: > > numactl --hardware Oh, it would also help in making specific suggestions if you could show the output of: mount | grep cpuset ... and a listing of "file names" in the mounted directory. There is some variation among distros in both

Re: [GENERAL] Increase in max_connections

2014-04-07 Thread Kevin Grittner
"Anand Kumar, Karthik" wrote: > We finally made some headway on this - we noticed messages like > the below > in /var/log/messages whenever the issue happened: > > Mar 26 07:39:58 site-db01b kernel: postmaster: page allocation failure. > Anyone have any idea why memory was so fragmented, and wh

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Anand Kumar, Karthik
Thanks Bruce. Really interesting, but, I show zone reclaim is already turned off on our system. root@site-db01b:~ # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17 node 0 size: 393181 MB node 0 free: 467 MB node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23 node 1

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Bruce Momjian
On Wed, Mar 26, 2014 at 08:22:01PM +, Anand Kumar, Karthik wrote: > Looking a little deeper, I saw signs of memory being heavily fragmented: > > root@site-db01b:/var/log # cat /proc/buddyinfo > Node 0, zone DMA 1 1 2 2 2 1 0 0 1 1 3 > Node 0, zone DMA32 8 7 8 7 10 8 7 11 9 5 92 > Node 0, zone

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Anand Kumar, Karthik
Hi all, We finally made some headway on this - we noticed messages like the below in /var/log/messages whenever the issue happened: Mar 26 07:39:58 site-db01b kernel: postmaster: page allocation failure. order:1, mode:0x20 Mar 26 07:39:58 site-db01b kernel: Pid: 39066, comm: postmaster Not tainte

Re: [GENERAL] Increase in max_connections

2014-03-14 Thread Anand Kumar, Karthik
For anyone that's still following - we tried upgrading to postgres 9.3.3 - that hasn't helped. Running an strace on the pid that was consuming the highest CPU at the time of the outage shows: semop(91521110, {{12, -1, 0}}, 1) = 0 semop(91521110, {{12, -1, 0}}, 1) = 0 semop(91521110, {

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread John R Pierce
On 3/11/2014 10:20 AM, Anand Kumar, Karthik wrote: We typically see about 500-700 active queries at a time if these are primarily small/fast queries, like OLTP operations, and you DONT have 200-400 CPU cores on this server, you will likely find that if you use a queueing mechanism to only exe

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Jeff Janes
On Tue, Mar 11, 2014 at 10:20 AM, Anand Kumar, Karthik < karthik.anandku...@classmates.com> wrote: > Thanks Jeff. We have scripts in place now to capture the incoming rate > of requests. Waiting on the crash to happen to see if it spikes up :) > > Re: min_log_duration - we *do* see a good numbe

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Anand Kumar, Karthik
Anand Kumar, Karthik" mailto:karthik.anandku...@classmates.com>> Cc: "pgsql-general@postgresql.org<mailto:pgsql-general@postgresql.org>" mailto:pgsql-general@postgresql.org>> Subject: Re: [GENERAL] Increase in max_connections On Mon, Mar 10, 2014 at 6:04 PM,

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Anand Kumar, Karthik
quot;Anand Kumar, Karthik" mailto:karthik.anandku...@classmates.com>> Cc: "pgsql-general@postgresql.org<mailto:pgsql-general@postgresql.org>" mailto:pgsql-general@postgresql.org>> Subject: Re: [GENERAL] Increase in max_connections On Tue, Mar 11, 2014 at 12:04

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Jeff Janes
On Mon, Mar 10, 2014 at 6:04 PM, Anand Kumar, Karthik < karthik.anandku...@classmates.com> wrote: > Hi all, > > We're running postgres 9.3.2, server configuration below. > > Seemingly randomly, we will see the number of active queries in postgres > go up until we hit max_connections. The DB wi

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Igor Neyman
From: pgsql-general-ow...@postgresql.org [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Anand Kumar, Karthik Sent: Monday, March 10, 2014 9:04 PM To: pgsql-general@postgresql.org Subject: [GENERAL] Increase in max_connections Hi all, We're running postgres 9.3.2, server configur

Re: [GENERAL] Increase in max_connections

2014-03-10 Thread Venkata Balaji Nagothi
On Tue, Mar 11, 2014 at 12:04 PM, Anand Kumar, Karthik < karthik.anandku...@classmates.com> wrote: > Hi all, > > We're running postgres 9.3.2, server configuration below. > > Seemingly randomly, we will see the number of active queries in postgres > go up until we hit max_connections. The DB w

[GENERAL] Increase in max_connections

2014-03-10 Thread Anand Kumar, Karthik
Hi all, We're running postgres 9.3.2, server configuration below. Seemingly randomly, we will see the number of active queries in postgres go up until we hit max_connections. The DB will recover after a few minutes. We had the issue a couple of times in Feb 2014. We then upgraded the postgres