Re: [GENERAL] Increase in max_connections

2014-04-08 Thread Kevin Grittner
Kevin Grittner wrote: > Please post the output of this: > > numactl --hardware Oh, it would also help in making specific suggestions if you could show the output of: mount | grep cpuset ... and a listing of "file names" in the mounted directory. There is some variation among distros in both

Re: [GENERAL] Increase in max_connections

2014-04-07 Thread Kevin Grittner
"Anand Kumar, Karthik" wrote: > We finally made some headway on this - we noticed messages like > the below > in /var/log/messages whenever the issue happened: > > Mar 26 07:39:58 site-db01b kernel: postmaster: page allocation failure. > Anyone have any idea why memory was so fragmented, and wh

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Anand Kumar, Karthik
Thanks Bruce. Really interesting, but, I show zone reclaim is already turned off on our system. root@site-db01b:~ # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17 node 0 size: 393181 MB node 0 free: 467 MB node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23 node 1

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Bruce Momjian
On Wed, Mar 26, 2014 at 08:22:01PM +, Anand Kumar, Karthik wrote: > Looking a little deeper, I saw signs of memory being heavily fragmented: > > root@site-db01b:/var/log # cat /proc/buddyinfo > Node 0, zone DMA 1 1 2 2 2 1 0 0 1 1 3 > Node 0, zone DMA32 8 7 8 7 10 8 7 11 9 5 92 > Node 0, zone

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Anand Kumar, Karthik
Hi all, We finally made some headway on this - we noticed messages like the below in /var/log/messages whenever the issue happened: Mar 26 07:39:58 site-db01b kernel: postmaster: page allocation failure. order:1, mode:0x20 Mar 26 07:39:58 site-db01b kernel: Pid: 39066, comm: postmaster Not tainte

Re: [GENERAL] Increase in max_connections

2014-03-14 Thread Anand Kumar, Karthik
For anyone that's still following - we tried upgrading to postgres 9.3.3 - that hasn't helped. Running an strace on the pid that was consuming the highest CPU at the time of the outage shows: semop(91521110, {{12, -1, 0}}, 1) = 0 semop(91521110, {{12, -1, 0}}, 1) = 0 semop(91521110, {

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread John R Pierce
On 3/11/2014 10:20 AM, Anand Kumar, Karthik wrote: We typically see about 500-700 active queries at a time if these are primarily small/fast queries, like OLTP operations, and you DONT have 200-400 CPU cores on this server, you will likely find that if you use a queueing mechanism to only exe

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Jeff Janes
On Tue, Mar 11, 2014 at 10:20 AM, Anand Kumar, Karthik < karthik.anandku...@classmates.com> wrote: > Thanks Jeff. We have scripts in place now to capture the incoming rate > of requests. Waiting on the crash to happen to see if it spikes up :) > > Re: min_log_duration - we *do* see a good numbe

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Anand Kumar, Karthik
Anand Kumar, Karthik" mailto:karthik.anandku...@classmates.com>> Cc: "pgsql-general@postgresql.org<mailto:pgsql-general@postgresql.org>" mailto:pgsql-general@postgresql.org>> Subject: Re: [GENERAL] Increase in max_connections On Mon, Mar 10, 2014 at 6:04 PM,

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Anand Kumar, Karthik
quot;Anand Kumar, Karthik" mailto:karthik.anandku...@classmates.com>> Cc: "pgsql-general@postgresql.org<mailto:pgsql-general@postgresql.org>" mailto:pgsql-general@postgresql.org>> Subject: Re: [GENERAL] Increase in max_connections On Tue, Mar 11, 2014 at 12:04

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Jeff Janes
On Mon, Mar 10, 2014 at 6:04 PM, Anand Kumar, Karthik < karthik.anandku...@classmates.com> wrote: > Hi all, > > We're running postgres 9.3.2, server configuration below. > > Seemingly randomly, we will see the number of active queries in postgres > go up until we hit max_connections. The DB wi

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Igor Neyman
From: pgsql-general-ow...@postgresql.org [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Anand Kumar, Karthik Sent: Monday, March 10, 2014 9:04 PM To: pgsql-general@postgresql.org Subject: [GENERAL] Increase in max_connections Hi all, We're running postgres 9.3.2, server configuration

Re: [GENERAL] Increase in max_connections

2014-03-10 Thread Venkata Balaji Nagothi
On Tue, Mar 11, 2014 at 12:04 PM, Anand Kumar, Karthik < karthik.anandku...@classmates.com> wrote: > Hi all, > > We're running postgres 9.3.2, server configuration below. > > Seemingly randomly, we will see the number of active queries in postgres > go up until we hit max_connections. The DB w