Re: [PERFORM] pg_dump and thousands of schemas

2012-05-24 Thread Ondrej Ivanič
Hi, On 25 May 2012 14:54, Hugo wrote: > Thanks for the replies. The number of relations in the database is really > high (~500,000) and I don't think we can shrink that. The truth is that > schemas bring a lot of advantages to our system and postgresql doesn't show > signs of stress with them. S

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-24 Thread Hugo
Thanks for the replies. The number of relations in the database is really high (~500,000) and I don't think we can shrink that. The truth is that schemas bring a lot of advantages to our system and postgresql doesn't show signs of stress with them. So I believe it should also be possible for pg_dum

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-24 Thread Bruce Momjian
On Thu, May 24, 2012 at 08:20:34PM -0700, Jeff Janes wrote: > On Thu, May 24, 2012 at 8:21 AM, Craig James wrote: > > > > > > On Thu, May 24, 2012 at 12:06 AM, Hugo wrote: > >> > >> Hi everyone, > >> > >> We have a production database (postgresql 9.0) with more than 20,000 > >> schemas > >> and

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-24 Thread Jeff Janes
On Thu, May 24, 2012 at 8:21 AM, Craig James wrote: > > > On Thu, May 24, 2012 at 12:06 AM, Hugo wrote: >> >> Hi everyone, >> >> We have a production database (postgresql 9.0) with more than 20,000 >> schemas >> and 40Gb size. In the past we had all that information in just one schema >> and pg_

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Claudio Freire
On Thu, May 24, 2012 at 2:09 PM, Stephen Frost wrote: > * Rajesh Kumar. Mallah (mal...@tradeindia.com) wrote: >> We are running linux with kernel 3.2.X >> (which has the lseek improvements) > > Ah, good. > >> Thanks for the reference , even i thought so (LockManager) , >> but we are actually also

Re: [PERFORM] heavly load system spec

2012-05-24 Thread Merlin Moncure
On Thu, Apr 5, 2012 at 10:39 AM, Gregg Jaskiewicz wrote: > I know this is a very general question. But if you guys had to specify > system (could be one server or cluster), with sustainable transaction > rate of 1.5M tps running postgresql, what configuration and hardware > would you be looking fo

Re: [PERFORM] heavly load system spec

2012-05-24 Thread Stephen Frost
Gregg, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Apr 5, 2012 at 11:39 AM, Gregg Jaskiewicz wrote: > > I know this is a very general question. But if you guys had to specify > > system (could be one server or cluster), with sustainable transaction > > rate of 1.5M tps running postgres

Re: [PERFORM] heavly load system spec

2012-05-24 Thread Robert Haas
On Thu, Apr 5, 2012 at 11:39 AM, Gregg Jaskiewicz wrote: > I know this is a very general question. But if you guys had to specify > system (could be one server or cluster), with sustainable transaction > rate of 1.5M tps running postgresql, what configuration and hardware > would you be looking fo

Re: [PERFORM] Any disadvantages of using =ANY(ARRAY()) instead of IN?

2012-05-24 Thread geirB
I've encoutered similar issues myself (with UNION so far), so I tried to build a simple test case, which may or may not cover Clemens's case. Test case 1 and 2 illustrates the issue, and case 3-9 are variations. My observation: Looks like the optimizer cannot be close friends with both UNION and

Re: [PERFORM] Any disadvantages of using =ANY(ARRAY()) instead of IN?

2012-05-24 Thread geirB
Ah, forgot one query: WHERE IN is of course fast when we supply id's directly, but not when they are wrapped as array and UNNEST'ed in query 6. (previous post from me) -- Test 6b: Fast. WHERE IN(explicit id list) SELECT * FROM ( SELECT * FROM table1 UNION SELECT * FROM table1

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Stephen Frost
* Rajesh Kumar. Mallah (mal...@tradeindia.com) wrote: > We are running linux with kernel 3.2.X > (which has the lseek improvements) Ah, good. > Thanks for the reference , even i thought so (LockManager) , > but we are actually also running out db max connections (also) > ( which is currently at

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Rajesh Kumar. Mallah
- "Stephen Frost" wrote: | From: "Stephen Frost" | To: "Rajesh Kumar. Mallah" | Cc: pgsql-performance@postgresql.org | Sent: Thursday, May 24, 2012 9:27:37 PM | Subject: Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle | | Rajesh, | | * Rajesh Kumar. Mallah

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Rajesh Kumar. Mallah
| From: "Steve Crawford" | To: "Rajesh Kumar. Mallah" | Cc: "Andy Colson" , "Claudio Freire" , pgsql-performance@postgresql.org | Sent: Thursday, May 24, 2012 9:23:47 PM | Subject: Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle | | On 05/24/2012 05:58 AM, Raje

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Stephen Frost
Rajesh, * Rajesh Kumar. Mallah (mal...@tradeindia.com) wrote: > We are puzzled why the CPU and DISK I/O system are not being utilized > fully and would seek lists' wisdom on that. What OS is this? What kernel version? > just a thought, will it be a good idea to partition the host hardware >

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Steve Crawford
On 05/24/2012 05:58 AM, Rajesh Kumar. Mallah wrote: Dear Andy , Following the discussion on load average we are now investigating on some other parts of the stack (other than db). Essentially we are bumping up the limits (on appserver) so that more requests goes to the DB server. Which leads

Re: [PERFORM] pg_dump and thousands of schemas

2012-05-24 Thread Craig James
On Thu, May 24, 2012 at 12:06 AM, Hugo wrote: > Hi everyone, > > We have a production database (postgresql 9.0) with more than 20,000 > schemas > and 40Gb size. In the past we had all that information in just one schema > and pg_dump used to work just fine (2-3 hours to dump everything). Then we

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Rajesh Kumar. Mallah
Dear Andy , Following the discussion on load average we are now investigating on some other parts of the stack (other than db). Essentially we are bumping up the limits (on appserver) so that more requests goes to the DB server. | | Maybe you are hitting some locks? If its not IO and no

Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle

2012-05-24 Thread Andy Colson
On 05/24/2012 12:26 AM, Rajesh Kumar. Mallah wrote: - "Claudio Freire" wrote: | From: "Claudio Freire" | To: "Rajesh Kumar. Mallah" | Cc: pgsql-performance@postgresql.org | Sent: Thursday, May 24, 2012 9:23:43 AM | Subject: Re: [PERFORM] High load average in 64-core server , no I/O wait an

[PERFORM] pg_dump and thousands of schemas

2012-05-24 Thread Hugo
Hi everyone, We have a production database (postgresql 9.0) with more than 20,000 schemas and 40Gb size. In the past we had all that information in just one schema and pg_dump used to work just fine (2-3 hours to dump everything). Then we decided to split the database into schemas, which makes a l