Re: [PERFORM] PG optimization question

2010-01-11 Thread Pierre Frédéric Caillau d
On Sun, 10 Jan 2010 19:45:32 +0100, Robert Haas wrote: 2010/1/10 Pierre Frédéric Caillaud : If you transfer (delete from staging, insert into archive) in one transaction , then it will be always visible in exactly one of them, and exatly once in a view over both staging and archive(s).  

Re: [PERFORM] PG optimization question

2010-01-10 Thread Robert Haas
2010/1/10 Pierre Frédéric Caillaud : > >> If you transfer (delete from staging, insert into archive) in one >> transaction , then it will be always visible in exactly one of them, >> and exatly once in a view over both staging and archive(s). > >        Does the latest version implement this : > >

Re: [PERFORM] PG optimization question

2010-01-10 Thread Pierre Frédéric Caillau d
If you transfer (delete from staging, insert into archive) in one transaction , then it will be always visible in exactly one of them, and exatly once in a view over both staging and archive(s). Does the latest version implement this : INSERT INTO archive (...) DELETE FROM staging WHE

Re: [PERFORM] PG optimization question

2010-01-09 Thread Craig Ringer
On 9/01/2010 6:32 PM, Nickolay wrote: Hi 2 all, Here is my typical configuration: 1(2) GB of RAM, HP ML 350(150) series server, SATA raid, Linux. I have 1 big table (called "archive") which contains short text messages with a plenty of additional service info. Currently this table contains more

Re: [PERFORM] PG optimization question

2010-01-09 Thread Ludwik Dylag
2010/1/9 Nickolay > Okay, I see your point with staging table. That's a good idea! > The only problem I see here is the transfer-to-archive-table process. As > you've correctly noticed, the system is kind of a real-time and there can be > dozens of processes writing to the staging table, i cannot

Re: [PERFORM] PG optimization question

2010-01-09 Thread Nickolay
Okay, I see your point with staging table. That's a good idea! The only problem I see here is the transfer-to-archive-table process. As you've correctly noticed, the system is kind of a real-time and there can be dozens of processes writing to the staging table, i cannot see how to make the tra

Re: [PERFORM] PG optimization question

2010-01-09 Thread Pierre Frédéric Caillau d
That may help with the queries speed (not a problem now), but we'll then have to add UNION statement for daily staging table for other 5% of requests, right? And there would be a moment when daily message is in archive table AND in daily table (while transferring from daily table to archi

Re: [PERFORM] PG optimization question

2010-01-09 Thread Nickolay
That may help with the queries speed (not a problem now), but we'll then have to add UNION statement for daily staging table for other 5% of requests, right? And there would be a moment when daily message is in archive table AND in daily table (while transferring from daily table to archive). O

Re: [PERFORM] PG optimization question

2010-01-09 Thread Kenneth Marshall
On Sat, Jan 09, 2010 at 03:42:08PM +0300, Nickolay wrote: > I do not see any way to normalize this table anymore. it's size is 4Gig for > ~4M rows, i.e. 1Kb per row, i think it's ok. > Also there are 2 indexes: by date_time and by a couple of service fields > (total index size is 250Mb now). > I

Re: [PERFORM] PG optimization question

2010-01-09 Thread Nickolay
I do not see any way to normalize this table anymore. it's size is 4Gig for ~4M rows, i.e. 1Kb per row, i think it's ok. Also there are 2 indexes: by date_time and by a couple of service fields (total index size is 250Mb now). I think i'll be going to partition by months (approx. 1M rows or 1Gig

Re: [PERFORM] PG optimization question

2010-01-09 Thread Thomas Kellerer
Nickolay wrote on 09.01.2010 11:24: it would be pretty hard with one table because of blocking What do you man with "because of blocking"? Thomas -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mail

Re: [PERFORM] PG optimization question

2010-01-09 Thread Grzegorz Jaśkiewicz
maybe that 'one big table' needs something called 'normalisation' first. See how much that will shed off. You might be surprised. The partitioning needs to be done by some constant intervals, of time - in your case. Whatever suits you, I would suggest to use the rate that will give you both ease of

[PERFORM] PG optimization question

2010-01-09 Thread Nickolay
Hi 2 all, Here is my typical configuration: 1(2) GB of RAM, HP ML 350(150) series server, SATA raid, Linux. I have 1 big table (called "archive") which contains short text messages with a plenty of additional service info. Currently this table contains more than 4M rows for a period of 4,5 m

[PERFORM] PG optimization question

2010-01-09 Thread Nickolay
Hi 2 all, Here is my typical configuration: 1(2) GB of RAM, HP ML 350(150) series server, SATA raid, Linux. I have 1 big table (called "archive") which contains short text messages with a plenty of additional service info. Currently this table contains more than 4M rows for a period of 4,5 m