Re: [GENERAL] Streaming large data into postgres [WORM like applications]

2007-05-12 Thread Kevin Hunter
At 8:49p on 12 May 2007, Dhaval Shah wrote: > That leads to the question, can the data be compressed? Since the data > is very similar, any compression would result in some 6x-10x > compression. Is there a way to identify which partitions are in which > data files and compress them until they are a

Re: [GENERAL] Streaming large data into postgres [WORM like applications]

2007-05-12 Thread Dhaval Shah
Consolidating my responses in one email. 1. The total data that is expected is some 1 - 1.5 Tb a day. 75% of the data comes in a period of 10 hours. Rest 25% comes in the 14 hours. Of course there are ways to smooth the load patterns, however the current scenario is as explained. 2 I do expect t

Re: [GENERAL] [ADMIN] increasing of the shared memory does not solve the problem of "OUT of shared memory"

2007-05-12 Thread Jim C. Nasby
On Fri, May 11, 2007 at 04:58:28PM +0300, Sorin N. Ciolofan wrote: > I increased significantly the number of shared buffers from 3000 to 100 000 > (80Mb) BTW, 100,000 shared buffers is actually 800MB, not 80. -- Jim Nasby[EMAIL PROTECTED] EnterpriseDB

Re: [GENERAL] typical schema for a forum?

2007-05-12 Thread David Lee Lambert
On Wednesday 09 May 2007 04:49, Louis-David Mitterrand wrote: > I'm trying to implement a forum with mason and postgresql. What is the > typical database schema of a forum (threaded or flat) application? Try downloading and installing a few free forum packages: phpbb phorum bugzilla redm

[GENERAL] Winsock error 10035 while trying to upgrade from 8.0 to 8.2

2007-05-12 Thread Cyril VELTER
I'm trying to upgrade a pretty big database (60G) from postgres 8.0 to postgres 8.2 on windows 2000 Server (both version running on the same machine on different ports). During the migration process, I always get an error at some point (never the same) : LOG: could not receive

[GENERAL] autovacuum

2007-05-12 Thread Tom Allison
This database is working GREAT!!! I noticed that under 8.2 the autovacuum isn't running (logging) every 60s like I'm used to seeing. I pretty much just took the defaults in the postgresql.conf file since that's always seemed to work before. I'm not making a lot of changes to the database ri

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Tom Allison
I think I fixed the rest of my sql statements with the following: insert into tokens (token)select values.token from (values TOKEN_LIST_STRING ) as values(token) left outer join tokens t using (token) where t.token_idx is null insert into user_token(user_idx, token_idx)select $self->{user_idx},

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Tom Allison
Thank you very much for all your help!!! Solved this one rather nicely: my $glue = q{'),(E'}; my $string = "(E'" . join($glue, map{quotemeta } @$tokens) . "')"; my $sql =<( select token_idx from user_token where user_idx = $self-> {user_idx}) "ut" using (token_idx) SQL $sql =~ s/TOK

Re: [GENERAL] Missing magic block

2007-05-12 Thread Tom Lane
Mario Munda <[EMAIL PROTECTED]> writes: > I've changed: > #ifdef PG_MODULE_MAGIC > PG_MODULE_MAGIC; > #endif > to: > #define PG_MODULE_MAGIC ; Uh, that turns it into a complete no-op. It should just be PG_MODULE_MAGIC; If you tried that and it didn't compile, that's proof that you're using pr

Re: [GENERAL] Partitioning on IS NULL / IS NOT NULL not supported?

2007-05-12 Thread Tom Lane
Aleksander Kmetec <[EMAIL PROTECTED]> writes: > I'm trying to split a table into 2 partitions based on whether a field's > value is NULL or NOT NULL, but when I run > "EXPLAIN SELECT * FROM tab WHERE version IS NULL" it shows that both > partitions are being scanned, not just the one > which co

[GENERAL] TWO SAME TABLES, ONE UPDATED. HOW TO SYNC THE OTHER?

2007-05-12 Thread L. Berger
Hello I have two tables -- A and B. The structure of both is the same. Only, B has many indexes and is used for heavy duty SELECTs. On the other hand, A only accepts heavy duty INSERTs, so has only one primary key index. So my DB design is such that A is only an INSERT table. Periodically, say ev

Re: [GENERAL] xml to db converter

2007-05-12 Thread Jerry
On May 10, 6:02 pm, [EMAIL PROTECTED] (garry saddington) wrote: > Does anyone know of a tool that will generate a postgres database schema > given an xml schema(xsd). Have triedxmlspywhich says it does so but it > only has limited postgres support and then it crashes. > regards > Garry > >

Re: [GENERAL] Missing magic block

2007-05-12 Thread Mario Munda
I've changed: #ifdef PG_MODULE_MAGIC PG_MODULE_MAGIC; #endif to: #define PG_MODULE_MAGIC ; and the .so does compile, but the database is still reporting the same error when i try to CREATE FUNCTION: ERROR: incompatible library "/home/mario/tests/psql_c_func/ test_func.so": missing magic block

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Gregory Stark
"Tom Allison" <[EMAIL PROTECTED]> writes: > This is getting really ugly... > it won't finish in less than .. minutes. > > spam=> explain select u2.token_idx, t.token_idx, foo.token from > tokens t left outer join user_token u1 using (token_idx), > users u left outer join user_token u2 using (user

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Tom Allison
I agree I'm going in the wrong direction. in a Venn sort of way, what I'm trying to do is: values(...) --> left outer --> tokens --> left outer --> (user_tokens where user_tokens.user_idx = users.user_idx and users.user_idx = 4) To give me a list of all values || any matching token || any m

[GENERAL] Partitioning on IS NULL / IS NOT NULL not supported?

2007-05-12 Thread Aleksander Kmetec
Hi, I'm trying to split a table into 2 partitions based on whether a field's value is NULL or NOT NULL, but when I run "EXPLAIN SELECT * FROM tab WHERE version IS NULL" it shows that both partitions are being scanned, not just the one which contains rows where version is null. Is this not sup

Re: [GENERAL] Database transaction related

2007-05-12 Thread Jasbinder Singh Bali
could you please elaborate this concept of queue table? ~Jas On 5/12/07, Martijn van Oosterhout <[EMAIL PROTECTED]> wrote: On Fri, May 11, 2007 at 04:07:10PM -0400, Jasbinder Singh Bali wrote: > I have a transaction in postgres database as follows: > > 1 Insert in table xyz > 2 trigger fired on

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Richard Broersma Jr
--- Tom Allison <[EMAIL PROTECTED]> wrote: > This is getting really ugly... > it won't finish in less than .. minutes. > > spam=> explain select u2.token_idx, t.token_idx, foo.token from > tokens t left outer join user_token u1 using (token_idx), > users u left outer join user_token u2 using (us

Re: [GENERAL] Streaming large data into postgres [WORM like applications]

2007-05-12 Thread Lincoln Yeoh
At 04:43 AM 5/12/2007, Dhaval Shah wrote: 1. Large amount of streamed rows. In the order of @50-100k rows per second. I was thinking that the rows can be stored into a file and the file then copied into a temp table using copy and then appending those rows to the master table. And then dropping

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Tom Allison
This is getting really ugly... it won't finish in less than .. minutes. spam=> explain select u2.token_idx, t.token_idx, foo.token from tokens t left outer join user_token u1 using (token_idx), users u left outer join user_token u2 using (user_idx), (values('one'),('want'),('examine'),('three'))

Re: [GENERAL] stuck on values in 8.2

2007-05-12 Thread Gregory Stark
"Tom Allison" <[EMAIL PROTECTED]> writes: > OK, after reviewing many emails and what I was trying to do I upgraded from > 8.2. > > Seems to work as it did in 8.1 which is a good start. > > I'm doing all of this so I can use the 'values' that was described as being > something like: > > select *

Re: [GENERAL] postgres upgrade

2007-05-12 Thread Martijn van Oosterhout
On Sat, May 12, 2007 at 09:36:45AM -0400, Tom Allison wrote: > I'm trying to upgrade from 8.1 to 8.2. > > Apparently the upgrade I just ran through installed a second database > on my server. > > I was hoping that the debian configuration would migrate this for me. > I recall there was once a t

[GENERAL] stuck on values in 8.2

2007-05-12 Thread Tom Allison
OK, after reviewing many emails and what I was trying to do I upgraded from 8.2. Seems to work as it did in 8.1 which is a good start. I'm doing all of this so I can use the 'values' that was described as being something like: select * from (values ('one','two','three')) "foo"; But inste

Re: [GENERAL] postgres upgrade

2007-05-12 Thread Ray Stell
On Sat, May 12, 2007 at 09:36:45AM -0400, Tom Allison wrote: > Meanwhile, can I just use pg_dumpall to pull from 8.1 and then reload > into 8.2? http://www.postgresql.org/docs/faqs.FAQ.html#item3.6 ---(end of broadcast)--- TIP 5: don't forget to

[GENERAL] postgres upgrade

2007-05-12 Thread Tom Allison
I'm trying to upgrade from 8.1 to 8.2. Apparently the upgrade I just ran through installed a second database on my server. I was hoping that the debian configuration would migrate this for me. I recall there was once a time when it would ask you about moving data from old to new databases.

Re: [GENERAL] Fault Tolerant Postgresql (two machines, two postmasters, one disk array)

2007-05-12 Thread Paul Lambert
Ron Johnson wrote: On 05/12/07 01:51, Paul Lambert wrote: Sure it's been around nearly since the dawn of time, but it's still an actively developed operating system. I've finally got my Alpha server at home up and running now too, and I What are you running? Off hand I couldn't tell you -

Re: [GENERAL] Streaming large data into postgres [WORM like applications]

2007-05-12 Thread Ron Johnson
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/11/07 21:35, Dhaval Shah wrote: > I do care about the following: > > 1. Basic type checking > 2. Knowing failed inserts. > 3. Non-corruption > 4. Macro transactions. That is a minimal read consistency. > > The following is not necessary > > 1.

Re: [GENERAL] Fault Tolerant Postgresql (two machines, two postmasters, one disk array)

2007-05-12 Thread Ron Johnson
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/12/07 01:51, Paul Lambert wrote: > Ron Johnson wrote: >> >> Dinosaurist? >> >> The big systems we use were last upgraded 5ish years ago, and are >> scheduled (eventually) to be replaced with Oracle on Linux. >> > > We've got some pretty new Alph

Re: [GENERAL] Database transaction related

2007-05-12 Thread Martijn van Oosterhout
On Fri, May 11, 2007 at 04:07:10PM -0400, Jasbinder Singh Bali wrote: > I have a transaction in postgres database as follows: > > 1 Insert in table xyz > 2 trigger fired on table xyz after insert > 3 Trigger calls a function func > 4 func is an implementation of a client socket in perl > > 1-4 ha

Re: [GENERAL] Fault Tolerant Postgresql (two machines, two postmasters, one disk array)

2007-05-12 Thread Joris Dobbelsteen
>-Original Message- >From: [EMAIL PROTECTED] >[mailto:[EMAIL PROTECTED] On Behalf Of Bruno >Wolff III >Sent: vrijdag 11 mei 2007 21:18 >To: John Gateley >Cc: pgsql-general@postgresql.org >Subject: Re: [GENERAL] Fault Tolerant Postgresql (two >machines, two postmasters, one disk array) >