Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
Craig, * Craig Ringer (cr...@postnewspapers.com.au) wrote: > In other cases, binary-format COPY would be unsafe without some way to > determine remote endianness and sizeof(various types). As Tom mentioned already, the binary protocol is actually pretty well defined, and it's in network-byte-orde

[PERFORM] error updating a very large table

2009-04-14 Thread Brian Cox
ts_defect_meta_values has 460M rows. The following query, in retrospect not too surprisingly, runs out of memory on a 32 bit postgres: update ts_defect_meta_values set ts_defect_date=(select ts_occur_date from ts_defects where ts_id=ts_defect_id) I changed the logic to update the table in 1M

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Tom Lane
Craig Ringer writes: > Unlike PQexecPrepared(...), binary-format COPY doesn't handle endian and > type size issues for you. You need to convert the data to the database > server's endianness and type sizes, but I don't think the PostgreSQL > protocol provides any way to find those out. The on-the

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Craig Ringer
Stephen Frost wrote: > * Matthew Wakeling (matt...@flymine.org) wrote: >> On Tue, 14 Apr 2009, Stephen Frost wrote: >>> Bacula should be using COPY for the batch data loads, so hopefully won't >>> suffer too much from having the fields split out. I think it would be >>> interesting to try doing PQ

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
* Matthew Wakeling (matt...@flymine.org) wrote: > On Tue, 14 Apr 2009, Stephen Frost wrote: >> Bacula should be using COPY for the batch data loads, so hopefully won't >> suffer too much from having the fields split out. I think it would be >> interesting to try doing PQexecPrepared with binary-fo

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Matthew Wakeling
On Tue, 14 Apr 2009, Stephen Frost wrote: Bacula should be using COPY for the batch data loads, so hopefully won't suffer too much from having the fields split out. I think it would be interesting to try doing PQexecPrepared with binary-format data instead of using COPY though. I'd be happy to

Re: [PERFORM] difficulties with time based queries

2009-04-14 Thread Nikolas Everett
> >If you often do range queries on date, consider partitioning your > table by date (something like 1 partition per month). >Of course, if you also often do range queries on something other > than date, and uncorrelated, forget it. If you pick your partition to line up with your

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
Craig, * Craig Ringer (cr...@postnewspapers.com.au) wrote: > I've been doing some testing for the Bacula project, which uses > PostgreSQL as one of the databases in which it stores backup catalogs. We also use Bacula with a PostgreSQL backend. > I've been evaluating a schema change for Bacula th

Re: [PERFORM] Nested query performance issue

2009-04-14 Thread Merlin Moncure
2009/4/9 Віталій Тимчишин : > OK, got to my postgres. Here you are: > > create or replace function explode_array(in_array anyarray) returns setof > anyelement as > $$ >     select ($1)[s] from generate_series(1,array_upper($1, 1)) as s; > $$ > language sql immutable; > in 8.4, this will be replace

Re: [PERFORM] Nested query performance issue

2009-04-14 Thread Glenn Maynard
On Tue, Apr 14, 2009 at 5:33 AM, Matthew Wakeling wrote: >> It's about 10% faster for me.  I'm surprised the planner can't figure >> out that this join is redundant. > > Because the join isn't redundant? You're making the assumption that for > every score.game_id there is exactly one game.id that

Re: [PERFORM] Shouldn't the planner have a higher cost for reverse index scans?

2009-04-14 Thread Matthew Wakeling
On Fri, 10 Apr 2009, Tom Lane wrote: Josh Berkus writes: Not as far as I can tell. It looks to me like the planner is assuming that a forwards index scan and a reverse index scan will have the same cost. Right, because they do. If you think otherwise, demonstrate it. They do when the cor

Re: [PERFORM] Nested query performance issue

2009-04-14 Thread Matthew Wakeling
On Thu, 9 Apr 2009, Glenn Maynard wrote: On Thu, Apr 9, 2009 at 7:29 AM, Heikki Linnakangas wrote: SELECT s.* FROM score s, game g WHERE s.game_id = g.id AND  s.id IN (    SELECT s2.id FROM score s2 WHERE s2.game_id=g.id ORDER BY s2.score DESC LIMIT 1  ); You don't really need the join with ga

Re: [PERFORM] difficulties with time based queries

2009-04-14 Thread PFC
What can I do to prevent the index from getting bloated, or in whatever state it was in? What else can I do to further improve queries on this table? Someone suggested posting details of my conf file. Which settings are most likely to be useful for this? If you often do range queries on

[PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Craig Ringer
Hi I've been doing some testing for the Bacula project, which uses PostgreSQL as one of the databases in which it stores backup catalogs. Insert times are critical in this environment, as the app may insert millions of records a day. I've been evaluating a schema change for Bacula that takes a f