Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-15 Thread Matthew Wakeling
On Wed, 15 Apr 2009, Matthew Wakeling wrote: If anyone needs this code in Java, we have a version at http://www.intermine.org/ Download source code: http://www.intermine.org/wiki/SVNCheckout Javadoc: http://www.intermine.org/api/ Sorry, that should be http://www.flymine.org/api/ Matthew --

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-15 Thread Matthew Wakeling
On Tue, 14 Apr 2009, Stephen Frost wrote: What does your test harness currently look like, and what would you like to see to test the binary-format COPY? I'd be happy to write up the code necessary to implement binary-format COPY for this. If anyone needs this code in Java, we have a version a

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
Craig, * Craig Ringer (cr...@postnewspapers.com.au) wrote: > In other cases, binary-format COPY would be unsafe without some way to > determine remote endianness and sizeof(various types). As Tom mentioned already, the binary protocol is actually pretty well defined, and it's in network-byte-orde

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Tom Lane
Craig Ringer writes: > Unlike PQexecPrepared(...), binary-format COPY doesn't handle endian and > type size issues for you. You need to convert the data to the database > server's endianness and type sizes, but I don't think the PostgreSQL > protocol provides any way to find those out. The on-the

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Craig Ringer
Stephen Frost wrote: > * Matthew Wakeling (matt...@flymine.org) wrote: >> On Tue, 14 Apr 2009, Stephen Frost wrote: >>> Bacula should be using COPY for the batch data loads, so hopefully won't >>> suffer too much from having the fields split out. I think it would be >>> interesting to try doing PQ

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
* Matthew Wakeling (matt...@flymine.org) wrote: > On Tue, 14 Apr 2009, Stephen Frost wrote: >> Bacula should be using COPY for the batch data loads, so hopefully won't >> suffer too much from having the fields split out. I think it would be >> interesting to try doing PQexecPrepared with binary-fo

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Matthew Wakeling
On Tue, 14 Apr 2009, Stephen Frost wrote: Bacula should be using COPY for the batch data loads, so hopefully won't suffer too much from having the fields split out. I think it would be interesting to try doing PQexecPrepared with binary-format data instead of using COPY though. I'd be happy to

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
Craig, * Craig Ringer (cr...@postnewspapers.com.au) wrote: > I've been doing some testing for the Bacula project, which uses > PostgreSQL as one of the databases in which it stores backup catalogs. We also use Bacula with a PostgreSQL backend. > I've been evaluating a schema change for Bacula th

[PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Craig Ringer
Hi I've been doing some testing for the Bacula project, which uses PostgreSQL as one of the databases in which it stores backup catalogs. Insert times are critical in this environment, as the app may insert millions of records a day. I've been evaluating a schema change for Bacula that takes a f

Re: [PERFORM] Insert Times

2004-01-27 Thread Alan Stange
width=0) (actual time=0.01..0.02 rows=1 loops=1) Total runtime: 0.12 msec -Original Message- From: Leeuw van der, Tim [mailto:[EMAIL PROTECTED] Sent: Tue 1/27/2004 12:38 AM To: PC Drew; [EMAIL PROTECTED] Cc: Subject:RE: [PERFORM] Insert Times Hi, My personal feeling on t

Re: [PERFORM] Insert Times

2004-01-27 Thread PC Drew
OTECTED] Sent: Tue 1/27/2004 12:38 AM To: PC Drew; [EMAIL PROTECTED] Cc: Subject:RE: [PERFORM] Insert Times Hi, My personal feeling on this is, that the long time taken for the first query is for loading all sorts of libraries, JVM startup overhead etc. What if you first do som

Re: [PERFORM] Insert Times

2004-01-27 Thread Leeuw van der, Tim
Hi, My personal feeling on this is, that the long time taken for the first query is for loading all sorts of libraries, JVM startup overhead etc. What if you first do some SELECT (whatever), on a different table, to warm up the JVM and the database? regards, --Tim THIS COMMUNICATION MAY CONTAI

Re: [PERFORM] Insert Times

2004-01-26 Thread Brad Gawne
lf Of PC Drew Sent: January 26, 2004 16:00 To: [EMAIL PROTECTED] Subject: [PERFORM] Insert Times I'm trying to track down some performance issues where time (in milliseconds) is critical. One thing that I've noticed is that it seems like the first insert takes an inordinate amount of

[PERFORM] Insert Times

2004-01-26 Thread PC Drew
I'm trying to track down some performance issues where time (in milliseconds) is critical. One thing that I've noticed is that it seems like the first insert takes an inordinate amount of time. Inserts after the first one are acceptable. My production environment is like this: Solaris 9 JBo