Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-15 Thread Matthew Wakeling
On Wed, 15 Apr 2009, Matthew Wakeling wrote: If anyone needs this code in Java, we have a version at http://www.intermine.org/ Download source code: http://www.intermine.org/wiki/SVNCheckout Javadoc: http://www.intermine.org/api/ Sorry, that should be http://www.flymine.org/api/ Matthew --

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-15 Thread Matthew Wakeling
On Tue, 14 Apr 2009, Stephen Frost wrote: What does your test harness currently look like, and what would you like to see to test the binary-format COPY? I'd be happy to write up the code necessary to implement binary-format COPY for this. If anyone needs this code in Java, we have a version a

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
Craig, * Craig Ringer (cr...@postnewspapers.com.au) wrote: > In other cases, binary-format COPY would be unsafe without some way to > determine remote endianness and sizeof(various types). As Tom mentioned already, the binary protocol is actually pretty well defined, and it's in network-byte-orde

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Tom Lane
Craig Ringer writes: > Unlike PQexecPrepared(...), binary-format COPY doesn't handle endian and > type size issues for you. You need to convert the data to the database > server's endianness and type sizes, but I don't think the PostgreSQL > protocol provides any way to find those out. The on-the

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Craig Ringer
Stephen Frost wrote: > * Matthew Wakeling (matt...@flymine.org) wrote: >> On Tue, 14 Apr 2009, Stephen Frost wrote: >>> Bacula should be using COPY for the batch data loads, so hopefully won't >>> suffer too much from having the fields split out. I think it would be >>> interesting to try doing PQ

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
* Matthew Wakeling (matt...@flymine.org) wrote: > On Tue, 14 Apr 2009, Stephen Frost wrote: >> Bacula should be using COPY for the batch data loads, so hopefully won't >> suffer too much from having the fields split out. I think it would be >> interesting to try doing PQexecPrepared with binary-fo

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Matthew Wakeling
On Tue, 14 Apr 2009, Stephen Frost wrote: Bacula should be using COPY for the batch data loads, so hopefully won't suffer too much from having the fields split out. I think it would be interesting to try doing PQexecPrepared with binary-format data instead of using COPY though. I'd be happy to

Re: [PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Stephen Frost
Craig, * Craig Ringer (cr...@postnewspapers.com.au) wrote: > I've been doing some testing for the Bacula project, which uses > PostgreSQL as one of the databases in which it stores backup catalogs. We also use Bacula with a PostgreSQL backend. > I've been evaluating a schema change for Bacula th

[PERFORM] INSERT times - same storage space but more fields -> much slower inserts

2009-04-14 Thread Craig Ringer
Hi I've been doing some testing for the Bacula project, which uses PostgreSQL as one of the databases in which it stores backup catalogs. Insert times are critical in this environment, as the app may insert millions of records a day. I've been evaluating a schema change for Bacula that takes a f