Re: [PERFORM] Guidance Requested - Bulk Inserting + Queries

2011-12-22 Thread Jeff Janes
On Wed, Dec 21, 2011 at 6:30 PM, Benjamin Johnson wrote: > Jeff, > > Sorry for the delayed response.  Please see (some) answers inline. > > On 12/1/2011 9:06 AM, Jeff Janes wrote: >> On Wed, Nov 30, 2011 at 7:27 AM, Benjamin Johnson >> Why shove it in as fast as you can?  If you want to both read

Re: [PERFORM] Guidance Requested - Bulk Inserting + Queries

2011-12-21 Thread Benjamin Johnson
Jeff, Sorry for the delayed response. Please see (some) answers inline. On 12/1/2011 9:06 AM, Jeff Janes wrote: > On Wed, Nov 30, 2011 at 7:27 AM, Benjamin Johnson > wrote: >> Experts, >> >> Quick Summary: data can now be inserted very quickly via COPY + removing >> indexes, but is there a desi

Re: [PERFORM] Guidance Requested - Bulk Inserting + Queries

2011-12-01 Thread Jeff Janes
On Wed, Nov 30, 2011 at 7:27 AM, Benjamin Johnson wrote: > Experts, > > Quick Summary: data can now be inserted very quickly via COPY + removing > indexes, but is there a design or some tricks to still allow someone to > query while the partition is still active and 'hot' ? > > - Postgres 9.1 > -

Re: [PERFORM] Guidance Requested - Bulk Inserting + Queries

2011-11-30 Thread Benjamin Johnson
We're trying to split the current day into hourly tables so that the size of the indexes that are popular is much lower and therefore we can support more rows across the day. We also are using numerics where we could be using bigints, so we're going to also work on that to see how much smaller we

Re: [PERFORM] Guidance Requested - Bulk Inserting + Queries

2011-11-30 Thread Leonardo Francalanci
> We now found (thanks Andres and Snow-Man in #postgresql) that in our > tests, after the indexes get too large performance drops signficantly > and our system limps forward due to  disk reads (presumably for the > indexes).  If we remove the indexes, performance for our entire sample > test is gre

[PERFORM] Guidance Requested - Bulk Inserting + Queries

2011-11-30 Thread Benjamin Johnson
Experts, Quick Summary: data can now be inserted very quickly via COPY + removing indexes, but is there a design or some tricks to still allow someone to query while the partition is still active and 'hot' ? - Postgres 9.1 - Windows 7 (64-bit) , although this is just for the current test and coul