Re: [PERFORM] How to improve insert speed with index on text column

2012-02-29 Thread Robert Haas
On Sun, Feb 5, 2012 at 12:29 PM, Saurabh wrote: > My intention to keep autovacuum as off is bulk loading only. I was > thinking after bullk load I will change it. > > I changed wal_buffer from 5MB to 16MB but I got same performance that > I got with 5MB (even less). Does it help if you create the

Re: [PERFORM] How to improve insert speed with index on text column

2012-02-03 Thread Scott Marlowe
On Tue, Jan 31, 2012 at 1:20 PM, Merlin Moncure wrote: > yeah -- postgresql.conf settings are not going to play a big role here unless: > *) you defer index build to the end of the load, and do CREATE INDEX > and crank maintenance_work_mem > *) you are doing lots of transactions and relax your syn

Re: [PERFORM] How to improve insert speed with index on text column

2012-02-03 Thread Kevin Grittner
Saurabh wrote: > wal_buffers = 5MB As as already been suggested, use 16MB (or if the version you're using supports it, the default of -1); > autovacuum = off If the only activity while this is off is a bulk load, that might be OK, but be sure *not* to leave this off. You will almost certa

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Claudio Freire
On Wed, Feb 1, 2012 at 12:49 AM, Claudio Freire wrote: > On Wed, Feb 1, 2012 at 12:29 AM, Rosser Schwarz > wrote: >> Remember, DDL is transactional in PostgreSQL.  In principle, you >> should be able to drop the index, do your inserts, and re-create the >> index without affecting concurrent users

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Claudio Freire
On Wed, Feb 1, 2012 at 12:29 AM, Rosser Schwarz wrote: > Remember, DDL is transactional in PostgreSQL.  In principle, you > should be able to drop the index, do your inserts, and re-create the > index without affecting concurrent users, if you do all of that inside > an explicit transaction.  Doin

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Rosser Schwarz
On Mon, Jan 30, 2012 at 9:46 AM, Saurabh wrote: > I can not create the index after insertion because user can > search the data as well while insertion. Remember, DDL is transactional in PostgreSQL. In principle, you should be able to drop the index, do your inserts, and re-create the index with

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Jeff Janes
On Tue, Jan 31, 2012 at 10:46 AM, Josh Berkus wrote: > >> Shared buffers is the cache maintained by PostgreSQL. All all the data >> that you read/write need to go through shared buffers. > > While this is technically true, I need to point out that you generally > increase shared_buffers for high c

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Merlin Moncure
On Tue, Jan 31, 2012 at 12:46 PM, Josh Berkus wrote: > >> Shared buffers is the cache maintained by PostgreSQL. All all the data >> that you read/write need to go through shared buffers. > > While this is technically true, I need to point out that you generally > increase shared_buffers for high c

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Josh Berkus
> Shared buffers is the cache maintained by PostgreSQL. All all the data > that you read/write need to go through shared buffers. While this is technically true, I need to point out that you generally increase shared_buffers for high concurrency, and for reads, not for writes, especially for row-

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Tomas Vondra
On 31 Leden 2012, 10:29, Saurabh wrote: > I changed the configuration in postgresql.conf. Following are the > changed parameters: > > shared_buffers = 1GB > maintenance_work_mem = 50MB > checkpoint_segments = 64 > wal_buffers = 5MB > autovacuum = off > > Insert the records in the database and got a

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-31 Thread Saurabh
I changed the configuration in postgresql.conf. Following are the changed parameters: shared_buffers = 1GB maintenance_work_mem = 50MB checkpoint_segments = 64 wal_buffers = 5MB autovacuum = off Insert the records in the database and got a very good performance it is increased by 6 times. Can yo

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-30 Thread Jeff Janes
On Mon, Jan 30, 2012 at 9:46 AM, Saurabh wrote: > Thank you for the information. > > Schema of table is: > > ID                         bigint > company_name     text > data_set                text > time                      timestamp > Date                     date > > Length of company_name is

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-30 Thread Andy Colson
On 1/30/2012 3:27 AM, Saurabh wrote: Hi all, I am using Postgresql database for our project and doing some performance testing. We need to insert millions of record with indexed columns. We have 5 columns in table. I created index on integer only then performance is good but when I created index

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-30 Thread Claudio Freire
On Mon, Jan 30, 2012 at 2:46 PM, Saurabh wrote: > max_connections = 100 > shared_buffers = 32MB > wal_buffers = 1024KB > checkpoint_segments = 3 That's a default config isn't it? You'd do well to try and optimize it for your system. The defaults are really, reeallly conservative. You should als

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-30 Thread Saurabh
Thank you for the information. Schema of table is: ID bigint company_name text data_settext time timestamp Date date Length of company_name is not known so it is of datatype text. I need to build the index on co

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-30 Thread Jeff Janes
On Mon, Jan 30, 2012 at 1:27 AM, Saurabh wrote: > Hi all, > > I am using Postgresql database for our project and doing some > performance testing. We need to insert millions of record with indexed > columns. We have 5 columns in table. I created index on integer only > then performance is good but

Re: [PERFORM] How to improve insert speed with index on text column

2012-01-30 Thread Claudio Freire
On Mon, Jan 30, 2012 at 6:27 AM, Saurabh wrote: > Hi all, > > I am using Postgresql database for our project and doing some > performance testing. We need to insert millions of record with indexed > columns. We have 5 columns in table. I created index on integer only > then performance is good but