Re: Bulk Insert into PostgreSQL

2018-07-05 Thread Srinivas Karthik V
Thanks for the link! Alternatively, when I am trying to create an index on a column of a table which is of size 400 GB, it is taking roughly 7 hrs. The index is created only on one column which is not a primary key. The query I am using is, create index on table (colname). I request your valuable

Re: Bulk Insert into PostgreSQL

2018-07-04 Thread Peter Geoghegan
On Tue, Jul 3, 2018 at 4:34 PM, Srinivas Karthik V wrote: > @Peter: I was indexing the primary key of all the tables in tpc-ds. Some of > the fact tables has multiple columns as part of the primary key. Also, most > of them are numeric type. Please see my mail to -hackers on suffix truncation: ht

Re: Bulk Insert into PostgreSQL

2018-07-03 Thread Ashwin Agrawal
On Sat, Jun 30, 2018 at 6:27 AM Craig Ringer wrote: > > You can also gain a bit by running with wal_level = minimal. On newer > version you can use UNLOGGED tables then convert them to logged, but that > won't be an option for 9.4. > Curious to know more on this does with standby also its faster

Re: Bulk Insert into PostgreSQL

2018-07-03 Thread Srinivas Karthik V
@Peter: I was indexing the primary key of all the tables in tpc-ds. Some of the fact tables has multiple columns as part of the primary key. Also, most of them are numeric type. On Mon, Jul 2, 2018 at 7:09 AM, Peter Geoghegan wrote: > On Sun, Jul 1, 2018 at 5:19 PM, Tsunakawa, Takayuki > wrote:

RE: Bulk Insert into PostgreSQL

2018-07-01 Thread Tsunakawa, Takayuki
From: Peter Geoghegan [mailto:p...@bowt.ie] > What kind of data was indexed? Was it a bigserial primary key, or > something else? Sorry, I don't remember it. But the table was for storing some machine usage data, and I don't think any sequence was used in the index. According to my faint memory

Re: Bulk Insert into PostgreSQL

2018-07-01 Thread Peter Geoghegan
On Sun, Jul 1, 2018 at 5:19 PM, Tsunakawa, Takayuki wrote: > 400 GB / 15 hours = 7.6 MB/s > > That looks too slow. I experienced a similar slowness. While our user tried > to INSERT (not COPY) a billion record, they reported INSERTs slowed down by > 10 times or so after inserting about 500 mil

RE: Bulk Insert into PostgreSQL

2018-07-01 Thread Tsunakawa, Takayuki
From: Srinivas Karthik V [mailto:skarthikv.i...@gmail.com] > I was using copy command to load. Removing the primary key constraint on > the table and then loading it helps a lot. In fact, a 400GB table was loaded > and the primary constraint was added in around 15 hours. Thanks for the > wonderful

Re: Bulk Insert into PostgreSQL

2018-06-30 Thread Craig Ringer
On 30 June 2018 at 06:47, Srinivas Karthik V wrote: > I was using copy command to load. Removing the primary key constraint on > the table and then loading it helps a lot. In fact, a 400GB table was > loaded and the primary constraint was added in around 15 hours. Thanks for > the wonderful sugg

Re: Bulk Insert into PostgreSQL

2018-06-29 Thread Srinivas Karthik V
I was using copy command to load. Removing the primary key constraint on the table and then loading it helps a lot. In fact, a 400GB table was loaded and the primary constraint was added in around 15 hours. Thanks for the wonderful suggestions. Regards, Srinivas Karthik On 28 Jun 2018 2:07 a.m.,

Re: Bulk Insert into PostgreSQL

2018-06-27 Thread Don Seiler
On Wed, Jun 27, 2018 at 6:25 AM, Pavel Stehule wrote: > > >> Other parameters are set to default value. Moreover, I have specified the >> primary key constraint during table creation. This is the only possible >> index being created before data loading and I am sure there are no other >> indexes

RE: Bulk Insert into PostgreSQL

2018-06-27 Thread ROS Didier
- SGBD Nanterre Picasso - E2 565D (aile nord-est) 32 Avenue Pablo Picasso 92000 Nanterre didier@edf.fr<mailto:didier@edf.fr> De : skarthikv.i...@gmail.com [mailto:skarthikv.i...@gmail.com] Envoyé : mercredi 27 juin 2018 13:19 À : pgsql-hack...@postgresql.org Objet : Bulk Inser

Re: Bulk Insert into PostgreSQL

2018-06-27 Thread Pavel Stehule
2018-06-27 13:18 GMT+02:00 Srinivas Karthik V : > Hi, > I am performing a bulk insert of 1TB TPC-DS benchmark data into PostgreSQL > 9.4. It's taking around two days to insert 100 GB of data. Please let me > know your suggestions to improve the performance. Below are the > configuration parameters

Bulk Insert into PostgreSQL

2018-06-27 Thread Srinivas Karthik V
Hi, I am performing a bulk insert of 1TB TPC-DS benchmark data into PostgreSQL 9.4. It's taking around two days to insert 100 GB of data. Please let me know your suggestions to improve the performance. Below are the configuration parameters I am using: shared_buffers =12GB maintainence_work_mem = 8