Re: [External] Multiple COPY on the same table

2018-08-20 Thread Ravi Krishna
1. The tables has no indexes at the time of load.2.  The create table and copy are in the same transaction. So I guess that's pretty much it.  I understand the long time it takes as some of the tables have 400+ million rows.Also the env is a container and since this is currently a POC system , n

Re: [External] Multiple COPY on the same table

2018-08-20 Thread Ron
Maybe he just has a large file that needs to be loaded into a table... On 08/20/2018 11:47 AM, Vijaykumar Jain wrote: Hey Ravi, What is the goal you are trying to achieve here. To make pgdump/restore faster? To make replication faster? To make backup faster ? Also no matter how small you split

Re: [External] Multiple COPY on the same table

2018-08-20 Thread Vijaykumar Jain
I guess this should help you, Ravi. https://www.postgresql.org/docs/10/static/populate.html On 8/20/18, 10:30 PM, "Christopher Browne" wrote: On Mon, 20 Aug 2018 at 12:53, Ravi Krishna wrote: > > What is the goal you are trying to achieve here. > > To make pgdump/restore

Re: [External] Multiple COPY on the same table

2018-08-20 Thread Christopher Browne
On Mon, 20 Aug 2018 at 12:53, Ravi Krishna wrote: > > What is the goal you are trying to achieve here. > > To make pgdump/restore faster? > > To make replication faster? > > To make backup faster ? > > None of the above. > > We got csv files from external vendor which are 880GB in total size, in

Re: [External] Multiple COPY on the same table

2018-08-20 Thread Ravi Krishna
> What is the goal you are trying to achieve here. > To make pgdump/restore faster? > To make replication faster? > To make backup faster ? None of the above.  We got csv files from external vendor which are 880GB in total size, in 44 files.  Some of the large tables had COPY running for severa

Re: [External] Multiple COPY on the same table

2018-08-20 Thread Vijaykumar Jain
Hey Ravi, What is the goal you are trying to achieve here. To make pgdump/restore faster? To make replication faster? To make backup faster ? Also no matter how small you split the files into, if network is your bottleneck then I am not sure you can attain n times the benefit my simply sending