Re: Logical Replication speed-up initial data

2021-08-05 Thread Jeff Janes
On Thu, Aug 5, 2021 at 12:57 AM Nikhil Shetty wrote: > Hi, > > Thank you for the suggestion. > > We tried by dropping indexes and it worked faster compared to what we saw > earlier. We wanted to know if anybody has done any other changes that helps > speed-up initial data load without dropping in

Re: Logical Replication speed-up initial data

2021-08-05 Thread Vijaykumar Jain
On Fri, 6 Aug 2021 at 00:15, Nikhil Shetty wrote: > Hi Vijaykumar, > > Thanks for the details. > In this method you are saying the pg_basebackup will make the initial load > faster ? > We intend to bring only a few tables. Using pg_basebackup will clone an > entire instance. > yeah. In that case

Re: Logical Replication speed-up initial data

2021-08-05 Thread Nikhil Shetty
Hi Vijaykumar, Thanks for the details. In this method you are saying the pg_basebackup will make the initial load faster ? We intend to bring only a few tables. Using pg_basebackup will clone an entire instance. Thanks, Nikhil On Thu, Aug 5, 2021 at 7:57 PM Vijaykumar Jain < vijaykumarjain.git

Re: Logical Replication speed-up initial data

2021-08-05 Thread Nikhil Shetty
Hi Avinash, Thank you for the detailed explanation. Indexes were dropped on the destination to increase initial data load speed. We cannot stop the App on source and it is highly transactional. I had thought about this method but I am not sure after the pg_restore from where the logical replicati

Re: Logical Replication speed-up initial data

2021-08-05 Thread Avinash Kumar
Hi, On Thu, Aug 5, 2021 at 11:28 AM Vijaykumar Jain < vijaykumarjain.git...@gmail.com> wrote: > On Thu, 5 Aug 2021 at 10:27, Nikhil Shetty wrote: > >> Hi, >> >> Thank you for the suggestion. >> >> We tried by dropping indexes and it worked faster compared to what we saw >> earlier. We wanted to

Re: Logical Replication speed-up initial data

2021-08-05 Thread Vijaykumar Jain
On Thu, 5 Aug 2021 at 10:27, Nikhil Shetty wrote: > Hi, > > Thank you for the suggestion. > > We tried by dropping indexes and it worked faster compared to what we saw > earlier. We wanted to know if anybody has done any other changes that helps > speed-up initial data load without dropping index

Re: Logical Replication speed-up initial data

2021-08-05 Thread Rick Otten
On Thu, Aug 5, 2021 at 12:57 AM Nikhil Shetty wrote: > Hi, > > Thank you for the suggestion. > > We tried by dropping indexes and it worked faster compared to what we saw > earlier. We wanted to know if anybody has done any other changes that helps > speed-up initial data load without dropping in

Re: Logical Replication speed-up initial data

2021-08-04 Thread Nikhil Shetty
Hi Stefano, Thank you for the information. Regards, Nikhil On Wed, Aug 4, 2021 at 9:25 PM Stefano Amoroso wrote: > Hello, > in my experience, to speed up the initial load, I had to drop UKs and FKs. > Unfortunately, the initial load doesn't work in parallel and, for each > table, there is only

Re: Logical Replication speed-up initial data

2021-08-04 Thread Nikhil Shetty
Hi, Thank you for the suggestion. We tried by dropping indexes and it worked faster compared to what we saw earlier. We wanted to know if anybody has done any other changes that helps speed-up initial data load without dropping indexes. Thanks, Nikhil On Wed, Aug 4, 2021 at 8:54 PM Hüseyin Demi

Re: Logical Replication speed-up initial data

2021-08-04 Thread Christophe Pettus
> On Aug 4, 2021, at 08:06, Nikhil Shetty wrote: > > How can we increase the speed of the initial data load without dropping the > indexes on destination? You can do the usual steps of increasing checkpoint_timeout and max_wal_size (since incoming logical replication changes are WAL logged)

Re: Logical Replication speed-up initial data

2021-08-04 Thread Stefano Amoroso
Hello, in my experience, to speed up the initial load, I had to drop UKs and FKs. Unfortunately, the initial load doesn't work in parallel and, for each table, there is only one sync worker. Regards Stefano Amoroso Il giorno mer 4 ago 2021 alle ore 17:24 Hüseyin Demir < demirhuseyinn...@gmail.co

Re: Logical Replication speed-up initial data

2021-08-04 Thread Hüseyin Demir
Hello, I also faced a similar issue. Try removing the indexes on the destination first if possible. After that, you can add the indexes. Regards. Nikhil Shetty , 4 Ağu 2021 Çar, 18:07 tarihinde şunu yazdı: > Hi Team, > > We have a highly transactional system as the source of logical replicatio

Logical Replication speed-up initial data

2021-08-04 Thread Nikhil Shetty
Hi Team, We have a highly transactional system as the source of logical replication and the database size is 500GB+. We are replicating all tables from source using logical replication. For two tables the initial data load is very slow and it never completes even after 24hrs+ Table size is under