Re: [PATCH] pg_dump: lock tables in batches

2023-01-03 Thread Tom Lane
Gilles Darold writes: > As it seems to have a consensus to apply this patch I will change the > commitfest status to ready for committers. Yeah. I still have a nagging worry about why I didn't do this already, but without evidence I can't fairly block the patch. Hence, pushed. I did cut the L

Re: [PATCH] pg_dump: lock tables in batches

2023-01-03 Thread Gilles Darold
Le 08/12/2022 à 01:03, Tom Lane a écrit : Andres Freund writes: On 2022-12-07 17:53:05 -0500, Tom Lane wrote: Is "-s" mode actually a relevant criterion here? With per-table COPY commands added into the mix you could not possibly get better than 2x improvement, and likely a good deal less. W

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Tom Lane
Andres Freund writes: > On 2022-12-07 17:53:05 -0500, Tom Lane wrote: >> Is "-s" mode actually a relevant criterion here? With per-table COPY >> commands added into the mix you could not possibly get better than 2x >> improvement, and likely a good deal less. > Well, -s isn't something used all

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Andres Freund
Hi, On 2022-12-07 17:53:05 -0500, Tom Lane wrote: > Andres Freund writes: > > With an artificial delay of 100ms, the perf difference between the batching > > patch and not using the batching patch is huge. Huge enough that I don't > > have > > the patience to wait for the non-batched case to com

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Jacob Champion
On Wed, Dec 7, 2022 at 2:53 PM Tom Lane wrote: > Is "-s" mode actually a relevant criterion here? With per-table COPY > commands added into the mix you could not possibly get better than 2x > improvement, and likely a good deal less. Don't we hit this code path in pg_upgrade? You won't see huge

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Tom Lane
Andres Freund writes: > With an artificial delay of 100ms, the perf difference between the batching > patch and not using the batching patch is huge. Huge enough that I don't have > the patience to wait for the non-batched case to complete. Clearly, if you insert a sufficiently large artificial r

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Andres Freund
Hi, On 2022-12-07 13:32:42 -0800, Andres Freund wrote: > On 2022-12-07 18:14:01 -0300, Fabrízio de Royes Mello wrote: > > Here we have some numbers about the Aleksander's patch: > > Hm. Were they taken in an assertion enabled build or such? Just testing the > t1 case on HEAD, I get 0:01.23 el

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Andres Freund
Hi, On 2022-12-07 18:14:01 -0300, Fabrízio de Royes Mello wrote: > Here we have some numbers about the Aleksander's patch: Hm. Were they taken in an assertion enabled build or such? Just testing the t1 case on HEAD, I get 0:01.23 elapsed for an unpatched pg_dump in an optimized build. And tha

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Fabrízio de Royes Mello
On Wed, Dec 7, 2022 at 2:28 PM Tom Lane wrote: > > Andres Freund writes: > > On 2022-12-07 10:44:33 -0500, Tom Lane wrote: > >> I have a strong sense of deja vu here. I'm pretty sure I experimented > >> with this idea last year and gave up on it. I don't recall exactly > >> why, but either it d

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Andres Freund
Hi, On 2022-12-07 12:28:03 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-12-07 10:44:33 -0500, Tom Lane wrote: > >> I have a strong sense of deja vu here. I'm pretty sure I experimented > >> with this idea last year and gave up on it. I don't recall exactly > >> why, but either it

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Tom Lane
Andres Freund writes: > On 2022-12-07 10:44:33 -0500, Tom Lane wrote: >> I have a strong sense of deja vu here. I'm pretty sure I experimented >> with this idea last year and gave up on it. I don't recall exactly >> why, but either it didn't show any meaningful performance improvement >> for me

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Andres Freund
Hi, On 2022-12-07 10:44:33 -0500, Tom Lane wrote: > Aleksander Alekseev writes: > > What he proposes is taking the locks in batches. > > I have a strong sense of deja vu here. I'm pretty sure I experimented > with this idea last year and gave up on it. I don't recall exactly > why, but either

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Tom Lane
Aleksander Alekseev writes: > What he proposes is taking the locks in batches. I have a strong sense of deja vu here. I'm pretty sure I experimented with this idea last year and gave up on it. I don't recall exactly why, but either it didn't show any meaningful performance improvement for me or

Re: [PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Fabrízio de Royes Mello
On Wed, Dec 7, 2022 at 12:09 PM Aleksander Alekseev < aleksan...@timescale.com> wrote: > > Hi hackers, > > A colleague of mine reported a slight inconvenience with pg_dump. > > He is dumping the data from a remote server. There are several > thousands of tables in the database. Making a dump locall

[PATCH] pg_dump: lock tables in batches

2022-12-07 Thread Aleksander Alekseev
Hi hackers, A colleague of mine reported a slight inconvenience with pg_dump. He is dumping the data from a remote server. There are several thousands of tables in the database. Making a dump locally and/or using pg_basebackup and/or logical replication is not an option. So what pg_dump currently