Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Tomas Vondra
On 6/16/21 4:23 PM, Tomas Vondra wrote: On 6/16/21 2:36 PM, Alexander Pyhalov wrote: Hi. It seems this commit commit b676ac443b6a83558d4701b2dd9491c0b37e17c4 Author: Tomas Vondra Date:   Fri Jun 11 20:19:48 2021 +0200 Optimize creation of slots for FDW bulk inserts has broken batch ins

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Tomas Vondra
On 6/16/21 2:36 PM, Alexander Pyhalov wrote: Hi. It seems this commit commit b676ac443b6a83558d4701b2dd9491c0b37e17c4 Author: Tomas Vondra Date:   Fri Jun 11 20:19:48 2021 +0200     Optimize creation of slots for FDW bulk inserts has broken batch insert for partitions with unique indexes.

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Alexander Pyhalov
Tomas Vondra писал 2021-06-12 00:01: On 6/9/21 1:08 PM, Tomas Vondra wrote: On 6/9/21 12:50 PM, Bharath Rupireddy wrote: On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra wrote: Hi, Here's a v2 fixing a silly bug with reusing the same variable in two nested loops (worked for simple postgres_fdw

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-11 Thread Tomas Vondra
On 6/9/21 1:08 PM, Tomas Vondra wrote: > > > On 6/9/21 12:50 PM, Bharath Rupireddy wrote: >> On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra >> wrote: >>> >>> Hi, >>> >>> Here's a v2 fixing a silly bug with reusing the same variable in two >>> nested loops (worked for simple postgres_fdw cases, but

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-09 Thread Bharath Rupireddy
On Wed, Jun 9, 2021 at 4:38 PM Tomas Vondra wrote: > > On 6/9/21 12:50 PM, Bharath Rupireddy wrote: > > On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> Here's a v2 fixing a silly bug with reusing the same variable in two > >> nested loops (worked for simple postgres

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-09 Thread Tomas Vondra
On 6/9/21 12:50 PM, Bharath Rupireddy wrote: On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra wrote: Hi, Here's a v2 fixing a silly bug with reusing the same variable in two nested loops (worked for simple postgres_fdw cases, but "make check" failed). I applied these patches and ran make check

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-09 Thread Bharath Rupireddy
On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra wrote: > > Hi, > > Here's a v2 fixing a silly bug with reusing the same variable in two > nested loops (worked for simple postgres_fdw cases, but "make check" > failed). I applied these patches and ran make check in postgres_fdw contrib module, I saw a

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-09 Thread Tomas Vondra
Hi, Here's a v2 fixing a silly bug with reusing the same variable in two nested loops (worked for simple postgres_fdw cases, but "make check" failed). regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company >From 494018fd3f2b983be474a85fc12fe3a4dbe

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-04 Thread Tomas Vondra
Argh! I forgot the attachments, of course. On 6/4/21 1:48 PM, Tomas Vondra wrote: > Hi, > > Here's two WIP patches that fixes the regression for me. The first part > is from [1], so make large batches work, 0002 just creates a copy of the > tupledesc to not cause issues in resource owner, 0003 en

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-04 Thread Tomas Vondra
Hi, Here's two WIP patches that fixes the regression for me. The first part is from [1], so make large batches work, 0002 just creates a copy of the tupledesc to not cause issues in resource owner, 0003 ensures we only initialize the slots once (not per batch). With the patches applied, the timin

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Tom Lane
Andres Freund writes: > On 2021-05-30 17:10:59 -0400, Tom Lane wrote: >> But it does seem like the hashing scheme somebody added to resowners >> is a bit too simplistic. It ought to be able to cope with lots of >> refs to the same object, or at least not be extra-awful for that case. > It's not

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Andres Freund
Hi, On 2021-05-30 17:10:59 -0400, Tom Lane wrote: > But it does seem like the hashing scheme somebody added to resowners > is a bit too simplistic. It ought to be able to cope with lots of > refs to the same object, or at least not be extra-awful for that case. It's not really the hashing that's

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Tom Lane
Andres Freund writes: > On 2021-05-30 22:22:10 +0200, Tomas Vondra wrote: >> The other problem is that ResourceArrayAdd/Remove seem to behave a bit >> poorly with very many elements - I'm not sure if it's O(N^2) or worse, >> but growing the array and linear searches seem to be a bit expensive. >

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-05-30 Thread Andres Freund
Hi, On 2021-05-30 22:22:10 +0200, Tomas Vondra wrote: > There are two problems at play, here. Firstly, the way it's coded now > the slots are pretty much re-created for each batch. So with 1M rows and > batches of 32k rows, that's ~30x drop/create. That seems a bit wasteful, > and it shouldn't be