Re: Vacuum: allow usage of more than 1GB of work mem

2018-10-01 Thread Michael Paquier
On Mon, Jul 16, 2018 at 02:33:17PM -0400, Andrew Dunstan wrote: > Ah, ok. Thanks. ignore the email I just sent about that. So... This thread has basically died of inactivity, while there have been a couple of interesting things discussed, like the version from Heikki here: https://www.postgresql.

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-17 Thread Robert Haas
On Fri, Apr 6, 2018 at 4:25 PM, Claudio Freire wrote: >> True all that. My point is that the multi-segmented array isn't all that >> simple and proven, compared to an also straightforward B-tree. It's pretty >> similar to a B-tree, actually, except that it has exactly two levels, and >> the node (

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-16 Thread Andrew Dunstan
On 07/16/2018 11:35 AM, Claudio Freire wrote: On Mon, Jul 16, 2018 at 11:34 AM Claudio Freire wrote: On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan wrote: On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: On 13/07/18 01:39, Andrew Dunstan wrote: On 07/12/2018 06:34 PM, Alvaro Herrera w

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-16 Thread Claudio Freire
On Mon, Jul 16, 2018 at 3:30 PM Andrew Dunstan wrote: > > > > On 07/16/2018 10:34 AM, Claudio Freire wrote: > > On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan > > wrote: > >> > >> > >> On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: > >>> On 13/07/18 01:39, Andrew Dunstan wrote: > On 07/12

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-16 Thread Andrew Dunstan
On 07/16/2018 10:34 AM, Claudio Freire wrote: On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan wrote: On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: On 13/07/18 01:39, Andrew Dunstan wrote: On 07/12/2018 06:34 PM, Alvaro Herrera wrote: On 2018-Jul-12, Andrew Dunstan wrote: I fully un

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-16 Thread Heikki Linnakangas
On 16/07/18 18:35, Claudio Freire wrote: On Mon, Jul 16, 2018 at 11:34 AM Claudio Freire wrote: On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan wrote: On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: Claudio raised a good point, that doing small pallocs leads to fragmentation, and in particu

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-16 Thread Claudio Freire
On Mon, Jul 16, 2018 at 11:34 AM Claudio Freire wrote: > > On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan > wrote: > > > > > > > > On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: > > > On 13/07/18 01:39, Andrew Dunstan wrote: > > >> On 07/12/2018 06:34 PM, Alvaro Herrera wrote: > > >>> On 2018-

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-16 Thread Claudio Freire
On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan wrote: > > > > On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: > > On 13/07/18 01:39, Andrew Dunstan wrote: > >> On 07/12/2018 06:34 PM, Alvaro Herrera wrote: > >>> On 2018-Jul-12, Andrew Dunstan wrote: > >>> > I fully understand. I think this

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-13 Thread Andrew Dunstan
On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: On 13/07/18 01:39, Andrew Dunstan wrote: On 07/12/2018 06:34 PM, Alvaro Herrera wrote: On 2018-Jul-12, Andrew Dunstan wrote: I fully understand. I think this needs to go back to "Waiting on Author". Why?  Heikki's patch applies fine and pa

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-13 Thread Heikki Linnakangas
On 13/07/18 01:39, Andrew Dunstan wrote: On 07/12/2018 06:34 PM, Alvaro Herrera wrote: On 2018-Jul-12, Andrew Dunstan wrote: I fully understand. I think this needs to go back to "Waiting on Author". Why? Heikki's patch applies fine and passes the regression tests. Well, I understood Claudi

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-12 Thread Andrew Dunstan
On 07/12/2018 06:34 PM, Alvaro Herrera wrote: On 2018-Jul-12, Andrew Dunstan wrote: I fully understand. I think this needs to go back to "Waiting on Author". Why? Heikki's patch applies fine and passes the regression tests. Well, I understood Claudio was going to do some more work (see

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-12 Thread Alvaro Herrera
On 2018-Jul-12, Andrew Dunstan wrote: > I fully understand. I think this needs to go back to "Waiting on Author". Why? Heikki's patch applies fine and passes the regression tests. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Tr

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-12 Thread Andrew Dunstan
On 07/12/2018 12:38 PM, Claudio Freire wrote: On Thu, Jul 12, 2018 at 10:44 AM Andrew Dunstan wrote: On 04/06/2018 08:00 PM, Claudio Freire wrote: On Fri, Apr 6, 2018 at 5:25 PM, Claudio Freire wrote: On Fri, Apr 6, 2018 at 10:39 AM, Heikki Linnakangas wrote: On 06/04/18 01:59, Claudi

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-12 Thread Claudio Freire
On Thu, Jul 12, 2018 at 10:44 AM Andrew Dunstan wrote: > > > > On 04/06/2018 08:00 PM, Claudio Freire wrote: > > On Fri, Apr 6, 2018 at 5:25 PM, Claudio Freire > > wrote: > >> On Fri, Apr 6, 2018 at 10:39 AM, Heikki Linnakangas > >> wrote: > >>> On 06/04/18 01:59, Claudio Freire wrote: >

Re: Vacuum: allow usage of more than 1GB of work mem

2018-07-12 Thread Andrew Dunstan
On 04/06/2018 08:00 PM, Claudio Freire wrote: On Fri, Apr 6, 2018 at 5:25 PM, Claudio Freire wrote: On Fri, Apr 6, 2018 at 10:39 AM, Heikki Linnakangas wrote: On 06/04/18 01:59, Claudio Freire wrote: The iteration interface, however, seems quite specific for the use case of vacuumlazy, so

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-06 Thread Claudio Freire
On Fri, Apr 6, 2018 at 5:25 PM, Claudio Freire wrote: > On Fri, Apr 6, 2018 at 10:39 AM, Heikki Linnakangas wrote: >> On 06/04/18 01:59, Claudio Freire wrote: >>> >>> The iteration interface, however, seems quite specific for the use >>> case of vacuumlazy, so it's not really a good abstraction.

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-06 Thread Alvaro Herrera
Claudio Freire wrote: > On Fri, Apr 6, 2018 at 11:00 AM, Alvaro Herrera > wrote: > > FWIW I liked the idea of having this abstraction possibly do other > > things -- for instance to vacuum brin indexes you'd like to mark index > > tuples as "containing tuples that were removed" and eventually >

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-06 Thread Alvaro Herrera
Heikki Linnakangas wrote: > On 06/04/18 01:59, Claudio Freire wrote: > > The iteration interface, however, seems quite specific for the use > > case of vacuumlazy, so it's not really a good abstraction. > > Can you elaborate? It does return the items one block at a time. Is that > what you mean by

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-06 Thread Heikki Linnakangas
On 06/04/18 16:39, Heikki Linnakangas wrote: Sure. I used the attached script to test this. Sorry, I attached the wrong script. Here is the correct one that I used. Here are also the results I got from running it - Heikki vacuumbenchone.sh Description: application/shellscript vacuumbench

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-06 Thread Heikki Linnakangas
On 06/04/18 01:59, Claudio Freire wrote: The iteration interface, however, seems quite specific for the use case of vacuumlazy, so it's not really a good abstraction. Can you elaborate? It does return the items one block at a time. Is that what you mean by being specific for vacuumlazy? I gues

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-05 Thread Claudio Freire
On Thu, Apr 5, 2018 at 5:02 PM, Heikki Linnakangas wrote: > On 03/04/18 17:20, Claudio Freire wrote: >> >> Ok, rebased patches attached > > > Thanks! I took a look at this. > > First, now that the data structure is more complicated, I think it's time to > abstract it, and move it out of vacuumlazy

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-05 Thread Heikki Linnakangas
On 03/04/18 17:20, Claudio Freire wrote: Ok, rebased patches attached Thanks! I took a look at this. First, now that the data structure is more complicated, I think it's time to abstract it, and move it out of vacuumlazy.c. The Tid Map needs to support the following operations: * Add TIDs,

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-03 Thread Claudio Freire
On Tue, Apr 3, 2018 at 11:09 AM, Claudio Freire wrote: > On Tue, Apr 3, 2018 at 11:06 AM, Claudio Freire > wrote: >> I didn't receive your comment, I just saw it. Nevertheless, I rebased the >> patches a while ago just because I noticed they didn't apply anymore in >> cputube, and they still s

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-03 Thread Claudio Freire
On Tue, Apr 3, 2018 at 11:06 AM, Claudio Freire wrote: > I didn't receive your comment, I just saw it. Nevertheless, I rebased the > patches a while ago just because I noticed they didn't apply anymore in > cputube, and they still seem to apply. Sorry, that is false. They appear green in cputu

Re: Vacuum: allow usage of more than 1GB of work mem

2018-04-03 Thread Claudio Freire
I didn't receive your comment, I just saw it. Nevertheless, I rebased the patches a while ago just because I noticed they didn't apply anymore in cputube, and they still seem to apply. Patch number 2 was committed a long while ago, that's why it's missing. It was a simple patch, it landed rewri

Re: Vacuum: allow usage of more than 1GB of work mem

2018-03-27 Thread Aleksander Alekseev
Hello everyone, I would like to let you know that unfortunately these patches don't apply anymore. Also personally I'm a bit confused by the last message that has 0001- and 0003- patches attached but not the 0002- one.

Re: Vacuum: allow usage of more than 1GB of work mem

2018-01-22 Thread Aleksander Alekseev
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed I can confirm that these patches don't break anything; the co