Le dimanche 18 juin 2023, 20:22:17 CEST Tomas Vondra a écrit :
> Hi Ronan,
>
> We briefly chatted about the glibc-tuning part of this thread at pgcon,
> so I wonder if you're still planning to pursue that. If you do, I
> suggest we start a fresh thread, so that it's not mixed with the already
> co
Hi Ronan,
We briefly chatted about the glibc-tuning part of this thread at pgcon,
so I wonder if you're still planning to pursue that. If you do, I
suggest we start a fresh thread, so that it's not mixed with the already
committed improvements of generation context.
I wonder what's the situation
Thanks for raising this.
On Sun, 24 Apr 2022 at 12:59, Noah Misch wrote:
> This (commit 77bae39) did not change function parameter counts, and
> TUPLESORT_RANDOMACCESS generally has same the same numeric value as "true". I
> get no warning if I pass "true" for the "sortopt" flags parameter. Hen
On Sat, Apr 23, 2022 at 5:59 PM Noah Misch wrote:
> This (commit 77bae39) did not change function parameter counts, and
> TUPLESORT_RANDOMACCESS generally has same the same numeric value as "true". I
> get no warning if I pass "true" for the "sortopt" flags parameter. Hence, I
> suspect this did
On Fri, Apr 01, 2022 at 10:00:08PM +1300, David Rowley wrote:
> 0002:
> This modifies the tuplesort API so that instead of having a
> randomAccess bool flag, this is changed to a bitwise flag that we can
> add further options in the future. It's slightly annoying to break
> the API, but it's not e
On Sat, 2 Apr 2022 at 02:25, Justin Pryzby wrote:
>
> On Fri, Apr 01, 2022 at 10:00:08PM +1300, David Rowley wrote:
> > 0002:
> > This modifies the tuplesort API so that instead of having a
> > randomAccess bool flag,
>
> Per cirrus, this missed updating one line for dtrace.
Thanks.
I've pushed
On Fri, Apr 01, 2022 at 10:00:08PM +1300, David Rowley wrote:
> 0002:
> This modifies the tuplesort API so that instead of having a
> randomAccess bool flag,
Per cirrus, this missed updating one line for dtrace.
On Fri, Apr 01, 2022 at 10:00:08PM +1300, David Rowley wrote:
> I feel this is a pret
On Wed, 23 Mar 2022 at 04:08, David Rowley wrote:
> If nobody has any objections to the 0001 patch then I'd like to move
> ahead with it in the next few days. For the 0002 patch, I'm currently
> feeling like we shouldn't be using the Generation context for bounded
> sorts. The only way I can thin
On Wed, 23 Feb 2022 at 15:25, Andres Freund wrote:
> On 2022-02-18 12:10:51 +1300, David Rowley wrote:
> > The other way I thought to fix it was by changing the logic for when
> > generation blocks are freed. In the problem case mentioned above, the
> > block being freed is the current block (whi
On 2/23/22 03:25, Andres Freund wrote:
> Hi,
>
> On 2022-02-18 12:10:51 +1300, David Rowley wrote:
>> The other way I thought to fix it was by changing the logic for when
>> generation blocks are freed. In the problem case mentioned above, the
>> block being freed is the current block (which was
Hi,
On 2022-02-18 12:10:51 +1300, David Rowley wrote:
> The other way I thought to fix it was by changing the logic for when
> generation blocks are freed. In the problem case mentioned above, the
> block being freed is the current block (which was just allocated). I
> made some changes to adjus
On Sun, 13 Feb 2022 at 09:56, Tomas Vondra
wrote:
> I'm not against pushing the generation context improvements, e.g.
> something like the patches posted in [1], because those seem reasonable
> in general. But I'm somewhat confused about the last patch (adjusting
> allocChunkLimit) causing fairly
On 1/20/22 08:36, Ronan Dunklau wrote:
> Le jeudi 20 janvier 2022, 02:31:15 CET David Rowley a écrit :
>> On Tue, 18 Jan 2022 at 19:45, Julien Rouhaud wrote:
>>> I'm also a bit confused about which patch(es) should actually be reviewed
>>> in that thread. My understanding is that the original pat
On 1/25/22 13:34, Ronan Dunklau wrote:
>
> ...
>
> I've run the same 1-to-32-columns sort benchmark, using both glibc malloc and
> jemalloc, with the standard aset memory manager or with David's generation v2
> patch. To use jemalloc, I just use a LD_PRELOAD env variable before starting
> postg
Hi:
On Thu, Jan 20, 2022 at 9:31 AM David Rowley wrote:
>
> As of now, I still believe we'll need Tomas' patches to allow the
> block size to grow up to a maximum size. I think those patches are
> likely needed before we think about making tuplesort use generation
> contexts. The reason I beli
Le jeudi 20 janvier 2022, 08:36:46 CET Ronan Dunklau a écrit :
> Le jeudi 20 janvier 2022, 02:31:15 CET David Rowley a écrit :
> > On Tue, 18 Jan 2022 at 19:45, Julien Rouhaud wrote:
> > > I'm also a bit confused about which patch(es) should actually be
> > > reviewed
> > > in that thread. My und
Le jeudi 20 janvier 2022, 02:31:15 CET David Rowley a écrit :
> On Tue, 18 Jan 2022 at 19:45, Julien Rouhaud wrote:
> > I'm also a bit confused about which patch(es) should actually be reviewed
> > in that thread. My understanding is that the original patch might not be
> > relevant anymore but I
On Tue, 18 Jan 2022 at 19:45, Julien Rouhaud wrote:
> I'm also a bit confused about which patch(es) should actually be reviewed in
> that thread. My understanding is that the original patch might not be
> relevant
> anymore but I might be wrong. The CF entry should probably be updated to
> clar
Hi,
On Fri, Jan 07, 2022 at 12:03:55PM +0100, Ronan Dunklau wrote:
>
> (Sorry for trying to merge back the discussion on the two sides of the thread)
>
> In https://www.postgresql.org/message-id/4776839.iZASKD2KPV%40aivenronan, I
> expressed the idea of being able to tune glibc's malloc behavi
Le vendredi 7 janvier 2022, 13:03:28 CET Tomas Vondra a écrit :
> On 1/7/22 12:03, Ronan Dunklau wrote:
> > Le vendredi 31 décembre 2021, 22:26:37 CET David Rowley a écrit :
> >> I've attached some benchmark results that I took recently. The
> >> spreadsheet contains results from 3 versions. maste
On 1/7/22 12:03, Ronan Dunklau wrote:
Le vendredi 31 décembre 2021, 22:26:37 CET David Rowley a écrit :
I've attached some benchmark results that I took recently. The
spreadsheet contains results from 3 versions. master, master + 0001 -
0002, then master + 0001 - 0003. The 0003 patch makes the
Le vendredi 17 décembre 2021, 14:39:10 CET Tomas Vondra a écrit :
> I wasn't really suggesting to investigate those other allocators in this
> patch - it seems like a task requiring a pretty significant amount of
> work/time. My point was that we should make it reasonably easy to add
> tweaks for t
On 12/17/21 09:08, Ronan Dunklau wrote:
> Le jeudi 16 décembre 2021, 18:00:56 CET Tomas Vondra a écrit :
>> On 12/16/21 17:03, Ronan Dunklau wrote:
>>> Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit :
I will follow up with a benchmark of the test sorting a table with a
widt
Le vendredi 17 décembre 2021, 09:08:06 CET Ronan Dunklau a écrit :
> It is my understanding that malloc will try to compact memory by moving it
> around. So the memory should be actually be released to the kernel at some
> point. In the meantime, malloc can reuse it for our next invocation (which
>
Le jeudi 16 décembre 2021, 18:00:56 CET Tomas Vondra a écrit :
> On 12/16/21 17:03, Ronan Dunklau wrote:
> > Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit :
> >> I will follow up with a benchmark of the test sorting a table with a
> >> width
> >> varying from 1 to 32 columns.
> >
>
On 12/16/21 17:03, Ronan Dunklau wrote:
Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit :
I will follow up with a benchmark of the test sorting a table with a width
varying from 1 to 32 columns.
So please find attached another benchmark for that case.
The 3 different patchsets
Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit :
> I will follow up with a benchmark of the test sorting a table with a width
> varying from 1 to 32 columns.
>
So please find attached another benchmark for that case.
The 3 different patchsets tested are:
- master
- fixed (David
Le mercredi 8 décembre 2021, 22:07:12 CET Tomas Vondra a écrit :
> On 12/8/21 16:51, Ronan Dunklau wrote:
> > Le jeudi 9 septembre 2021, 15:37:59 CET Tomas Vondra a écrit :
> >> And now comes the funny part - if I run it in the same backend as the
> >>
> >> "full" benchmark, I get roughly the same
On 12/8/21 16:51, Ronan Dunklau wrote:
> Le jeudi 9 septembre 2021, 15:37:59 CET Tomas Vondra a écrit :
>> And now comes the funny part - if I run it in the same backend as the
>> "full" benchmark, I get roughly the same results:
>>
>> block_size | chunk_size | mem_allocated | alloc_ms | free
Le jeudi 9 septembre 2021, 15:37:59 CET Tomas Vondra a écrit :
> And now comes the funny part - if I run it in the same backend as the
> "full" benchmark, I get roughly the same results:
>
> block_size | chunk_size | mem_allocated | alloc_ms | free_ms
> ++---
On Mon, 9 Aug 2021 at 00:38, Tomas Vondra wrote:
> I'm not sure quadrupling the size is a good idea, though, because it
> increases the amount of memory we might be wasting. With the doubling,
> the amount of wasted /unused memory is limited to ~50%, because the next
> block is (roughly) equal to
On Wed, 4 Aug 2021 at 02:10, Tomas Vondra wrote:
> I did run the same set of benchmarks as for Slab, measuring some usual
> allocation patterns. The results for i5-2500k machine are attached (for
> the xeon it's almost exactly the same behavior). While running those
> tests I realized the last pat
On 8/8/21 12:11 PM, David Rowley wrote:
On Sat, 7 Aug 2021 at 12:10, Tomas Vondra wrote:
All of the tests show that the patches to improve the allocation
efficiency of generation.c don't help to improve the results of the
test cases. I wondered if it's maybe worth trying to see what happens
if
On 8/8/21 9:02 AM, David Rowley wrote:
On Sat, 7 Aug 2021 at 12:10, Tomas Vondra wrote:
On 8/6/21 3:07 PM, David Rowley wrote:
All of the tests show that the patches to improve the allocation
efficiency of generation.c don't help to improve the results of the
test cases. I wondered if it's
On Sat, 7 Aug 2021 at 12:10, Tomas Vondra wrote:
> > All of the tests show that the patches to improve the allocation
> > efficiency of generation.c don't help to improve the results of the
> > test cases. I wondered if it's maybe worth trying to see what happens
> > if instead of doubling the all
On Sat, 7 Aug 2021 at 12:10, Tomas Vondra wrote:
>
> On 8/6/21 3:07 PM, David Rowley wrote:
> > All of the tests show that the patches to improve the allocation
> > efficiency of generation.c don't help to improve the results of the
> > test cases. I wondered if it's maybe worth trying to see what
On 8/6/21 3:07 PM, David Rowley wrote:
On Wed, 4 Aug 2021 at 02:10, Tomas Vondra wrote:
A review would be nice, although it can wait - It'd be interesting to
know if those patches help with the workload(s) you've been looking at.
I tried out the v2 set of patches using the attached scripts.
On Wed, 4 Aug 2021 at 02:10, Tomas Vondra wrote:
> A review would be nice, although it can wait - It'd be interesting to
> know if those patches help with the workload(s) you've been looking at.
I tried out the v2 set of patches using the attached scripts. The
attached spreadsheet includes the o
Hi,
On 2021-08-03 10:59:25 +1200, David Rowley wrote:
> On Sat, 31 Jul 2021 at 08:38, Andres Freund wrote:
> > There is at least one path using tuplecontext that reaches code that
> > could end up freeing memory to a significant enough degree to care about
> > generation.c effectively not using t
On Sat, 31 Jul 2021 at 08:38, Andres Freund wrote:
> There is at least one path using tuplecontext that reaches code that
> could end up freeing memory to a significant enough degree to care about
> generation.c effectively not using that memory:
> tuplesort_putdatum()->datumCopy()->EOH_flatten_in
On Sat, 31 Jul 2021 at 14:34, Tomas Vondra
wrote:
> I spent a bit of time hacking on the Generation context, adding the two
> improvements discussed in this thread:
>
> 1) internal handling of block sizes, similar to what AllocSet does (it
> pretty much just copies parts of it)
>
> 2) keeper block
Hi,
I spent a bit of time hacking on the Generation context, adding the two
improvements discussed in this thread:
1) internal handling of block sizes, similar to what AllocSet does (it
pretty much just copies parts of it)
2) keeper block (we keep one empry block instead of freeing it)
3)
On 7/30/21 10:38 PM, Andres Freund wrote:
Hi,
On 2021-07-30 18:42:18 +1200, David Rowley wrote:
While reviewing and benchmarking 91e9e89dc (Make nodeSort.c use Datum
sorts for single column sorts), I noticed that we use a separate
memory context to store tuple data and we just reset when we're
Hi,
On 2021-07-30 18:42:18 +1200, David Rowley wrote:
> While reviewing and benchmarking 91e9e89dc (Make nodeSort.c use Datum
> sorts for single column sorts), I noticed that we use a separate
> memory context to store tuple data and we just reset when we're done
> if the sort fits in memory, or w
On Fri, Jul 30, 2021 at 2:42 AM David Rowley wrote:
> Master:
>Sort Method: quicksort Memory: 5541kB
> Patched:
>Sort Method: quicksort Memory: 3197kB
Whoa.
> work_mem = '4GB';
> Testmastergen sortcompare
> Test1317.2665.6210%
> Test2228.6388.9170%
>
45 matches
Mail list logo
