Re: Bug in huge simplehash

2021-08-13 Thread Andres Freund
On 2021-08-13 14:40:08 +0300, Yura Sokolov wrote: > Ranier Vilela писал 2021-08-13 14:12: > > Em sex., 13 de ago. de 2021 às 07:15, Andres Freund > > escreveu: > > > Personally I find it more obvious to understand the intended > > > behaviour > > > with ~0 (i.e. all bits set) than with a width t

Re: Bug in huge simplehash

2021-08-13 Thread Yura Sokolov
Ranier Vilela писал 2021-08-13 14:12: Em sex., 13 de ago. de 2021 às 07:15, Andres Freund escreveu: Hi, On 2021-08-13 12:44:17 +0300, Yura Sokolov wrote: Andres Freund писал 2021-08-13 12:21: Any chance you'd write a test for simplehash with such huge amount of values? It'd require a smal

Re: Bug in huge simplehash

2021-08-13 Thread Ranier Vilela
Em sex., 13 de ago. de 2021 às 07:15, Andres Freund escreveu: > Hi, > > On 2021-08-13 12:44:17 +0300, Yura Sokolov wrote: > > Andres Freund писал 2021-08-13 12:21: > > > Any chance you'd write a test for simplehash with such huge amount of > > > values? It'd require a small bit of trickery to be

Re: Bug in huge simplehash

2021-08-13 Thread Andres Freund
Hi, On 2021-08-13 12:44:17 +0300, Yura Sokolov wrote: > Andres Freund писал 2021-08-13 12:21: > > Any chance you'd write a test for simplehash with such huge amount of > > values? It'd require a small bit of trickery to be practical. On systems > > with MAP_NORESERVE it should be feasible. > > Wh

Re: Bug in huge simplehash

2021-08-13 Thread Yura Sokolov
Andres Freund писал 2021-08-13 12:21: Hi, On 2021-08-10 11:52:59 +0300, Yura Sokolov wrote: - sizemask is set only in SH_COMPUTE_PARAMETERS . And it is set in this way: /* now set size */ tb->size = size; if (tb->size == SH_MAX_SIZE) tb->sizemask = 0;

Re: Bug in huge simplehash

2021-08-13 Thread Andres Freund
Hi, On 2021-08-10 11:52:59 +0300, Yura Sokolov wrote: > - sizemask is set only in SH_COMPUTE_PARAMETERS . And it is set in this way: > > /* now set size */ > tb->size = size; > > if (tb->size == SH_MAX_SIZE) > tb->sizemask = 0; > else >

Re: Bug in huge simplehash

2021-08-12 Thread David Rowley
On Wed, 11 Aug 2021 at 00:10, Yura Sokolov wrote: > Attached v2. Eyeballing this it looks fine, but I was a little nervous backpatching without testing it properly, so I ended up provisioning a machine with 256GB and doing a round of testing. I just created the most simple table I could: create

Re: Bug in huge simplehash

2021-08-10 Thread Yura Sokolov
Ranier Vilela писал 2021-08-10 14:21: Em ter., 10 de ago. de 2021 às 05:53, Yura Sokolov escreveu: I went to check SH_GROW and It is `SH_GROW(SH_TYPE *tb, uint32 newsize)` :-((( Therefore when `tb->size == SH_MAX_SIZE/2` and we call `SH_GROW(tb, tb->size * 2)`, then SH_GROW(tb, 0) is call

Re: Bug in huge simplehash

2021-08-10 Thread Ranier Vilela
Em ter., 10 de ago. de 2021 às 05:53, Yura Sokolov escreveu: > Good day, hackers. > > Our client caught process stuck in tuplehash_grow. There was a query > like > `select ts, count(*) from really_huge_partitioned_table group by ts`, > and > planner decided to use hash aggregation. > > Backtrace

Re: Bug in huge simplehash

2021-08-10 Thread David Rowley
On Tue, 10 Aug 2021 at 20:53, Yura Sokolov wrote: > EXPLAIN shows that there are 2604186278 rows in all partitions, but > planner > thinks there will be only 200 unique rows after group by. Looks like we > was > mistaken. This looks unrelated. Looks like the planner used DEFAULT_NUM_DISTINCT. >

Bug in huge simplehash

2021-08-10 Thread Yura Sokolov
Good day, hackers. Our client caught process stuck in tuplehash_grow. There was a query like `select ts, count(*) from really_huge_partitioned_table group by ts`, and planner decided to use hash aggregation. Backtrace shows that oldsize were 2147483648 at the moment. While newsize were opti