On 4/20/2019 21:14, Tomas Vondra wrote:
Maybe. But before wasting any more time on the memory leak investigation,
I suggest you first try the patch moving the BufFile allocations to a
separate context. That'll either confirm or disprove the theory.
OK, fair enough. So, first patch 0001-* applie
I am now running Justin's patch after undoing Tomas' patches and any of
my own hacks (which would not have interfered with Tomas' patch)
On 4/20/2019 15:30, Justin Pryzby wrote:
With my patch, the number of batches is nonlinear WRT work_mem, and reaches a
maximum for moderately small work_mem.
On Sat, Apr 20, 2019 at 08:33:46PM -0400, Gunther wrote:
On 4/20/2019 16:01, Tomas Vondra wrote:
For me, this did the trick:
update pg_class set (relpages, reltuples) = (100, 1) where
relname = 'tmp_r';
update pg_class set (relpages, reltuples) = (1, 100) where
relname = 'tmp_q';
On 4/20/2019 16:01, Tomas Vondra wrote:
For me, this did the trick:
update pg_class set (relpages, reltuples) = (100, 1) where
relname = 'tmp_r';
update pg_class set (relpages, reltuples) = (1, 100) where
relname = 'tmp_q';
YES! For me too. My EXPLAIN ANALYZE actually succeeded.
On Sat, Apr 20, 2019 at 06:20:15PM -0400, Tom Lane wrote:
Justin Pryzby writes:
On Sat, Apr 20, 2019 at 04:46:03PM -0400, Tom Lane wrote:
Maybe we just need to account for the per-batch buffers while estimating
the amount of memory used during planning. That would force this case
into a merge
Justin Pryzby writes:
> On Sat, Apr 20, 2019 at 04:46:03PM -0400, Tom Lane wrote:
>> Maybe we just need to account for the per-batch buffers while estimating
>> the amount of memory used during planning. That would force this case
>> into a mergejoin instead, given that work_mem is set so small.
On Sat, Apr 20, 2019 at 04:46:03PM -0400, Tom Lane wrote:
> Tomas Vondra writes:
> > I think it's really a matter of underestimate, which convinces the planner
> > to hash the larger table. In this case, the table is 42GB, so it's
> > possible it actually works as expected. With work_mem = 4MB I'v
On Sat, Apr 20, 2019 at 10:36:50PM +0200, Tomas Vondra wrote:
On Sat, Apr 20, 2019 at 04:26:34PM -0400, Tom Lane wrote:
Tomas Vondra writes:
Considering how rare this issue likely is, we need to be looking for a
solution that does not break the common case.
Agreed. What I think we need to f
On Sat, Apr 20, 2019 at 04:46:03PM -0400, Tom Lane wrote:
Tomas Vondra writes:
I think it's really a matter of underestimate, which convinces the planner
to hash the larger table. In this case, the table is 42GB, so it's
possible it actually works as expected. With work_mem = 4MB I've seen 32k
Gunther writes:
> and checked my log file and there was nothing before the call
> MemoryContextStats(TopPortalContext) so I don't understand where this
> printf stuff is ending up.
It's going to stdout, which is likely block-buffered whereas stderr
is line-buffered, so data from the latter will
Tomas Vondra writes:
> I think it's really a matter of underestimate, which convinces the planner
> to hash the larger table. In this case, the table is 42GB, so it's
> possible it actually works as expected. With work_mem = 4MB I've seen 32k
> batches, and that's not that far off, I'd day. Maybe
On Sat, Apr 20, 2019 at 04:26:34PM -0400, Tom Lane wrote:
Tomas Vondra writes:
Considering how rare this issue likely is, we need to be looking for a
solution that does not break the common case.
Agreed. What I think we need to focus on next is why the code keeps
increasing the number of bat
Tomas Vondra writes:
> Considering how rare this issue likely is, we need to be looking for a
> solution that does not break the common case.
Agreed. What I think we need to focus on next is why the code keeps
increasing the number of batches. It seems like there must be an undue
amount of data
On Tue, Apr 16, 2019 at 11:46:51PM -0500, Justin Pryzby wrote:
I wonder if it'd be useful to compile with
./configure CFLAGS=-DHJDEBUG=1
Could you try this, too ?
OK, doing it now, here is what I'm getting in the log file now. I am
surprised I get so few rows here when there
2019-04-20 17:
On Sat, Apr 20, 2019 at 02:30:09PM -0500, Justin Pryzby wrote:
On Sun, Apr 14, 2019 at 11:24:59PM -0400, Tom Lane wrote:
Gunther writes:
>ExecutorState: 2234123384 total in 266261 blocks; 3782328 free (17244
chunks); 2230341056 used
Oooh, that looks like a memory leak right enough. T
On Fri, Apr 19, 2019 at 11:34:54PM -0400, Gunther wrote:
...
It would be so nice if there was a way to force a specific plan for
purposes of the testing. I tried giving false data in pg_class
reltuples and relpages:
foo=# analyze tmp_q;
ANALYZE
foo=# analyze tmp_r;
ANALYZE
foo=# select rel
On Sun, Apr 14, 2019 at 11:24:59PM -0400, Tom Lane wrote:
> Gunther writes:
> >ExecutorState: 2234123384 total in 266261 blocks; 3782328 free
> > (17244 chunks); 2230341056 used
>
> Oooh, that looks like a memory leak right enough. The ExecutorState
> should not get that big for any rea
> The only problem is that I can't test that this actually would trigger the
> memory problem, because I can't force the plan to use the right join, it
> always reverts to the left join hashing the tmp_q:
I think the table on the "OUTER" side is the one which needs to be iterated
over (not hashed)
On Fri, Apr 19, 2019 at 11:34:54PM -0400, Gunther wrote:
> On 4/19/2019 17:01, Justin Pryzby wrote:
> >Were you able to reproduce the issue in some minimized way ? Like after
> >joining fewer tables or changing to join with fewer join conditions ?
> >
> >On Thu, Apr 18, 2019 at 05:21:28PM +0200, T
19 matches
Mail list logo
