Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-13 Thread Robert Haas
On Thu, Jan 9, 2014 at 10:57 PM, Etsuro Fujita wrote: > I wrote: >> Robert Haas wrote: >> > Hmm, fair point. But I'm still not convinced that we really need to >> > add extra accounting for this. What's wrong with just reporting the >> > number of exact and lossy pages? > >> No. I intended to s

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-09 Thread Etsuro Fujita
I wrote: > Robert Haas wrote: > > Hmm, fair point. But I'm still not convinced that we really need to > > add extra accounting for this. What's wrong with just reporting the > > number of exact and lossy pages? > No. I intended to show the desired memory space for a TIDBitmap rather > than the

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-07 Thread Etsuro Fujita
Robert Haas wrote: > On Mon, Jan 6, 2014 at 9:40 PM, Etsuro Fujita > wrote: > > Thank you for taking time to look at this patch. The peak memory > > usage for the discarded bitmap *can* be reflected in the number > > displayed for the bitmap heap scan by the following code in tbm_union() > or tbm

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-07 Thread Robert Haas
On Mon, Jan 6, 2014 at 9:40 PM, Etsuro Fujita wrote: > Robert Haas wrote: >> I spent some time looking at this tonight. I don't think the value that >> is displayed for the bitmap memory tracking will be accurate in complex >> cases. The bitmap heap scan may sit on top of one or more bitmap-and

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-06 Thread Etsuro Fujita
Robert Haas wrote: > I spent some time looking at this tonight. I don't think the value that > is displayed for the bitmap memory tracking will be accurate in complex > cases. The bitmap heap scan may sit on top of one or more bitmap-and or > bitmap-or nodes. When a bitmap-and operation happens,

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 4:27 AM, Andres Freund wrote: > On 2014-01-01 21:15:46 -0500, Robert Haas wrote: >> [ sensible reasoning ] However, I'm not sure it's really worth it. >> I think what people really care about is knowing whether the bitmap >> lossified or not, and generally how much got lossi

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-02 Thread Andres Freund
On 2014-01-01 21:15:46 -0500, Robert Haas wrote: > [ sensible reasoning ] However, I'm not sure it's really worth it. > I think what people really care about is knowing whether the bitmap > lossified or not, and generally how much got lossified. The counts of > exact and lossy pages are sufficient

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-01 Thread Robert Haas
On Fri, Dec 27, 2013 at 1:47 AM, Etsuro Fujita wrote: >> I wrote: >> > Robert Haas wrote: >> > > I'd be wary of showing a desired value unless it's highly likely to >> > > be accurate. > >> > The desired value is accurately estimated based on (a) the total >> > number of exact/lossy pages stored i

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-26 Thread Etsuro Fujita
> I wrote: > > Robert Haas wrote: > > > I'd be wary of showing a desired value unless it's highly likely to > > > be accurate. > > The desired value is accurately estimated based on (a) the total > > number of exact/lossy pages stored in the TIDBitmap and (b) the > > following equation in tbm_crea

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-13 Thread Etsuro Fujita
I wrote: > Robert Haas wrote: > > I'd be wary of showing a desired value unless it's highly likely to be > > accurate. > The desired value is accurately estimated based on (a) the total number > of exact/lossy pages stored in the TIDBitmap and (b) the following equation > in tbm_create(), except f

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-08 Thread Etsuro Fujita
Robert Haas wrote: > On Fri, Dec 6, 2013 at 5:02 AM, Etsuro Fujita > wrote: > > Though at first I agreed on this, while working on this I start to > > think information about (2) is enough for tuning work_mem. Here are > > examples using a version under development, where "Bitmap Memory > > Usage

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-07 Thread Robert Haas
On Fri, Dec 6, 2013 at 5:02 AM, Etsuro Fujita wrote: > Though at first I agreed on this, while working on this I start to think > information about (2) is enough for tuning work_mem. Here are examples using > a version under development, where "Bitmap Memory Usage" means (peak) memory > space

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-06 Thread Etsuro Fujita
I wrote: > Amit Khandekar wrote: > > Yes, I agree that rather than looking at the bitmap heap scan to track > > the number of pages, we should look somewhere in the underlying index > > scan. Yes, we should get a constant number of index pages regardless > > of the actual parent table rows. > I ag

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-26 Thread Etsuro Fujita
Amit Khandekar wrote: > On 25 November 2013 13:37, Etsuro Fujita wrote: >> So, my question is, we should show the number of exact/lossy pages in a >> TIDBitmap, not the number of these pages that has been fetched in the bitmap >> heap scan? > Yes, I agree that rather than looking at the bitmap

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-25 Thread Amit Khandekar
On 25 November 2013 13:37, Etsuro Fujita wrote: > > Reconsidering that, I wish to know your opinion. The patch shows the > number of exact/lossy pages that has been fetched in a bitmap heap scan. > But the number varies with the fraction of tuples to be retrieved like the > following. > > postgr

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-25 Thread Etsuro Fujita
From: Amit Khandekar [mailto:amit.khande...@enterprisedb.com] > On 1 November 2013 16:32, Etsuro Fujita wrote: >> From: Fujii Masao [mailto:masao.fu...@gmail.com] >>> I'm not sure if it's good idea to show the number of the fetches because it >>> seems difficult to tune work_mem from that number.

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-24 Thread Amit Khandekar
On 1 November 2013 16:32, Etsuro Fujita wrote: > > From: Fujii Masao [mailto:masao.fu...@gmail.com] > > > This is what I'm looking for! This feature is really useful for tuning > work_mem > > when using full text search with pg_trgm. > > > > I'm not sure if it's good idea to show the number of th

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-01 Thread Etsuro Fujita
> From: Fujii Masao [mailto:masao.fu...@gmail.com] > This is what I'm looking for! This feature is really useful for tuning work_mem > when using full text search with pg_trgm. > > I'm not sure if it's good idea to show the number of the fetches because it > seems difficult to tune work_mem from

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-10-31 Thread Fujii Masao
On Thu, Oct 31, 2013 at 7:54 PM, Etsuro Fujita wrote: > Hi, > > I think that lossy-heap-block information for a bitmap heap scan, not just > "Rows > Removed by Index Recheck" information, would also be a clue used to tune > work_mem for better performance especially when the bitmap heap scan uses

[HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-10-31 Thread Etsuro Fujita
Hi, I think that lossy-heap-block information for a bitmap heap scan, not just "Rows Removed by Index Recheck" information, would also be a clue used to tune work_mem for better performance especially when the bitmap heap scan uses an index such as gin or gist, not btree. So here's a patch that a