Re: Use read streams in pg_visibility

2024-09-11 Thread Noah Misch
On Wed, Sep 11, 2024 at 09:19:09AM +0300, Nazir Bilal Yavuz wrote: > On Wed, 11 Sept 2024 at 01:38, Noah Misch wrote: > > I also fixed the mix of tabs and spaces inside test file string literals. > > I ran both pgindent and pgperltidy but they didn't catch them. Is > there an automated way to cat

Re: Use read streams in pg_visibility

2024-09-10 Thread Nazir Bilal Yavuz
Hi, On Wed, 11 Sept 2024 at 01:38, Noah Misch wrote: > > On Tue, Sep 10, 2024 at 02:35:46PM +0300, Nazir Bilal Yavuz wrote: > > Your patch is correct. I wrongly assumed it would catch blockno bug, > > the attached version catches it. I made blockno = 0 invisible and not > > frozen before copying

Re: Use read streams in pg_visibility

2024-09-10 Thread Noah Misch
On Tue, Sep 10, 2024 at 02:35:46PM +0300, Nazir Bilal Yavuz wrote: > Your patch is correct. I wrongly assumed it would catch blockno bug, > the attached version catches it. I made blockno = 0 invisible and not > frozen before copying the vm file. So, in the blockno buggy version; > callback will sk

Re: Use read streams in pg_visibility

2024-09-10 Thread Nazir Bilal Yavuz
Hi, On Tue, 10 Sept 2024 at 00:32, Noah Misch wrote: > > Copying the vm file is a good technique. In my test runs, this does catch the > "never detect" bug, but it doesn't catch the blkno bug. Can you make it catch > both? It's possible my hand-patching to recreate the blkno bug is what went >

Re: Use read streams in pg_visibility

2024-09-09 Thread Noah Misch
On Mon, Sep 09, 2024 at 06:25:07PM +0300, Nazir Bilal Yavuz wrote: > On Thu, 5 Sept 2024 at 18:54, Noah Misch wrote: > > On Thu, Sep 05, 2024 at 03:59:53PM +0300, Nazir Bilal Yavuz wrote: > > > On Wed, 4 Sept 2024 at 21:43, Noah Misch wrote: > > > > https://postgr.es/m/caeudqaozv3wty5tv2t29jcwpyd

Re: Use read streams in pg_visibility

2024-09-09 Thread Nazir Bilal Yavuz
Hi, On Thu, 5 Sept 2024 at 18:54, Noah Misch wrote: > > On Thu, Sep 05, 2024 at 03:59:53PM +0300, Nazir Bilal Yavuz wrote: > > On Wed, 4 Sept 2024 at 21:43, Noah Misch wrote: > > > https://postgr.es/m/caeudqaozv3wty5tv2t29jcwpydbmkbiwqkzd42s2ogzdixp...@mail.gmail.com > > > then observed that col

Re: Use read streams in pg_visibility

2024-09-05 Thread Noah Misch
On Thu, Sep 05, 2024 at 03:59:53PM +0300, Nazir Bilal Yavuz wrote: > On Wed, 4 Sept 2024 at 21:43, Noah Misch wrote: > > https://postgr.es/m/caeudqaozv3wty5tv2t29jcwpydbmkbiwqkzd42s2ogzdixp...@mail.gmail.com > > then observed that collect_corrupt_items() was now guaranteed to never > > detect > >

Re: Use read streams in pg_visibility

2024-09-05 Thread Nazir Bilal Yavuz
Hi, On Wed, 4 Sept 2024 at 21:43, Noah Misch wrote: > > https://postgr.es/m/caeudqaozv3wty5tv2t29jcwpydbmkbiwqkzd42s2ogzdixp...@mail.gmail.com > then observed that collect_corrupt_items() was now guaranteed to never detect > corruption. I have pushed revert ddfc556 of the pg_visibility.c changes

Re: Use read streams in pg_visibility

2024-09-04 Thread Noah Misch
On Tue, Sep 03, 2024 at 10:46:34PM +0300, Nazir Bilal Yavuz wrote: > On Tue, 3 Sept 2024 at 22:20, Noah Misch wrote: > > On Tue, Sep 03, 2024 at 10:50:11AM -0700, Noah Misch wrote: > > > Pushed with some other cosmetic changes. > > Thanks! > > > I see I pushed something unacceptable under ASAN.

Re: Use read streams in pg_visibility

2024-09-03 Thread Nazir Bilal Yavuz
Hi, On Tue, 3 Sept 2024 at 22:20, Noah Misch wrote: > > On Tue, Sep 03, 2024 at 10:50:11AM -0700, Noah Misch wrote: > > On Mon, Sep 02, 2024 at 03:20:12PM +0300, Nazir Bilal Yavuz wrote: > > > Thanks, updated patches are attached. > > > > > +/* > > > + * Ask the callback which block it would like

Re: Use read streams in pg_visibility

2024-09-03 Thread Noah Misch
On Tue, Sep 03, 2024 at 10:50:11AM -0700, Noah Misch wrote: > On Mon, Sep 02, 2024 at 03:20:12PM +0300, Nazir Bilal Yavuz wrote: > > Thanks, updated patches are attached. > > > +/* > > + * Ask the callback which block it would like us to read next, with a small > > + * buffer in front to allow rea

Re: Use read streams in pg_visibility

2024-09-03 Thread Noah Misch
On Mon, Sep 02, 2024 at 03:20:12PM +0300, Nazir Bilal Yavuz wrote: > Thanks, updated patches are attached. > +/* > + * Ask the callback which block it would like us to read next, with a small > + * buffer in front to allow read_stream_unget_block() to work and to allow > the > + * fast path to sk

Re: Use read streams in pg_visibility

2024-09-02 Thread Nazir Bilal Yavuz
Hi, On Sat, 31 Aug 2024 at 02:51, Noah Misch wrote: > > To read blocks 10 and 11, I would expect to initialize the struct with one of: > > { .first=10, .nblocks=2 } > { .first=10, .last_inclusive=11 } > { .first=10, .last_exclusive=12 } > > With the patch's API, I would need {.first=10,.nblocks=1

Re: Use read streams in pg_visibility

2024-08-30 Thread Noah Misch
On Tue, Aug 27, 2024 at 10:49:19AM +0300, Nazir Bilal Yavuz wrote: > On Fri, 23 Aug 2024 at 22:01, Noah Misch wrote: > > On Fri, Aug 23, 2024 at 02:20:06PM +0300, Nazir Bilal Yavuz wrote: > > > On Tue, 20 Aug 2024 at 21:47, Noah Misch wrote: > > > > On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir

Re: Use read streams in pg_visibility

2024-08-27 Thread Nazir Bilal Yavuz
Hi, On Fri, 23 Aug 2024 at 22:01, Noah Misch wrote: > > On Fri, Aug 23, 2024 at 02:20:06PM +0300, Nazir Bilal Yavuz wrote: > > On Tue, 20 Aug 2024 at 21:47, Noah Misch wrote: > > > On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir Bilal Yavuz wrote: > > > > The downside of this approach is there a

Re: Use read streams in pg_visibility

2024-08-23 Thread Noah Misch
On Fri, Aug 23, 2024 at 02:20:06PM +0300, Nazir Bilal Yavuz wrote: > On Tue, 20 Aug 2024 at 21:47, Noah Misch wrote: > > On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir Bilal Yavuz wrote: > > > The downside of this approach is there are too many "vmbuffer is valid > > > but BufferGetBlockNumber(*v

Re: Use read streams in pg_visibility

2024-08-23 Thread Nazir Bilal Yavuz
Hi, Thanks for the review and feedback! On Tue, 20 Aug 2024 at 21:47, Noah Misch wrote: > > On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir Bilal Yavuz wrote: > > 2- collect_corrupt_items() > > > > This one is more complicated. The read stream callback function loops > > until it finds a suitabl

Re: Use read streams in pg_visibility

2024-08-20 Thread Noah Misch
On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir Bilal Yavuz wrote: > 2- collect_corrupt_items() > > This one is more complicated. The read stream callback function loops > until it finds a suitable block to read. So, if the callback returns > an InvalidBlockNumber; it means that the stream process

Re: Use read streams in pg_visibility

2024-08-19 Thread Nazir Bilal Yavuz
Hi, On Mon, 19 Aug 2024 at 09:30, Michael Paquier wrote: > > On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir Bilal Yavuz wrote: > > Hi, > > > > I am working on using the read stream in pg_visibility. There are two > > places to use it: > > > > 1- collect_visibility_data() > > > > This one is stra

Re: Use read streams in pg_visibility

2024-08-18 Thread Michael Paquier
On Tue, Aug 13, 2024 at 03:22:27PM +0300, Nazir Bilal Yavuz wrote: > Hi, > > I am working on using the read stream in pg_visibility. There are two > places to use it: > > 1- collect_visibility_data() > > This one is straightforward. I created a read stream object if > 'include_pd' is true becaus

Use read streams in pg_visibility

2024-08-13 Thread Nazir Bilal Yavuz
Hi, I am working on using the read stream in pg_visibility. There are two places to use it: 1- collect_visibility_data() This one is straightforward. I created a read stream object if 'include_pd' is true because read I/O is done when 'include_pd' is true. There is ~4% timing improvement with th