Re: Proposal to add page headers to SLRU pages

> On Nov 10, 2024, at 01:32, Rishu Bagga wrote: > > External Email > >> Thanks for pointing this out. Here is what I have tried: >> 1. Manually build and install PostgreSQL from the latest source code. >> 2. Following the instructions from src/bin/pg_upgrade to manually dump the >> regression >

Re: Proposal to add page headers to SLRU pages

o get this in? Thanks, Rishu Bagga On Sat, Nov 9, 2024 at 8:50 AM Li, Yong wrote: > > > > > On Jun 10, 2024, at 16:01, Michael Paquier wrote: > > > > External Email > > > > From: Michael Paquier > > Subject: Re: Proposal to add page headers to SL

Re: Proposal to add page headers to SLRU pages

> On Jun 10, 2024, at 16:01, Michael Paquier wrote: > > External Email > > From: Michael Paquier > Subject: Re: Proposal to add page headers to SLRU pages > Date: June 10, 2024 at 16:01:50 GMT+8 > To: Bertrand Drouvot > Cc: "Li, Yong" , Jeff Davis

Re: Proposal to add page headers to SLRU pages

On Mon, Jun 10, 2024 at 07:19:56AM +, Bertrand Drouvot wrote: > On Tue, Mar 19, 2024 at 06:48:33AM +, Li, Yong wrote: >> Unfortunately, the test requires a setup of two different versions of PG. I >> am not >> aware of an existing test infrastructure which can run automated tests using >>

Re: Proposal to add page headers to SLRU pages

Hi, On Tue, Mar 19, 2024 at 06:48:33AM +, Li, Yong wrote: > > Unfortunately, the test requires a setup of two different versions of PG. I > am not > aware of an existing test infrastructure which can run automated tests using > two > PGs. I did manually verify the output of pg_upgrade. I

Re: Proposal to add page headers to SLRU pages

>> - New comments have been added to pg_upgrade to mention the SLRU >> page header change as the reason for upgrading clog files. > > That seems reasonable, but were any alternatives discussed? Do we have > consensus that this is the right thing to do? In general, there are two approaches. Eith

Re: Proposal to add page headers to SLRU pages

On Mon, 2024-03-11 at 10:01 +, Li, Yong wrote: > - The clog LSN group has been brought back. >   Now the page LSN on each clog page is used for honoring the write- > ahead rule >   and it is always the highest LSN of all the LSN groups on the page. >   The LSN groups are used by TransactionIdGe

Re: Proposal to add page headers to SLRU pages

> On Mar 9, 2024, at 05:22, Jeff Davis wrote: > > External Email > > On Wed, 2024-03-06 at 12:01 +, Li, Yong wrote: >> Rebase the patch against the latest HEAD. > > The upgrade logic could use more comments explaining what's going on > and why. As I understand it, it's a one-time conversio

Re: Proposal to add page headers to SLRU pages

On Wed, 2024-03-06 at 12:01 +, Li, Yong wrote: > Rebase the patch against the latest HEAD. The upgrade logic could use more comments explaining what's going on and why. As I understand it, it's a one-time conversion that needs to happen between 16 and 17. Is that right? Was the way CLOG is up

Re: Proposal to add page headers to SLRU pages

On Fri, 2024-03-08 at 12:39 -0800, Jeff Davis wrote: > Making a better secondary structure seems doable to me. Just to > brainstorm: We can also keep the group_lsns, and then just update both the CLOG page LSN and the group_lsn when setting the transaction status. The former would be used for all

Re: Proposal to add page headers to SLRU pages

On Fri, 2024-03-08 at 13:58 +0100, Alvaro Herrera wrote: > In the new code we effectively store only one LSN per page, which I > understand is strictly worse. To quote from the README: "Instead, we defer the setting of the hint bit if we have not yet flushed WAL as far as the LSN associated with

Re: Proposal to add page headers to SLRU pages

On 2024-Mar-08, Stephen Frost wrote: > Thanks for testing! That strikes me as perfectly reasonable and seems > unlikely that we'd get much benefit from parallelizing it, so I'd say it > makes sense to keep this code simple. Okay, agreed, that amount of time sounds reasonable to me too; but I don

Re: Proposal to add page headers to SLRU pages

Greetings, * Li, Yong (y...@ebay.com) wrote: > > On Mar 7, 2024, at 03:09, Stephen Frost wrote: > > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > >> I suppose this is important to do if we ever want to move SLRUs into > >> shared buffers. However, I wonder about the extra time this adds to

Re: Proposal to add page headers to SLRU pages

> On Mar 7, 2024, at 03:09, Stephen Frost wrote: > > External Email > > From: Stephen Frost > Subject: Re: Proposal to add page headers to SLRU pages > Date: March 7, 2024 at 03:09:59 GMT+8 > To: Alvaro Herrera > Cc: "Li, Yong" , Aleksander Alekseev >

Re: Proposal to add page headers to SLRU pages

Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > I suppose this is important to do if we ever want to move SLRUs into > shared buffers. However, I wonder about the extra time this adds to > pg_upgrade. Is this something we should be concerned about? Is there > any measurement/esti

Re: Proposal to add page headers to SLRU pages

Hello, I suppose this is important to do if we ever want to move SLRUs into shared buffers. However, I wonder about the extra time this adds to pg_upgrade. Is this something we should be concerned about? Is there any measurement/estimates to tell us how long this would be? Right now, if you us

Re: Proposal to add page headers to SLRU pages

Rebase the patch against the latest HEAD. Regards, Yong slru_page_header_v5.patch Description: slru_page_header_v5.patch

Re: Proposal to add page headers to SLRU pages

Rebase the patch against the latest HEAD. Regards, Yong slru_page_header_v4.patch Description: slru_page_header_v4.patch

Re: Proposal to add page headers to SLRU pages

> On Jan 2, 2024, at 19:35, Aleksander Alekseev > wrote: > > Thanks for the updated patch. > > cfbot seems to have some complaints regarding compiler warnings and > also building the patch on Windows: > > http://cfbot.cputube.org/ Thanks for the information. Here is the updated patch. Regard

Re: Proposal to add page headers to SLRU pages

Hi, > I have also added this thread to the current Commitfest and hope this patch > will be part of the 17 release. > > The commitfest link: > https://commitfest.postgresql.org/46/4709/ Thanks for the updated patch. cfbot seems to have some complaints regarding compiler warnings and also buildi

Re: Proposal to add page headers to SLRU pages

> This work is being done in file.c – it seems to me the proper way to > proceed would be to continue writing on-disk upgrade logic here. > Besides that this looks good to me, would like to hear what others have to > say. Thank you, Rishu for taking time to review the code. I've updated the pat

Re: Proposal to add page headers to SLRU pages

On Thu, Dec 8, 2023 at 1:36 AM Li, Yong wrote: >Given so many different approaches were discussed, I have started a >wiki to record and collaborate all efforts towards SLRU >improvements. The wiki provides a concise overview of all the ideas >discussed and can serve as a portal for all histo

Re: Proposal to add page headers to SLRU pages

: Re: Proposal to add page headers to SLRU pages External Email On Thu, Dec 7, 2023 at 1:28 PM Debnath, Shawn wrote: > What is being proposed now is the simple and core functionality of introducing > page headers to SLRU pages while continuing to be in the SLRU cache. This > allows

Re: Proposal to add page headers to SLRU pages

On Thu, Dec 7, 2023 at 1:28 PM Debnath, Shawn wrote: > What is being proposed now is the simple and core functionality of introducing > page headers to SLRU pages while continuing to be in the SLRU cache. This > allows the whole project to be iterative and reviewers to better reason about > the sm

Re: Proposal to add page headers to SLRU pages

>> Sounds like a half-measure to me. If we really want to go down this >> rabbit hole IMO SLRU should be moved to shared buffers as proposed >> elsewhere [1]. > Thread that I cited stopped in 2018 for this exact reason. 5 years ago. Is > this argument still valid? Meanwhile checksums of buffer p

Re: Proposal to add page headers to SLRU pages

07.12.2023, 19:17, "Aleksander Alekseev" :Hi, +1 to the idea to protect SLRUs from corruption. I'm slightly leaning towards the idea of separating checksums from data pages, but anyway this checksums are better than no checksums. On 7 Dec 2023, at 10:06, Li, Yong wrote: I am still w

Re: Proposal to add page headers to SLRU pages

Hi, > +1 to the idea to protect SLRUs from corruption. I'm slightly leaning towards > the idea of separating checksums from data pages, but anyway this checksums > are better than no checksums. > > On 7 Dec 2023, at 10:06, Li, Yong wrote: > > I am still working on patching the pg_upgrade. Just

Re: Proposal to add page headers to SLRU pages

Hi Yong! +1 to the idea to protect SLRUs from corruption. I'm slightly leaning towards the idea of separating checksums from data pages, but anyway this checksums are better than no checksums. > On 7 Dec 2023, at 10:06, Li, Yong wrote: > > I am still working on patching the pg_upgrade. Just