Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Sat, Aug 10, 2024 at 10:50:55AM -0700, Noah Misch wrote: > On Sat, Jul 27, 2024 at 07:24:33AM +0900, Michael Paquier wrote: >> I've just applied now the remaining pieces down to 17. > > Comparing commit c9e2457 to the patch in zqgvzsbw5tgkq...@paquier.xyz, the > commit lacks the slru.c portion.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Sat, Jul 27, 2024 at 07:24:33AM +0900, Michael Paquier wrote: > I've just applied now the remaining pieces down to 17. Comparing commit c9e2457 to the patch in zqgvzsbw5tgkq...@paquier.xyz, the commit lacks the slru.c portion.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Fri, Jul 26, 2024 at 11:50:48PM +0300, Alexander Korotkov wrote: > Thanks to everybody for working on this. It's pity I didn't notice > this is v17 open item on me. Sorry for this. No problem. I've just applied now the remaining pieces down to 17. -- Michael signature.asc Description: PGP

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Fri, Jul 26, 2024 at 3:42 AM Noah Misch wrote: > On Thu, Jul 25, 2024 at 10:52:13AM +0900, Michael Paquier wrote: > > On Wed, Jul 24, 2024 at 06:00:59AM -0700, Noah Misch wrote: > > > I'm still seeing need for s/int/int64/ at: > > > I am attaching a patch for all these you have spotted, switchi

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Thu, Jul 25, 2024 at 10:52:13AM +0900, Michael Paquier wrote: > On Wed, Jul 24, 2024 at 06:00:59AM -0700, Noah Misch wrote: > > I'm still seeing need for s/int/int64/ at: > I am attaching a patch for all these you have spotted, switching the > logs to use %llx. Does that look fine for you? Ye

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Wed, Jul 24, 2024 at 06:00:59AM -0700, Noah Misch wrote: > I'm still seeing need for s/int/int64/ at: Nice catches! I've missed these. > - "pagesegno" variable > - return value of MultiXactIdToOffsetSegment() Only used in four places for two elog(DEBUG1) entries with %x. > - return value of

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Tue, Jul 23, 2024 at 06:01:44PM +0900, Michael Paquier wrote: > Hearing nothing but cicadas as now is their season, I have taken the > initiative here to address this open item. > > 0001 felt a bit too complicated in slru.h, so I have simplified it and > kept all the details in slru.c with Slru

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > Hearing nothing but cicadas as now is their season, I have taken the > initiative here to address this open item. > > 0001 felt a bit too complicated in slru.h, so I have simplified it and > kept all the details in slru.c with SlruFileName(). > > I have reviewed all the code that uses SLRUs,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Fri, Jul 12, 2024 at 12:44:54PM +0300, Aleksander Alekseev wrote: > Fair enough. Here is the updated patchset. Hearing nothing but cicadas as now is their season, I have taken the initiative here to address this open item. 0001 felt a bit too complicated in slru.h, so I have simplified it and

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > A comment about v3-0001. > > -* If true, use long segment filenames formed from lower 48 bits of > the > -* segment number, e.g. pg_xact/1234. Otherwise, use short > -* filenames formed from lower 16 bits of the segment number e.g. > -* pg_xact/1234.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Thu, Jul 11, 2024 at 01:11:05PM +0300, Aleksander Alekseev wrote: > Thanks, Michael. I prepared a corrected patchset. A comment about v3-0001. -* If true, use long segment filenames formed from lower 48 bits of the -* segment number, e.g. pg_xact/1234. Otherwise, use sh

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > The proposed patch looks rather incomplete to me, based on the fact > that this stuff has a lot of inconsistencies with the types used when > manipulating 64b SLRU pages. Some of them are harder to catch as the > variables don't specifically refer to pages. > > So, even after v2, there are

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Jul 08, 2024 at 12:30:09PM +0300, Aleksander Alekseev wrote: > TWIMC this is currently listed as an open item for PG17 [1]. > Sorry if everyone interested is already aware. > > [1]: https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items The proposed patch looks rather incomplete to me,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > Here is the corrected patchset. TWIMC this is currently listed as an open item for PG17 [1]. Sorry if everyone interested is already aware. [1]: https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items -- Best regards, Aleksander Alekseev

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > > I prepared the patch for clog.c. The rest of the warnings don't strike > > me as something we should immediately act on unless we have a bug > > report. Or perhaps there is a particular warning that worries you? > > Is "int" acceptable or unacceptable in the following grep match? > > src/b

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Wed, Jun 26, 2024 at 02:09:58PM +0300, Aleksander Alekseev wrote: > > This yielded commit 4ed8f09 "Index SLRUs by 64-bit integers rather than by > > 32-bit integers" and left some expressions coercing SLRU page numbers to > > int. > > Two sources: > > > > grep -i 'int\b.*page' $(git grep -l Si

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi Noah, > This yielded commit 4ed8f09 "Index SLRUs by 64-bit integers rather than by > 32-bit integers" and left some expressions coercing SLRU page numbers to int. > Two sources: > > grep -i 'int\b.*page' $(git grep -l SimpleLruInit) > make && touch $(git grep -l SimpleLruInit) && make PROFILE

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Nov 27, 2023 at 01:43:26AM +0200, Alexander Korotkov wrote: > v61 looks good to me. I'm going to push it as long as there are no > objections. This yielded commit 4ed8f09 "Index SLRUs by 64-bit integers rather than by 32-bit integers" and left some expressions coercing SLRU page numbers

Re: XID formatting and SLRU refactorings

On Tue, Nov 28, 2023 at 11:14 PM Heikki Linnakangas wrote: > I think it's pretty sloppy that the "short" filenames can be 4, 5 or 6 > chars long. For pg_multixact/members, which introduced the 5-char case, > I think we should always pad the filenames 5 characters, and for > commit_ts which introdu

Re: XID formatting and SLRU refactorings

On Sun, Dec 17, 2023 at 1:48 AM Thomas Munro wrote: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=loach&dt=2023-12-16%2005%3A25%3A18 > > TRAP: failed Assert("epoch > 0"), File: "twophase.c", Line: 969, PID: 71030 > 0xa8edcd at > /usr/home/pgbf/buildroot/HEAD/pgsql.build/tmp_install/ho

Re: XID formatting and SLRU refactorings

Hi, https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=loach&dt=2023-12-16%2005%3A25%3A18 TRAP: failed Assert("epoch > 0"), File: "twophase.c", Line: 969, PID: 71030 0xa8edcd at /usr/home/pgbf/buildroot/HEAD/pgsql.build/tmp_install/home/pgbf/buildroot/HEAD/inst/bin/postgres 0x613863 at /us

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Dec 4, 2023 at 3:12 PM Pavel Borisov wrote: > I think this is complete and could be closed. Done.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, 4 Dec 2023 at 10:34, John Naylor wrote: > On Thu, Nov 30, 2023 at 4:37 PM Alexander Korotkov > wrote: > > > > On Thu, Nov 30, 2023 at 10:29 AM Pavel Borisov > wrote: > > > Agree. The fix is attached. > > > > What an oversight. > > Thank you, pushed! > > With that, is there any more work

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Thu, Nov 30, 2023 at 4:37 PM Alexander Korotkov wrote: > > On Thu, Nov 30, 2023 at 10:29 AM Pavel Borisov wrote: > > Agree. The fix is attached. > > What an oversight. > Thank you, pushed! With that, is there any more work pending, or can we close the CF entry?

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Thu, Nov 30, 2023 at 10:29 AM Pavel Borisov wrote: > Agree. The fix is attached. What an oversight. Thank you, pushed! -- Regards, Alexander Korotkov

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Thu, 30 Nov 2023 at 08:03, Tom Lane wrote: > Alexander Lakhin writes: > > And a warning: > > $ CC=gcc-12 CFLAGS="-Wall -Wextra -Wno-unused-parameter > -Wno-sign-compare -Wno-clobbered > > -Wno-missing-field-initializers" ./configure -q && make -s > > slru.c:63:1: warning: ‘inline’ is not at b

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Alexander Lakhin writes: > And a warning: > $ CC=gcc-12 CFLAGS="-Wall -Wextra -Wno-unused-parameter -Wno-sign-compare > -Wno-clobbered > -Wno-missing-field-initializers" ./configure -q && make -s > slru.c:63:1: warning: ‘inline’ is not at beginning of declaration > [-Wold-style-declaration] >

Re: XID formatting and SLRU refactorings

Hi! On Tue, Nov 28, 2023 at 2:06 PM Aleksander Alekseev wrote: > > >> > I think what's done in patch 0001 is just an extension of existing > >> > logic and moving it into separate function. > >> > >> That's right. I'm arguing that now is a good time to clean it up. > >> > >> I won't insist if Ale

Re: XID formatting and SLRU refactorings

Hi, >> > I think what's done in patch 0001 is just an extension of existing >> > logic and moving it into separate function. >> >> That's right. I'm arguing that now is a good time to clean it up. >> >> I won't insist if Alexander prefers to commit it as it is, though. But >> let's at least explai

Re: XID formatting and SLRU refactorings

On Tue, 28 Nov 2023 at 14:37, Heikki Linnakangas wrote: > On 28/11/2023 12:14, Pavel Borisov wrote: > > On Tue, 28 Nov 2023 at 13:13, Heikki Linnakangas > wrote: > >> > >> On 27/11/2023 01:43, Alexander Korotkov wrote: > >>> v61 looks good to me. I'm going to push it as long as there are no > o

Re: XID formatting and SLRU refactorings

On 28/11/2023 12:14, Pavel Borisov wrote: On Tue, 28 Nov 2023 at 13:13, Heikki Linnakangas wrote: On 27/11/2023 01:43, Alexander Korotkov wrote: v61 looks good to me. I'm going to push it as long as there are no objections. This was discussed earlier, but is still present in v61: +/* + *

Re: XID formatting and SLRU refactorings

Hi, Heikki! On Tue, 28 Nov 2023 at 13:13, Heikki Linnakangas wrote: > > On 27/11/2023 01:43, Alexander Korotkov wrote: > > v61 looks good to me. I'm going to push it as long as there are no > > objections. > This was discussed earlier, but is still present in v61: > > > +/* > > + * An internal

Re: XID formatting and SLRU refactorings

On 27/11/2023 01:43, Alexander Korotkov wrote: v61 looks good to me. I'm going to push it as long as there are no objections. This was discussed earlier, but is still present in v61: +/* + * An internal function used by SlruScanDirectory(). + * + * Returns true if a file with a name of a give

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Андрей, привет! Текущее положение у меня такое. . *pg_stats and range statisticsTracking statements entry timestamp in pg_stat_statements* Уже закоммичены. *XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)* Если всё будет и замечаний не возникнет ок, завтра утром

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hello Alexander, 27.11.2023 02:43, Alexander Korotkov wrote: v61 looks good to me. I'm going to push it as long as there are no objections. I've looked at the patch set and found a typo: occured And a warning: $ CC=gcc-12 CFLAGS="-Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-c

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Alexander, Maxim, Thank you for revisions. On Thu, Nov 9, 2023 at 6:22 PM Maxim Orlov wrote: > Aleksander Alekseev, > > > Maxim, > > I see both of us accounted for Alexanders feedback and submitted v59. > > Your newer version seems to have issues on cfbot, so resubmitting the > > previous patchs

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Aleksander Alekseev, > Maxim, > I see both of us accounted for Alexanders feedback and submitted v59. > Your newer version seems to have issues on cfbot, so resubmitting the > previous patchset that passes the tests. Please feel free to add > changes. For unknown reasons, I do not receive any of

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Maxim, I see both of us accounted for Alexanders feedback and submitted v59. Your newer version seems to have issues on cfbot, so resubmitting the previous patchset that passes the tests. Please feel free to add changes. > See SlruCorrectSegmentFilenameLength(): > > ``` > if (ctl->long_segmen

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, 6 Nov 2023 at 16:07, Alexander Korotkov wrote: > Hi! > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev < > aleksan...@timescale.com> wrote: > > PFE the corrected patchset v58. > > I'd like to revive this thread. > Hi! Great news! > BTW, there is a typo in a word "exceeed". > Fixed.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi again, > PFA the corrected patchset v59. On second thought, I believe this Assert is incorrect: ``` +else +{ +Assert(segno >= 0 && segno <= 0x); +return snprintf(path, MAXPGPATH, "%s/%04X", (ctl)->Dir, +(unsigned int) segno); +} ``` See

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi Alexander, > -#define TransactionIdToCTsPage(xid) \ > - ((xid) / (TransactionId) COMMIT_TS_XACTS_PER_PAGE) > + > +/* > + * Although we return an int64 the actual value can't currently exceeed > 2**32. > + */ > +static inline int64 > +TransactionIdToCTsPage(TransactionId xid) > +{ > + retur

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Nov 6, 2023 at 4:38 PM Aleksander Alekseev wrote: > > > PFE the corrected patchset v58. > > > > I'd like to revive this thread. > > Many thanks for your comments and suggestions. > > > I think it worth adding asserts here to verify there is no overflow making us mapping different segments

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Alexander, > > PFE the corrected patchset v58. > > I'd like to revive this thread. Many thanks for your comments and suggestions. > I think it worth adding asserts here to verify there is no overflow making us > mapping different segments into the same files. Sorry, I didn't understand this on

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > > > If I remember right, the compiler will make equivalent code from > > > inline functions and macros, and functions has an additional benefit: > > > the compiler will report type mismatch if any. That was the only > > > reason. > > > > Then it's OK to leave it as an inline function. +1 >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, 6 Nov 2023 at 18:01, Alexander Korotkov wrote: > > On Mon, Nov 6, 2023 at 3:42 PM Pavel Borisov wrote: > > > > On Mon, 6 Nov 2023 at 17:07, Alexander Korotkov > > wrote: > > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev > > > wrote: > > > > PFE the corrected patchset v58. > > >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Nov 6, 2023 at 3:42 PM Pavel Borisov wrote: > > On Mon, 6 Nov 2023 at 17:07, Alexander Korotkov wrote: > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev > > wrote: > > > PFE the corrected patchset v58. > > > > I'd like to revive this thread. > > > > This patchset is extracted from

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, 6 Nov 2023 at 17:07, Alexander Korotkov wrote: > > Hi! > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev > wrote: > > PFE the corrected patchset v58. > > I'd like to revive this thread. > > This patchset is extracted from a larger patchset implementing 64-bit xids. > It converts p

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi! On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev wrote: > PFE the corrected patchset v58. I'd like to revive this thread. This patchset is extracted from a larger patchset implementing 64-bit xids. It converts page numbers in SLRUs into 64 bits. The most SLRUs save the same file naming

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > > Reviewing this now. I think it's almost ready to be committed. > > > > There's another big effort going on to move SLRUs to the regular buffer > > cache (https://commitfest.postgresql.org/43/3514/). I wonder how moving > > to 64 bit page numbers will affect that. BlockNumber is still 32 bi

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > Reviewing this now. I think it's almost ready to be committed. > > There's another big effort going on to move SLRUs to the regular buffer > cache (https://commitfest.postgresql.org/43/3514/). I wonder how moving > to 64 bit page numbers will affect that. BlockNumber is still 32 bits, > afte

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 09/03/2023 17:21, Aleksander Alekseev wrote: v57-0001-Index-SLRUs-by-64-bit-integers-rather-than-by-32.patch Reviewing this now. I think it's almost ready to be committed. There's another big effort going on to move SLRUs to the regular buffer cache (https://commitfest.postgresql.org/43/35

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > As suggested before by Heikki Linnakangas, I've added a patch for making 2PC > transaction state 64-bit. > At first, my intention was to rebuild all twophase interface to use > FullTransactionId. But doing this in a proper > manner would lead to switching from TransactionId to FullTransact

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi! As suggested before by Heikki Linnakangas, I've added a patch for making 2PC transaction state 64-bit. At first, my intention was to rebuild all twophase interface to use FullTransactionId. But doing this in a proper manner would lead to switching from TransactionId to FullTransactionId in PGP

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > 1. Use internal 64 bit page numbers in SLRUs without changing segments naming. > 2. Use the larger segment file names in async.c, to lift the current 8 > GB limit on the max number of pending notifications. > [...] > > Where our main focus for PG16 is going to be 1 and 2, and we may try > to

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Tue, 7 Mar 2023 at 15:38, Heikki Linnakangas wrote: > > That is true for pg_multixact/offsets. We will indeed need to add an > epoch and introduce the concept of FullMultiXactIds for that. However, > we can change pg_multixact/members independently of that. We can extend > MultiXactOffset from

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > Here's how I see the development path for this [...] > So, in my view, the plan should be [...] > Thoughts? The plan looks great! I would also explicitly include this: > As we start to refactor these things, I also think it would be good to > have more explicit tracking of the valid range

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 07/03/2023 13:38, Maxim Orlov wrote: As for making pg_multixact 64 bit, I spend the last couple of days to make proper pg_upgrade for pg_multixact's and for pg_xact's with wraparound and I've understood, that it is not a simple task compare to pg_xact's. The problem is, we do not have epoch f

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Tue, 17 Jan 2023 at 16:33, Aleksander Alekseev wrote: > Hi hackers, > > Maxim, perhaps you could share with us what your reasoning was here? > > I'm really sorry for late response, but better late than never. Yes, we can not access shared memory without lock. In this particular case, we use Xi

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 01/03/2023 12:21, Aleksander Alekseev wrote: Hi, I'm surprised that these patches extend the page numbering to 64 bits, but never actually uses the high bits. The XID "epoch" is not used, and pg_xact still wraps around and the segment names are still reused. I thought we could stop doing tha

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > I'm surprised that these patches extend the page numbering to 64 bits, > but never actually uses the high bits. The XID "epoch" is not used, and > pg_xact still wraps around and the segment names are still reused. I > thought we could stop doing that. To clarify, the idea is to let CLOG gro

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 28/02/2023 16:17, Maxim Orlov wrote: Either I do not understand something, or the files from pg_commit_ts directory are not copied. Huh, yeah you're right, pg_upgrade doesn't copy pg_commit_ts. - Heikki

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, 27 Feb 2023 at 12:10, Heikki Linnakangas wrote: > (CC list trimmed, gmail wouldn't let me send otherwise) > > That sounds right for pg_serial, pg_notify, and pg_subtrans. But not for > pg_commit_ts and the pg_multixacts. > > This needs tests for pg_upgrading those SLRUs, after 0, 1 and N

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

(CC list trimmed, gmail wouldn't let me send otherwise) On 22/02/2023 16:29, Maxim Orlov wrote: On Tue, 21 Feb 2023 at 16:59, Aleksander Alekseev mailto:aleksan...@timescale.com>> wrote: One thing that still bothers me is that during the upgrade we only migrate the CLOG segments (i.e. p

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Tue, 21 Feb 2023 at 16:59, Aleksander Alekseev wrote: > Hi, > > One thing that still bothers me is that during the upgrade we only > migrate the CLOG segments (i.e. pg_xact / pg_clog) and completely > ignore all the rest of SLRUs: > > * pg_commit_ts > * pg_multixact/offsets > * pg_multixact/me

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > The convert_pg_xact_segments() function is still obviously > overengineered. As I understand, all it has to do is simply renaming > pg_xact/ to pg_xact/. Unfortunately I used up all the > mana for today and have to take a long rest in order to continue. PFA the corrected pat

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, > OK, here is the patchset v53 where I mostly modified the commit > messages. It is explicitly said that 0001 modifies the WAL records and > why we decided to do it in this patch. Additionally any mention of > 64-bit XIDs is removed since it is not guaranteed that the rest of the > patches are

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi hackers, > Yeah, pg_upgrade will briefly start and stop the old server to make > sure all the WAL is replayed, and won't transfer any of the files > over. AFAIK, major-version WAL changes are fine; it was the previous > claim that we could do it in a minor version that I was unsure about. OK,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Wed, Jan 11, 2023 at 1:48 AM Aleksander Alekseev wrote: > After reading [1] carefully it looks like we shouldn't worry about > this. The upgrade procedure explicitly requires to run `pg_ctl stop` > during step 8 of the upgrade procedure, i.e. not in the immediate mode > [2]. Yeah, pg_upgrade w

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi Maxim, > Secondly, shouldn't we introduce a new WAL record type in order to > make the code backward compatible with previous PG versions? I'm not > 100% sure how the upgrade procedure works in all the details. If it > requires the DBMS to be gracefully shut down before the upgrade then > we ar

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi Maxim, > Here is a new patch set. > I've added comments and make use GetClogDirName call in copy_subdir_files. Jacob Champion pointed out (offlist, cc:'ed) that we may be wrong on this one: > 0001 patch changes the SLRU internals without affecting the callers. > 0001 - should make SLRU inter

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi! Here is a new patch set. I've added comments and make use GetClogDirName call in copy_subdir_files. -- Best regards, Maxim Orlov. v52-0003-Make-pg_upgrade-from-32-bit-to-64-bit-SLRU.patch Description: Binary data v52-0002-Use-64-bit-pages-representation-in-SLRU-callers.patch Description:

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Fri, 6 Jan 2023 at 09:51, Japin Li wrote: > > For v51-0003. We can use GetClogDirName instead of GET_MAJOR_VERSION in > copy_subdir_files(). > Of course! Tanks! I'll address this in the next iteration, v52. -- Best regards, Maxim Orlov.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi Andrey, > Hi! I think that 64-bit xids are a very important feature and I want > to help advance it. That's why I want to try to understand a patch > better. Thanks for your interest to the patchset! > Do I get it right that the proposed v51 patchset only changes the SLRU > filenames and type

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

> > Do I get it right that the proposed v51 patchset only changes the SLRU > filenames and type of pageno representation? Is SLRU wraparound still > exactly there after 0x byte? > After applying the whole patch set, SLRU will become 64–bit without a wraparound. Thus, no wraparound should be

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, 19 Dec 2022 at 22:40, Maxim Orlov wrote: > Hi! > > As a result of discussion in the thread [0], Robert Haas proposed to focus > on making SLRU 64 bit, as a first step towards 64 bit XIDs. > Here is the patch set. > > In overall, code of this patch set is based on the existing code from [

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Dec 19, 2022 at 6:41 AM Maxim Orlov wrote: > > As always, reviews and opinions are very welcome. Hi! I think that 64-bit xids are a very important feature and I want to help advance it. That's why I want to try to understand a patch better. Do I get it right that the proposed v51 patchse

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi! As a result of discussion in the thread [0], Robert Haas proposed to focus on making SLRU 64 bit, as a first step towards 64 bit XIDs. Here is the patch set. In overall, code of this patch set is based on the existing code from [0] and may be simplified, due to the fact, that SLRU_PAGES_PER_S

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Wed, Dec 7, 2022 at 9:50 AM Andres Freund wrote: > performing post-bootstrap initialization ... > ../src/backend/access/transam/slru.c:1520:9: runtime error: load of > misaligned address 0x7fff6362db8c for type 'int64', which requires 8 byte > alignment > 0x7fff6362db8c: note: pointer points

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, On 2022-12-07 11:40:08 +0300, Aleksander Alekseev wrote: > Hi Michael, > > > The CF bot is showing some failures here. You may want to > > double-check. > > Thanks! PFA v48. This causes a lot of failures with ubsan: https://cirrus-ci.com/task/6035600772431872 performing post-bootstrap init

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Mon, Nov 21, 2022 at 12:21:09PM +0300, Aleksander Alekseev wrote: > After merging 1489b1ce [1] the patchset needed a rebase. PFA v47. > > [1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=1489b1ce The CF bot is showing some failures here. You may want to double-check. -- Mic

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022年11月3日(木) 17:15 Maxim Orlov : > > Hi! > >> >> This entry was marked "Ready for committer" in the CommitFest app but cfbot >> reports the patch no longer applies. > > Thanks for the reminder. I think, this should work. Thanks for the quick update, cfbot reports no issues. I've set the CF entry

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Thu, Nov 3, 2022 at 1:46 PM Maxim Orlov wrote: > > Hi! > >> >> This entry was marked "Ready for committer" in the CommitFest app but cfbot >> reports the patch no longer applies. > > Thanks for the reminder. I think, this should work. Have we measured the WAL overhead because of this patch set

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022年10月10日(月) 18:16 Aleksander Alekseev : > > Hi hackers, > > > Sorry about that. > > No problem. > > > Thanks, Simon. Not 100% sure who exactly is invited to vote, but just > > in case here is +1 from me. These patches were "Ready for Committer" > > for several months now and are unlikely to get

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

> > This is the wrong thread / CF entry. Please see > Yep, my fault. Sorry about that. -- Best regards, Maxim Orlov.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Maxim, > Here is a rebased version of the patch set. This is the wrong thread / CF entry. Please see http://cfbot.cputube.org/ and https://commitfest.postgresql.org/ and the first email in the thread. -- Best regards, Aleksander Alekseev

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Tue, 26 Jul 2022 at 21:35, Simon Riggs wrote: > On Fri, 15 Jul 2022 at 12:29, Aleksander Alekseev > wrote: > > Personally I didn't expect that > > merging patches in this thread would take that long. They are in > > "Ready for Committer" state for a long time now and there are no known > > is

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Fri, 15 Jul 2022 at 12:29, Aleksander Alekseev wrote: > Personally I didn't expect that > merging patches in this thread would take that long. They are in > "Ready for Committer" state for a long time now and there are no known > issues with them other than unit tests for SLRU [1] should be mer

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Pavel, Justin, >> Is there any reason to continue with two separate threads and CF entries ? >> The original reason was to have a smaller patch for considerate late in v15. > I see the main goal of this split is to make discussion of this (easier) > thread separate to the discussion of a whole p

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

> > Is there any reason to continue with two separate threads and CF entries ? > The original reason was to have a smaller patch for considerate late in > v15. > > But right now, it just seems to cause every update to imply two email > messages > rather than one. > > Since the patch is split into 0

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On Fri, May 13, 2022 at 04:21:29PM +0300, Maxim Orlov wrote: > We have posted an updated version v34 of the whole patchset in [1]. > Changes of patches 0001-0003 there are identical to v33. So, no update is > needed in this thread. Is there any reason to continue with two separate threads and CF e

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

> here is the rebased v32 version of the patch. > > The patchset rotted a bit. Here is a rebased version. > We have posted an updated version v34 of the whole patchset in [1]. Changes of patches 0001-0003 there are identical to v33. So, no update is needed in this thread. [1] https://www.postgres

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi Peter, > That isn't really what I meant. When I said > > > At this point, I'm more concerned that code review is still finding > > bugs, and that we have no test coverage for any of this > > this meant especially the SLRU refactoring patch. Got it, and sorry for the confusion. I decided to

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 29.03.22 15:09, Aleksander Alekseev wrote: Hi hackers, I took another look at the patchset. Personally I don't think it will get much better than it is now. I'm inclined to change the status of the CF entry to "Ready for Committer" unless anyone disagrees. About v25-0002-Use-64-bit-format

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi hackers, > I took another look at the patchset. Personally I don't think it will get much > better than it is now. I'm inclined to change the status of the CF entry to > "Ready for Committer" unless anyone disagrees. > > About v25-0002-Use-64-bit-format-to-output-XIDs.patch: > > I don't see th

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 28.03.22 12:46, Aleksander Alekseev wrote: Personally I don't have a strong opinion here. Merging the patch sooner will allow us to move toward 64-bit XIDs faster (e.g. to gather the feedback from the early adopters, allow the translators to do their thing earlier, etc). Merging it later will

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi hackers, > Here is v30. I took another look at the patchset. Personally I don't think it will get much better than it is now. I'm inclined to change the status of the CF entry to "Ready for Committer" unless anyone disagrees. cfbot reports a problem with t/013_partition.pl but the test seems

Re: XID formatting and SLRU refactorings

At Fri, 25 Mar 2022 00:02:55 +0400, Pavel Borisov wrote in > Hi! > It seems that CFbot was still unhappy with pg_upgrade test due to epoch > removal from NextXid in controldata. > I've reverted this change as support for "epochless" 64-bit control data > with xids that haven't yet switched to 64

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Just forgot to mention that answers in a previous message are describing the changes that are in v26. -- Best regards, Pavel Borisov Postgres Professional: http://postgrespro.com

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

Hi, Peter! Thanks for your review! About v25-0001-Use-unsigned-64-bit-numbering-of-SLRU-pages.patch: > > -static bool CLOGPagePrecedes(int page1, int page2); > +static bool CLOGPagePrecedes(uint64 page1, uint64 page2); > > You are changing the API from signed to unsigned. Is this intentional? > I

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

On 23.03.22 10:51, Maxim Orlov wrote: Thanks for updating the patchs. I have a little comment on 0002. +                                errmsg_internal("found xmax %llu" " (infomask 0x%04x) not frozen, not multi, not normal", +

  1   2   >