Re: Add 64-bit XIDs into PostgreSQL 15

2024-12-17 Thread wenhui qiu
Hi Evgeny > we are causing the target solution to become staled I think the community is not positive enough about xid64,There is also a lot of resistance, such as the wal log becomes larger >I also hope you and the community will support attempts to retain the > whole xid64 solution in the con

Re: Add 64-bit XIDs into PostgreSQL 15

2024-12-17 Thread Evgeny Voropaev
Hi Evgeny      xid64 path split several threads ,The current one should be this:(https://www.postgresql.org/message-id/flat/CACG=ezawg7_nt-8ey4akv2w9lculthhknwcawmbgeetnjrj...@mail.gmail.com) ,We can do some tests on path so that  can merge earlier Thanks Hello Wenhui! Thank you for pointing

Re: Add 64-bit XIDs into PostgreSQL 15

2024-12-16 Thread wenhui qiu
Hi Evgeny xid64 path split several threads ,The current one should be this:( https://www.postgresql.org/message-id/flat/CACG=ezawg7_nt-8ey4akv2w9lculthhknwcawmbgeetnjrj...@mail.gmail.com) ,We can do some tests on path so that can merge earlier Thanks On Tue, Dec 10, 2024 at 7:47 PM Evgeny

Re: Add 64-bit XIDs into PostgreSQL 15

2024-09-12 Thread Aleksander Alekseev
Hi, > Agree. > > To me it seems like we didn't reach a consensus regarding switching to > 64-bit XIDs. Given that and the fact that the patchset is rather > difficult to rebase (and review) I suggest focusing on something we > reached a consensus for. I'm going to close a CF entry for this > parti

Re: Add 64-bit XIDs into PostgreSQL 15

2024-09-12 Thread Aleksander Alekseev
Michael, Maxim, > Apparently, the original thread will inevitably disintegrate into many > separate ones. > For me, looks like some kind of road map. One for 64-bit GUCs, another one to > remove > short file names from SLRUs and, to make things more complicated, [1] for cf > entry [0], > to get

Re: Add 64-bit XIDs into PostgreSQL 15

2024-09-12 Thread Maxim Orlov
Apparently, the original thread will inevitably disintegrate into many separate ones. For me, looks like some kind of road map. One for 64-bit GUCs, another one to remove short file names from SLRUs and, to make things more complicated, [1] for cf entry [0], to get rid of MultiXactOffset wraparound

Re: Add 64-bit XIDs into PostgreSQL 15

2024-09-11 Thread Michael Paquier
On Thu, Jul 25, 2024 at 02:09:10PM +0300, Heikki Linnakangas wrote: > On 25/07/2024 13:19, Peter Eisentraut wrote: >> Conversely, if there is still some threshold (not disaster, but >> efficiency or something else), would it still be useful to keep these >> settings well below 2^31?  In which case,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-08-18 Thread Michael Paquier
On Sat, Aug 10, 2024 at 10:50:55AM -0700, Noah Misch wrote: > On Sat, Jul 27, 2024 at 07:24:33AM +0900, Michael Paquier wrote: >> I've just applied now the remaining pieces down to 17. > > Comparing commit c9e2457 to the patch in zqgvzsbw5tgkq...@paquier.xyz, the > commit lacks the slru.c portion.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-08-10 Thread Noah Misch
On Sat, Jul 27, 2024 at 07:24:33AM +0900, Michael Paquier wrote: > I've just applied now the remaining pieces down to 17. Comparing commit c9e2457 to the patch in zqgvzsbw5tgkq...@paquier.xyz, the commit lacks the slru.c portion.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-26 Thread Michael Paquier
On Fri, Jul 26, 2024 at 11:50:48PM +0300, Alexander Korotkov wrote: > Thanks to everybody for working on this. It's pity I didn't notice > this is v17 open item on me. Sorry for this. No problem. I've just applied now the remaining pieces down to 17. -- Michael signature.asc Description: PGP

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-26 Thread Alexander Korotkov
On Fri, Jul 26, 2024 at 3:42 AM Noah Misch wrote: > On Thu, Jul 25, 2024 at 10:52:13AM +0900, Michael Paquier wrote: > > On Wed, Jul 24, 2024 at 06:00:59AM -0700, Noah Misch wrote: > > > I'm still seeing need for s/int/int64/ at: > > > I am attaching a patch for all these you have spotted, switchi

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-25 Thread Noah Misch
On Thu, Jul 25, 2024 at 10:52:13AM +0900, Michael Paquier wrote: > On Wed, Jul 24, 2024 at 06:00:59AM -0700, Noah Misch wrote: > > I'm still seeing need for s/int/int64/ at: > I am attaching a patch for all these you have spotted, switching the > logs to use %llx. Does that look fine for you? Ye

Re: Add 64-bit XIDs into PostgreSQL 15

2024-07-25 Thread wenhui qiu
Maybe a setting similar to max_wal_size could be better for that? +1 Thanks Peter Eisentraut 于2024年7月25日周四 21:31写道: > On 25.07.24 13:09, Heikki Linnakangas wrote: > >> However, if there is no more disaster threshold at 2^31, what is the > >> guidance for setting these? Or more radically, why e

Re: Add 64-bit XIDs into PostgreSQL 15

2024-07-25 Thread Peter Eisentraut
On 25.07.24 13:09, Heikki Linnakangas wrote: However, if there is no more disaster threshold at 2^31, what is the guidance for setting these?  Or more radically, why even run transaction-count-based vacuum at all? To allow the CLOG to be truncated. There's no disaster anymore, but without fre

Re: Add 64-bit XIDs into PostgreSQL 15

2024-07-25 Thread Chris Travers
On Thu, Jul 25, 2024 at 5:19 PM Peter Eisentraut wrote: > On 23.07.24 11:13, Aleksander Alekseev wrote: > >> Here is the fix. It can be tested like this: > >> [...] > > > > PFA the rebased patchset. > > I'm wondering about the 64-bit GUCs. > > At first, it makes sense that if there are settings t

Re: Add 64-bit XIDs into PostgreSQL 15

2024-07-25 Thread Heikki Linnakangas
On 25/07/2024 13:19, Peter Eisentraut wrote: I'm wondering about the 64-bit GUCs. At first, it makes sense that if there are settings that are counted in terms of transactions, and transaction numbers are 64-bit integers, then those settings should accept 64-bit integers. But what is the pur

Re: Add 64-bit XIDs into PostgreSQL 15

2024-07-25 Thread Peter Eisentraut
On 23.07.24 11:13, Aleksander Alekseev wrote: Here is the fix. It can be tested like this: [...] PFA the rebased patchset. I'm wondering about the 64-bit GUCs. At first, it makes sense that if there are settings that are counted in terms of transactions, and transaction numbers are 64-bit i

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-24 Thread Michael Paquier
On Wed, Jul 24, 2024 at 06:00:59AM -0700, Noah Misch wrote: > I'm still seeing need for s/int/int64/ at: Nice catches! I've missed these. > - "pagesegno" variable > - return value of MultiXactIdToOffsetSegment() Only used in four places for two elog(DEBUG1) entries with %x. > - return value of

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-24 Thread Noah Misch
On Tue, Jul 23, 2024 at 06:01:44PM +0900, Michael Paquier wrote: > Hearing nothing but cicadas as now is their season, I have taken the > initiative here to address this open item. > > 0001 felt a bit too complicated in slru.h, so I have simplified it and > kept all the details in slru.c with Slru

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-23 Thread Aleksander Alekseev
Hi, > Hearing nothing but cicadas as now is their season, I have taken the > initiative here to address this open item. > > 0001 felt a bit too complicated in slru.h, so I have simplified it and > kept all the details in slru.c with SlruFileName(). > > I have reviewed all the code that uses SLRUs,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-23 Thread Michael Paquier
On Fri, Jul 12, 2024 at 12:44:54PM +0300, Aleksander Alekseev wrote: > Fair enough. Here is the updated patchset. Hearing nothing but cicadas as now is their season, I have taken the initiative here to address this open item. 0001 felt a bit too complicated in slru.h, so I have simplified it and

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-12 Thread Aleksander Alekseev
Hi, > A comment about v3-0001. > > -* If true, use long segment filenames formed from lower 48 bits of > the > -* segment number, e.g. pg_xact/1234. Otherwise, use short > -* filenames formed from lower 16 bits of the segment number e.g. > -* pg_xact/1234.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-11 Thread Michael Paquier
On Thu, Jul 11, 2024 at 01:11:05PM +0300, Aleksander Alekseev wrote: > Thanks, Michael. I prepared a corrected patchset. A comment about v3-0001. -* If true, use long segment filenames formed from lower 48 bits of the -* segment number, e.g. pg_xact/1234. Otherwise, use sh

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-11 Thread Aleksander Alekseev
Hi, > The proposed patch looks rather incomplete to me, based on the fact > that this stuff has a lot of inconsistencies with the types used when > manipulating 64b SLRU pages. Some of them are harder to catch as the > variables don't specifically refer to pages. > > So, even after v2, there are

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-08 Thread Michael Paquier
On Mon, Jul 08, 2024 at 12:30:09PM +0300, Aleksander Alekseev wrote: > TWIMC this is currently listed as an open item for PG17 [1]. > Sorry if everyone interested is already aware. > > [1]: https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items The proposed patch looks rather incomplete to me,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-07-08 Thread Aleksander Alekseev
Hi, > Here is the corrected patchset. TWIMC this is currently listed as an open item for PG17 [1]. Sorry if everyone interested is already aware. [1]: https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items -- Best regards, Aleksander Alekseev

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-06-27 Thread Aleksander Alekseev
Hi, > > I prepared the patch for clog.c. The rest of the warnings don't strike > > me as something we should immediately act on unless we have a bug > > report. Or perhaps there is a particular warning that worries you? > > Is "int" acceptable or unacceptable in the following grep match? > > src/b

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-06-26 Thread Noah Misch
On Wed, Jun 26, 2024 at 02:09:58PM +0300, Aleksander Alekseev wrote: > > This yielded commit 4ed8f09 "Index SLRUs by 64-bit integers rather than by > > 32-bit integers" and left some expressions coercing SLRU page numbers to > > int. > > Two sources: > > > > grep -i 'int\b.*page' $(git grep -l Si

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-06-26 Thread Aleksander Alekseev
Hi Noah, > This yielded commit 4ed8f09 "Index SLRUs by 64-bit integers rather than by > 32-bit integers" and left some expressions coercing SLRU page numbers to int. > Two sources: > > grep -i 'int\b.*page' $(git grep -l SimpleLruInit) > make && touch $(git grep -l SimpleLruInit) && make PROFILE

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2024-06-25 Thread Noah Misch
On Mon, Nov 27, 2023 at 01:43:26AM +0200, Alexander Korotkov wrote: > v61 looks good to me. I'm going to push it as long as there are no > objections. This yielded commit 4ed8f09 "Index SLRUs by 64-bit integers rather than by 32-bit integers" and left some expressions coercing SLRU page numbers

Re: Add 64-bit XIDs into PostgreSQL 15

2024-01-18 Thread Shubham Khanna
On Wed, Dec 13, 2023 at 5:56 PM Maxim Orlov wrote: > > Hi! > > Just to keep this thread up to date, here's a new version after recent > changes in SLRU. > I'm also change order of the patches in the set, to make adding initdb MOX > options after the > "core 64 xid" patch, since MOX patch is unli

Re: Add 64-bit XIDs into PostgreSQL 15

2023-12-15 Thread wenhui qiu
Hi Pavel Borisov Many thanks Best whish Pavel Borisov 于2023年12月15日周五 17:13写道: > Hi, Wenhui! > > On Fri, 15 Dec 2023 at 05:52, wenhui qiu wrote: > >> Hi Maxim Orlov >> Good news,xid64 has achieved a successful first phase,I tried to >> change the path status (https://commitfest.postgres

Re: Add 64-bit XIDs into PostgreSQL 15

2023-12-15 Thread Pavel Borisov
Hi, Wenhui! On Fri, 15 Dec 2023 at 05:52, wenhui qiu wrote: > Hi Maxim Orlov > Good news,xid64 has achieved a successful first phase,I tried to > change the path status (https://commitfest.postgresql.org/43/3594/) ,But > it seems incorrect > > Maxim Orlov 于2023年12月13日周三 20:26写道: > >> Hi! >>

Re: Add 64-bit XIDs into PostgreSQL 15

2023-12-14 Thread wenhui qiu
Hi Maxim Orlov Good news,xid64 has achieved a successful first phase,I tried to change the path status (https://commitfest.postgresql.org/43/3594/) ,But it seems incorrect Maxim Orlov 于2023年12月13日周三 20:26写道: > Hi! > > Just to keep this thread up to date, here's a new version after recent > c

回复: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-12-04 Thread Thomas wen
akhin ; Maxim Orlov ; Aleksander Alekseev ; Postgres hackers ; Heikki Linnakangas ; Japin Li ; Andres Freund ; Michael Paquier ; Peter Eisentraut 主题: Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15) On Mon, Dec 4, 2023 at 3:12 PM Pavel Borisov wrote: > I think

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-12-04 Thread John Naylor
On Mon, Dec 4, 2023 at 3:12 PM Pavel Borisov wrote: > I think this is complete and could be closed. Done.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-12-04 Thread Pavel Borisov
On Mon, 4 Dec 2023 at 10:34, John Naylor wrote: > On Thu, Nov 30, 2023 at 4:37 PM Alexander Korotkov > wrote: > > > > On Thu, Nov 30, 2023 at 10:29 AM Pavel Borisov > wrote: > > > Agree. The fix is attached. > > > > What an oversight. > > Thank you, pushed! > > With that, is there any more work

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-12-03 Thread John Naylor
On Thu, Nov 30, 2023 at 4:37 PM Alexander Korotkov wrote: > > On Thu, Nov 30, 2023 at 10:29 AM Pavel Borisov wrote: > > Agree. The fix is attached. > > What an oversight. > Thank you, pushed! With that, is there any more work pending, or can we close the CF entry?

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-30 Thread Alexander Korotkov
On Thu, Nov 30, 2023 at 10:29 AM Pavel Borisov wrote: > Agree. The fix is attached. What an oversight. Thank you, pushed! -- Regards, Alexander Korotkov

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-30 Thread Pavel Borisov
On Thu, 30 Nov 2023 at 08:03, Tom Lane wrote: > Alexander Lakhin writes: > > And a warning: > > $ CC=gcc-12 CFLAGS="-Wall -Wextra -Wno-unused-parameter > -Wno-sign-compare -Wno-clobbered > > -Wno-missing-field-initializers" ./configure -q && make -s > > slru.c:63:1: warning: ‘inline’ is not at b

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-29 Thread Tom Lane
Alexander Lakhin writes: > And a warning: > $ CC=gcc-12 CFLAGS="-Wall -Wextra -Wno-unused-parameter -Wno-sign-compare > -Wno-clobbered > -Wno-missing-field-initializers" ./configure -q && make -s > slru.c:63:1: warning: ‘inline’ is not at beginning of declaration > [-Wold-style-declaration] >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-28 Thread Alexander Korotkov
Андрей, привет! Текущее положение у меня такое. . *pg_stats and range statisticsTracking statements entry timestamp in pg_stat_statements* Уже закоммичены. *XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)* Если всё будет и замечаний не возникнет ок, завтра утром

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-26 Thread Alexander Lakhin
Hello Alexander, 27.11.2023 02:43, Alexander Korotkov wrote: v61 looks good to me. I'm going to push it as long as there are no objections. I've looked at the patch set and found a typo: occured And a warning: $ CC=gcc-12 CFLAGS="-Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-c

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-26 Thread Alexander Korotkov
Alexander, Maxim, Thank you for revisions. On Thu, Nov 9, 2023 at 6:22 PM Maxim Orlov wrote: > Aleksander Alekseev, > > > Maxim, > > I see both of us accounted for Alexanders feedback and submitted v59. > > Your newer version seems to have issues on cfbot, so resubmitting the > > previous patchs

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-09 Thread Maxim Orlov
Aleksander Alekseev, > Maxim, > I see both of us accounted for Alexanders feedback and submitted v59. > Your newer version seems to have issues on cfbot, so resubmitting the > previous patchset that passes the tests. Please feel free to add > changes. For unknown reasons, I do not receive any of

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-08 Thread Aleksander Alekseev
Maxim, I see both of us accounted for Alexanders feedback and submitted v59. Your newer version seems to have issues on cfbot, so resubmitting the previous patchset that passes the tests. Please feel free to add changes. > See SlruCorrectSegmentFilenameLength(): > > ``` > if (ctl->long_segmen

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-07 Thread Maxim Orlov
On Mon, 6 Nov 2023 at 16:07, Alexander Korotkov wrote: > Hi! > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev < > aleksan...@timescale.com> wrote: > > PFE the corrected patchset v58. > > I'd like to revive this thread. > Hi! Great news! > BTW, there is a typo in a word "exceeed". > Fixed.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-07 Thread Aleksander Alekseev
Hi again, > PFA the corrected patchset v59. On second thought, I believe this Assert is incorrect: ``` +else +{ +Assert(segno >= 0 && segno <= 0x); +return snprintf(path, MAXPGPATH, "%s/%04X", (ctl)->Dir, +(unsigned int) segno); +} ``` See

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-07 Thread Aleksander Alekseev
Hi Alexander, > -#define TransactionIdToCTsPage(xid) \ > - ((xid) / (TransactionId) COMMIT_TS_XACTS_PER_PAGE) > + > +/* > + * Although we return an int64 the actual value can't currently exceeed > 2**32. > + */ > +static inline int64 > +TransactionIdToCTsPage(TransactionId xid) > +{ > + retur

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Alexander Korotkov
On Mon, Nov 6, 2023 at 4:38 PM Aleksander Alekseev wrote: > > > PFE the corrected patchset v58. > > > > I'd like to revive this thread. > > Many thanks for your comments and suggestions. > > > I think it worth adding asserts here to verify there is no overflow making us mapping different segments

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Aleksander Alekseev
Alexander, > > PFE the corrected patchset v58. > > I'd like to revive this thread. Many thanks for your comments and suggestions. > I think it worth adding asserts here to verify there is no overflow making us > mapping different segments into the same files. Sorry, I didn't understand this on

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Aleksander Alekseev
Hi, > > > If I remember right, the compiler will make equivalent code from > > > inline functions and macros, and functions has an additional benefit: > > > the compiler will report type mismatch if any. That was the only > > > reason. > > > > Then it's OK to leave it as an inline function. +1 >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Pavel Borisov
On Mon, 6 Nov 2023 at 18:01, Alexander Korotkov wrote: > > On Mon, Nov 6, 2023 at 3:42 PM Pavel Borisov wrote: > > > > On Mon, 6 Nov 2023 at 17:07, Alexander Korotkov > > wrote: > > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev > > > wrote: > > > > PFE the corrected patchset v58. > > >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Alexander Korotkov
On Mon, Nov 6, 2023 at 3:42 PM Pavel Borisov wrote: > > On Mon, 6 Nov 2023 at 17:07, Alexander Korotkov wrote: > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev > > wrote: > > > PFE the corrected patchset v58. > > > > I'd like to revive this thread. > > > > This patchset is extracted from

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Pavel Borisov
On Mon, 6 Nov 2023 at 17:07, Alexander Korotkov wrote: > > Hi! > > On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev > wrote: > > PFE the corrected patchset v58. > > I'd like to revive this thread. > > This patchset is extracted from a larger patchset implementing 64-bit xids. > It converts p

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-11-06 Thread Alexander Korotkov
Hi! On Wed, Jul 5, 2023 at 4:46 PM Aleksander Alekseev wrote: > PFE the corrected patchset v58. I'd like to revive this thread. This patchset is extracted from a larger patchset implementing 64-bit xids. It converts page numbers in SLRUs into 64 bits. The most SLRUs save the same file naming

回复: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-10-03 Thread Thomas wen
ood idea 发件人: Aleksander Alekseev 发送时间: 2023年9月4日 22:41 收件人: Postgres hackers 抄送: Heikki Linnakangas ; Maxim Orlov ; Jacob Champion ; Japin Li ; Andres Freund ; Michael Paquier ; Pavel Borisov ; Peter Eisentraut ; Alexander Korotkov 主题: Re: XID formatting and SLRU r

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-09-04 Thread Aleksander Alekseev
Hi, > > Reviewing this now. I think it's almost ready to be committed. > > > > There's another big effort going on to move SLRUs to the regular buffer > > cache (https://commitfest.postgresql.org/43/3514/). I wonder how moving > > to 64 bit page numbers will affect that. BlockNumber is still 32 bi

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-07-05 Thread Aleksander Alekseev
Hi, > Reviewing this now. I think it's almost ready to be committed. > > There's another big effort going on to move SLRUs to the regular buffer > cache (https://commitfest.postgresql.org/43/3514/). I wonder how moving > to 64 bit page numbers will affect that. BlockNumber is still 32 bits, > afte

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-07-05 Thread Heikki Linnakangas
On 09/03/2023 17:21, Aleksander Alekseev wrote: v57-0001-Index-SLRUs-by-64-bit-integers-rather-than-by-32.patch Reviewing this now. I think it's almost ready to be committed. There's another big effort going on to move SLRUs to the regular buffer cache (https://commitfest.postgresql.org/43/35

Re: Add 64-bit XIDs into PostgreSQL 15

2023-07-04 Thread Aleksander Alekseev
Hi, > This patch hasn't applied in quite some time, and the thread has moved to > discussing higher lever items rather than the suggested patch, so I'm closing > this as Returned with Feedback. Please feel free to resubmit when there is > renewed interest and a concensus on how/what to proceed wi

Re: Add 64-bit XIDs into PostgreSQL 15

2023-07-04 Thread Daniel Gustafsson
This patch hasn't applied in quite some time, and the thread has moved to discussing higher lever items rather than the suggested patch, so I'm closing this as Returned with Feedback. Please feel free to resubmit when there is renewed interest and a concensus on how/what to proceed with. -- Danie

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-29 Thread Aleksander Alekseev
Hi, > As suggested before by Heikki Linnakangas, I've added a patch for making 2PC > transaction state 64-bit. > At first, my intention was to rebuild all twophase interface to use > FullTransactionId. But doing this in a proper > manner would lead to switching from TransactionId to FullTransact

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-20 Thread Maxim Orlov
Hi! As suggested before by Heikki Linnakangas, I've added a patch for making 2PC transaction state 64-bit. At first, my intention was to rebuild all twophase interface to use FullTransactionId. But doing this in a proper manner would lead to switching from TransactionId to FullTransactionId in PGP

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-09 Thread Aleksander Alekseev
Hi, > 1. Use internal 64 bit page numbers in SLRUs without changing segments naming. > 2. Use the larger segment file names in async.c, to lift the current 8 > GB limit on the max number of pending notifications. > [...] > > Where our main focus for PG16 is going to be 1 and 2, and we may try > to

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-07 Thread Maxim Orlov
On Tue, 7 Mar 2023 at 15:38, Heikki Linnakangas wrote: > > That is true for pg_multixact/offsets. We will indeed need to add an > epoch and introduce the concept of FullMultiXactIds for that. However, > we can change pg_multixact/members independently of that. We can extend > MultiXactOffset from

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-07 Thread Aleksander Alekseev
Hi, > Here's how I see the development path for this [...] > So, in my view, the plan should be [...] > Thoughts? The plan looks great! I would also explicitly include this: > As we start to refactor these things, I also think it would be good to > have more explicit tracking of the valid range

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-07 Thread Heikki Linnakangas
On 07/03/2023 13:38, Maxim Orlov wrote: As for making pg_multixact 64 bit, I spend the last couple of days to make proper pg_upgrade for pg_multixact's and for pg_xact's with wraparound and I've understood, that it is not a simple task compare to pg_xact's. The problem is, we do not have epoch f

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-07 Thread Maxim Orlov
On Tue, 17 Jan 2023 at 16:33, Aleksander Alekseev wrote: > Hi hackers, > > Maxim, perhaps you could share with us what your reasoning was here? > > I'm really sorry for late response, but better late than never. Yes, we can not access shared memory without lock. In this particular case, we use Xi

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-06 Thread Heikki Linnakangas
On 01/03/2023 12:21, Aleksander Alekseev wrote: Hi, I'm surprised that these patches extend the page numbering to 64 bits, but never actually uses the high bits. The XID "epoch" is not used, and pg_xact still wraps around and the segment names are still reused. I thought we could stop doing tha

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-03-01 Thread Aleksander Alekseev
Hi, > I'm surprised that these patches extend the page numbering to 64 bits, > but never actually uses the high bits. The XID "epoch" is not used, and > pg_xact still wraps around and the segment names are still reused. I > thought we could stop doing that. To clarify, the idea is to let CLOG gro

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-28 Thread Heikki Linnakangas
On 28/02/2023 16:17, Maxim Orlov wrote: Either I do not understand something, or the files from pg_commit_ts directory are not copied. Huh, yeah you're right, pg_upgrade doesn't copy pg_commit_ts. - Heikki

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-28 Thread Maxim Orlov
On Mon, 27 Feb 2023 at 12:10, Heikki Linnakangas wrote: > (CC list trimmed, gmail wouldn't let me send otherwise) > > That sounds right for pg_serial, pg_notify, and pg_subtrans. But not for > pg_commit_ts and the pg_multixacts. > > This needs tests for pg_upgrading those SLRUs, after 0, 1 and N

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-27 Thread Heikki Linnakangas
(CC list trimmed, gmail wouldn't let me send otherwise) On 22/02/2023 16:29, Maxim Orlov wrote: On Tue, 21 Feb 2023 at 16:59, Aleksander Alekseev mailto:aleksan...@timescale.com>> wrote: One thing that still bothers me is that during the upgrade we only migrate the CLOG segments (i.e. p

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-22 Thread Maxim Orlov
On Tue, 21 Feb 2023 at 16:59, Aleksander Alekseev wrote: > Hi, > > One thing that still bothers me is that during the upgrade we only > migrate the CLOG segments (i.e. pg_xact / pg_clog) and completely > ignore all the rest of SLRUs: > > * pg_commit_ts > * pg_multixact/offsets > * pg_multixact/me

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-21 Thread Aleksander Alekseev
Hi, > The convert_pg_xact_segments() function is still obviously > overengineered. As I understand, all it has to do is simply renaming > pg_xact/ to pg_xact/. Unfortunately I used up all the > mana for today and have to take a long rest in order to continue. PFA the corrected pat

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-20 Thread Aleksander Alekseev
Hi, > OK, here is the patchset v53 where I mostly modified the commit > messages. It is explicitly said that 0001 modifies the WAL records and > why we decided to do it in this patch. Additionally any mention of > 64-bit XIDs is removed since it is not guaranteed that the rest of the > patches are

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-17 Thread Aleksander Alekseev
Hi hackers, > Yeah, pg_upgrade will briefly start and stop the old server to make > sure all the WAL is replayed, and won't transfer any of the files > over. AFAIK, major-version WAL changes are fine; it was the previous > claim that we could do it in a minor version that I was unsure about. OK,

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-11 Thread Jacob Champion
On Wed, Jan 11, 2023 at 1:48 AM Aleksander Alekseev wrote: > After reading [1] carefully it looks like we shouldn't worry about > this. The upgrade procedure explicitly requires to run `pg_ctl stop` > during step 8 of the upgrade procedure, i.e. not in the immediate mode > [2]. Yeah, pg_upgrade w

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-11 Thread Aleksander Alekseev
Hi Maxim, > Secondly, shouldn't we introduce a new WAL record type in order to > make the code backward compatible with previous PG versions? I'm not > 100% sure how the upgrade procedure works in all the details. If it > requires the DBMS to be gracefully shut down before the upgrade then > we ar

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-11 Thread Aleksander Alekseev
Hi Maxim, > Here is a new patch set. > I've added comments and make use GetClogDirName call in copy_subdir_files. Jacob Champion pointed out (offlist, cc:'ed) that we may be wrong on this one: > 0001 patch changes the SLRU internals without affecting the callers. > 0001 - should make SLRU inter

Re: Add 64-bit XIDs into PostgreSQL 15

2023-01-10 Thread Aleksander Alekseev
Hi Maxim, > Anyway. Let's discuss on-disk page format, shall we? Here are my two cents. > AFAICS, we have a following options: > [...] > 2. Put special in every page where base for XIDs are stored. This is what we > have done in the current patch set. The approach of using special space IMO is

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Maxim Orlov
Hi! Here is a new patch set. I've added comments and make use GetClogDirName call in copy_subdir_files. -- Best regards, Maxim Orlov. v52-0003-Make-pg_upgrade-from-32-bit-to-64-bit-SLRU.patch Description: Binary data v52-0002-Use-64-bit-pages-representation-in-SLRU-callers.patch Description:

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Maxim Orlov
On Fri, 6 Jan 2023 at 09:51, Japin Li wrote: > > For v51-0003. We can use GetClogDirName instead of GET_MAJOR_VERSION in > copy_subdir_files(). > Of course! Tanks! I'll address this in the next iteration, v52. -- Best regards, Maxim Orlov.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Aleksander Alekseev
Hi Andrey, > Hi! I think that 64-bit xids are a very important feature and I want > to help advance it. That's why I want to try to understand a patch > better. Thanks for your interest to the patchset! > Do I get it right that the proposed v51 patchset only changes the SLRU > filenames and type

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Maxim Orlov
> > Do I get it right that the proposed v51 patchset only changes the SLRU > filenames and type of pageno representation? Is SLRU wraparound still > exactly there after 0x byte? > After applying the whole patch set, SLRU will become 64–bit without a wraparound. Thus, no wraparound should be

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-05 Thread Japin Li
On Mon, 19 Dec 2022 at 22:40, Maxim Orlov wrote: > Hi! > > As a result of discussion in the thread [0], Robert Haas proposed to focus > on making SLRU 64 bit, as a first step towards 64 bit XIDs. > Here is the patch set. > > In overall, code of this patch set is based on the existing code from [

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-05 Thread Andrey Borodin
On Mon, Dec 19, 2022 at 6:41 AM Maxim Orlov wrote: > > As always, reviews and opinions are very welcome. Hi! I think that 64-bit xids are a very important feature and I want to help advance it. That's why I want to try to understand a patch better. Do I get it right that the proposed v51 patchse

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-30 Thread adherent postgres
Konstantin Knizhnik ; Nikita Glukhov ; Yura Sokolov ; Simon Riggs 主题: Re: Add 64-bit XIDs into PostgreSQL 15 Hi! I want to make a quick summary here. 1. An overall consensus has been reached: we shall focus on committing SLRU changes first. 2. I've created an appropriate patch set he

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-28 Thread Maxim Orlov
Hi! I want to make a quick summary here. 1. An overall consensus has been reached: we shall focus on committing SLRU changes first. 2. I've created an appropriate patch set here [0]. 3. How [0] is waiting for a review. As always, all opinions will be welcome. 4. While discussing error/warning mes

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022-12-19 Thread Maxim Orlov
Hi! As a result of discussion in the thread [0], Robert Haas proposed to focus on making SLRU 64 bit, as a first step towards 64 bit XIDs. Here is the patch set. In overall, code of this patch set is based on the existing code from [0] and may be simplified, due to the fact, that SLRU_PAGES_PER_S

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread Maxim Orlov
On Fri, 9 Dec 2022 at 16:54, adherent postgres < adherent_postg...@hotmail.com> wrote: > Hi Aleksander Alekseev > I think the xids 32bit transformation project has been dragged on for too > long. Huawei's openGauss referenced this patch to implement xids 64bit, and > Postgrespro also implemented

回复: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread adherent postgres
opinion? Best whish 发件人: Pavel Borisov 发送时间: 2022年12月9日 22:13 收件人: adherent postgres 抄送: Aleksander Alekseev ; pgsql-hackers@lists.postgresql.org ; Chris Travers ; Chris Travers ; Bruce Momjian 主题: Re: Add 64-bit XIDs into PostgreSQL 15 Hi, Adherent! On Fri, 9

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread Pavel Borisov
Hi, Adherent! On Fri, 9 Dec 2022 at 17:54, adherent postgres wrote: > > Hi Aleksander Alekseev > I think the xids 32bit transformation project has been dragged on for too > long. Huawei's openGauss referenced this patch to implement xids 64bit, and > Postgrespro also implemented xids 64bit, wh

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread Maxim Orlov
> So I'd vote for an evolutionary approach and give my +1 for > undertaking efforts to first committing [1] to 16. > > [1]: > https://www.postgresql.org/message-id/CAFiTN-uudj2PY8GsUzFtLYFpBoq_rKegW3On_8ZHdxB1mVv3-A%40mail.gmail.com > > Kind regards, > Pavel Borisov, > Supabase. > +1 Totally suppo

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread adherent postgres
l has no reason not to implement xid 64 bit. What about your opinion? Best whish 发件人: Aleksander Alekseev 发送时间: 2022年12月9日 20:49 收件人: pgsql-hackers@lists.postgresql.org 抄送: adherent postgres ; Chris Travers ; Chris Travers ; Bruce Momjian 主题: Re: Add 64-bit

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread Pavel Borisov
Hi, Robert! > Lest we miss the forest for the trees, there is an aspect of this > patch that I find to be an extremely good idea and think we should try > to get committed even if the rest of the patch set ends up in the > rubbish bin. Specifically, there are a couple of patches in here that > have

Re: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread Aleksander Alekseev
Hi adherent, > Robertmhaas said that the project Zheap is > dead(https://twitter.com/andy_pavlo/status/1590703943176589312), which means > that we cannot use Zheap to deal with the issue of xid wraparound and dead > tuples in tables. The dead tuple issue is not a big deal because I can sti

回复: Add 64-bit XIDs into PostgreSQL 15

2022-12-09 Thread adherent postgres
:35 收件人: Robert Haas 抄送: Chris Travers ; Bruce Momjian ; Aleksander Alekseev ; pgsql-hackers@lists.postgresql.org ; Chris Travers ; Fedor Sigaev ; Alexander Korotkov ; Konstantin Knizhnik ; Nikita Glukhov ; Yura Sokolov ; Maxim Orlov ; Pavel Borisov ; Simon Riggs 主题: Re: Add 64-bit XIDs int

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022-12-07 Thread Peter Geoghegan
On Wed, Dec 7, 2022 at 9:50 AM Andres Freund wrote: > performing post-bootstrap initialization ... > ../src/backend/access/transam/slru.c:1520:9: runtime error: load of > misaligned address 0x7fff6362db8c for type 'int64', which requires 8 byte > alignment > 0x7fff6362db8c: note: pointer points

  1   2   3   >