Re: POC: make mxidoff 64 bits

On Thu, 2 Jan 2025 at 01:12, Heikki Linnakangas wrote: > > It might be best to just refuse the upgrade if oldestOffsetKnown==false. > It's a very ancient corner case. It seems reasonable to require you to > upgrade to a newer minor version and run VACUUM before upgrading. IIRC > that sets oldestO

Re: POC: make mxidoff 64 bits

On 27/12/2024 19:09, Maxim Orlov wrote: On Wed, 18 Dec 2024 at 13:21, Heikki Linnakangas > wrote: Does the pg_upgrade code work though, if you have that buggy situation where oldestOffsetKnown == false ? ... > >               if (!TransactionIdIsVal

Re: POC: make mxidoff 64 bits

On Wed, 18 Dec 2024 at 13:21, Heikki Linnakangas wrote: > > Attached is some more cleanup on top of patch set v9, removing more dead > stuff related to wraparound. I also removed the oldestOffsetKnown > variable and related code. It was needed to deal with clusters upgraded > from buggy 9.3 and 9

Re: POC: make mxidoff 64 bits

Oops! Sorry for the noise. I've must have been overworking yesterday and messed up the working branches. v7 was a correct set and v8 don't. Here is the correction with extended Perl test. The test itself is in src/bin/pg_upgrade/t/005_offset.pl It is rather heavy and took about 45 minutes on my i5

Re: POC: make mxidoff 64 bits

Shame on me! I've sent an erroneous patch set. Version 7 is defective. Here is the proper version v8 with minor refactoring in segresize.c. Also, I rename bump cat version patch into txt in order not to break cfbot. -- Best regards, Maxim Orlov. From 73b8663093ff1c58def9a80abab142a12c993bf6 Mon

Re: POC: make mxidoff 64 bits

On Fri, 15 Nov 2024 at 14:06, Heikki Linnakangas wrote: > Hmm, so if I understand correctly, this is related to how we determine > the length of the members array, by looking at the next multixid's > offset. This is explained in GetMultiXactIdMembers: > Correct. > If we accept that, we don't ne

Re: POC: make mxidoff 64 bits

On 13/11/2024 17:44, Maxim Orlov wrote: On Tue, 12 Nov 2024 at 02:31, Heikki Linnakangas > wrote: On a different note, I'm surprised you're rewriting member segments from scratch, parsing all the individual member groups and writing them out again. There's

Re: POC: make mxidoff 64 bits

Here is the test scripts. The generate.sh script is used to generate data dir with multimple clusters in it. This script will call multixids.py in order to generate data. If you are not use system psql consider using LD_LIBRARY_PATH env to specify path to the lib directory. OLDBIN=/.../pgsql-new ./

Re: POC: make mxidoff 64 bits

On Tue, 12 Nov 2024 at 02:31, Heikki Linnakangas wrote: > The wraparound logic is still not correct. Yep, my fault. I forget to reset segment counter if wraparound is happened. Fixed. When I try to select from a table after upgrade that contains > post-wraparound multixids: > > TRAP: failed Ass

Re: POC: make mxidoff 64 bits

On 08/11/2024 20:10, Maxim Orlov wrote: Sorry for a late reply. There was a problem in upgrade with offset wraparound. Here is a fixed version. Test also added. I decide to use my old patch to set a non-standard multixacts for the old cluster, fill it with data and do pg_upgrade. The wraparou

Re: POC: make mxidoff 64 bits

HI Maxim Orlov > After a bit of thought, I've realized that to be conservative here is the way to go. >We can reuse a maximum of existing logic. I mean, we can remove offset wraparound "error logic" and reuse "warning logic". But set the threshold for "warning >logic" to a much higher value. For no

Re: POC: make mxidoff 64 bits

After a bit of thought, I've realized that to be conservative here is the way to go. We can reuse a maximum of existing logic. I mean, we can remove offset wraparound "error logic" and reuse "warning logic". But set the threshold for "warning logic" to a much higher value. For now, I choose 2^32-1

Re: POC: make mxidoff 64 bits

On Tue, 22 Oct 2024 at 12:43, Heikki Linnakangas wrote: > MultiXactMemberFreezeThreshold looks quite bogus now. Now that > MaxMultiXactOffset==2^64-1, you cannot get anywhere near the > MULTIXACT_MEMBER_SAFE_THRESHOLD and MULTIXACT_MEMBER_DANGER_THRESHOLD > values anymore. Can we just get rid of

Re: POC: make mxidoff 64 bits

On 07/09/2024 07:36, Maxim Orlov wrote: Here is v3. MultiXactMemberFreezeThreshold looks quite bogus now. Now that MaxMultiXactOffset==2^64-1, you cannot get anywhere near the MULTIXACT_MEMBER_SAFE_THRESHOLD and MULTIXACT_MEMBER_DANGER_THRESHOLD values anymore. Can we just get rid of MultiXa

Re: POC: make mxidoff 64 bits

On 2024-Sep-12, Pavel Borisov wrote: > Hi, Maxim! > > Previously we accessed offsets in shared MultiXactState without locks as > 32-bit read is always atomic. But I'm not sure it's so when offset become > 64-bit. > E.g. GetNewMultiXactId(): > > nextOffset = MultiXactState->nextOffset; > is outsi

Re: POC: make mxidoff 64 bits

On Thu, 12 Sept 2024 at 16:09, Pavel Borisov wrote: > Hi, Maxim! > > Previously we accessed offsets in shared MultiXactState without locks as > 32-bit read is always atomic. But I'm not sure it's so when offset become > 64-bit. > E.g. GetNewMultiXactId(): > > nextOffset = MultiXactState->nextOffs

Re: POC: make mxidoff 64 bits

Hi, Maxim! Previously we accessed offsets in shared MultiXactState without locks as 32-bit read is always atomic. But I'm not sure it's so when offset become 64-bit. E.g. GetNewMultiXactId(): nextOffset = MultiXactState->nextOffset; is outside lock. There might be other places we do the same as

Re: POC: make mxidoff 64 bits

Here is v3. I removed CATALOG_VERSION_NO change, so this should be done by the actual commiter. -- Best regards, Maxim Orlov. v3-0002-Use-64-bit-multixact-offsets.patch Description: Binary data v3-0001-Use-64-bit-format-output-for-multixact-offsets.patch Description: Binary data v3-0003-Mak

Re: POC: make mxidoff 64 bits

On Tue, 3 Sept 2024 at 16:32, Alexander Korotkov wrote: > I don't think you need to maintain CATALOG_VERSION_NO change in your > patch for the exact reason you have mentioned: patch will get conflict > each time CATALOG_VERSION_NO is advanced. It's responsibility of > committer to advance CATALO

Re: POC: make mxidoff 64 bits

On Tue, Sep 3, 2024 at 4:30 PM Maxim Orlov wrote: > Here is rebase. Apparently I'll have to do it often, since the > CATALOG_VERSION_NO changed in the patch. I don't think you need to maintain CATALOG_VERSION_NO change in your patch for the exact reason you have mentioned: patch will get confli

Re: POC: make mxidoff 64 bits

Here is rebase. Apparently I'll have to do it often, since the CATALOG_VERSION_NO changed in the patch. -- Best regards, Maxim Orlov. v2-0001-Use-64-bit-format-output-for-multixact-offsets.patch Description: Binary data v2-0003-Make-pg_upgrade-convert-multixact-offsets.patch Description: Bin

Re: POC: make mxidoff 64 bits

Hi! Sorry for delay. I was a bit busy last month. Anyway, here is my proposal for making multioffsets 64 bit. The patch set consists of three parts: 0001 - making user output of offsets 64-bit ready; 0002 - making offsets 64-bit; 0003 - provide 32 to 64 bit conversion in pg_upgarde. I'm pretty su

Re: POC: make mxidoff 64 bits

On Tue, 23 Apr 2024 at 12:37, Heikki Linnakangas wrote: > This is really a bug fix. It didn't matter when TransactionId and > MultiXactOffset were both typedefs of uint32, but it was always wrong. > The argument name 'xid' is also misleading. > > I think there are some more like that, MXOffsetToF

Re: POC: make mxidoff 64 bits

Hi Maxim Orlov Thank you so much for your tireless work on this. Increasing the WAL size by a few bytes should have very little impact with today's disk performance(Logical replication of this feature wal log is also increased a lot, logical replication is a milestone new feature, and the commun

Re: POC: make mxidoff 64 bits

> On 23 Apr 2024, at 11:23, Maxim Orlov wrote: > > Make multixact offsets 64 bit. - ereport(ERROR, - (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), -errmsg("multixact \"members\" limit exceeded"), Personally, I'd be happy

Re: POC: make mxidoff 64 bits

On 23/04/2024 11:23, Maxim Orlov wrote: PROPOSAL Make multixact offsets 64 bit. +1, this is a good next step and useful regardless of 64-bit XIDs. @@ -156,7 +148,7 @@ ((uint32) ((0x % MULTIXACT_MEMBERS_PER_PAGE) + 1)) /* page in which a member is to be found */ -#d

POC: make mxidoff 64 bits

Hi! I've been trying to introduce 64-bit transaction identifications to Postgres for quite a while [0]. All this implies, of course, an enormous amount of change that will have to take place in various modules. Consider this, the patch set become too big to be committed “at once”. The obvious sol