Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-19 Thread Sébastien
Ok you mean that xid64 will remove the need for freezing... it's a way to see things. Le mar. 18 févr. 2025 à 15:57, Greg Sabino Mullane a écrit : > On Tue, Feb 18, 2025 at 9:17 AM Sébastien wrote: > >> Sorry it won't work. It just delays the problem. But still the freeze >> procedure must rewr

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-18 Thread wenhui qiu
HI > Ok you mean that xid64 will remove the need for freezing... it's a way to see things. When xid64 is implemented, there will be no need to trigger vacuum_failsafe_age,it has a long enough time to vacuum freeze, it will have less of an impact on performance,I think that problem may be due to

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-18 Thread Greg Sabino Mullane
On Tue, Feb 18, 2025 at 9:17 AM Sébastien wrote: > Sorry it won't work. It just delays the problem. But still the freeze > procedure must rewrite all pages. > Actually, a 64-bit transaction ID allows for quite a "delay" - like hundreds of millions of years at your current rate. :) (Yes, there a

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-18 Thread Sébastien
Sorry it won't work. It just delays the problem. But still the freeze procedure must rewrite all pages. Le mar. 18 févr. 2025, 04:12, wenhui qiu a écrit : > Hello Sébastien > this case can be solved by xid64,but it seems like very few people > are interested.But it seems to me that xid64 s

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-18 Thread Sébastien
Hello weuhui, It's not a problem with heavy insert table but heavy delete. Also triggering more frequent autovacuum will not help because autovacuum does not delete recently dead tuples when a large and slow vacuum freeze operation older than their delete is still running in parallel. The solution

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-17 Thread wenhui qiu
Hello Sébastien this case can be solved by xid64,but it seems like very few people are interested.But it seems to me that xid64 should be implemented as soon as possible. Thanks On Mon, Feb 17, 2025 at 9:47 PM Sébastien wrote: > Hello weuhui, > > It's not a problem with heavy insert tabl

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-17 Thread wenhui qiu
HI Sébastien You can check out the email subject:Trigger more frequent autovacuums of heavy insert tables , I think it can alleviate the problem Thanks On Sat, Feb 15, 2025 at 3:13 AM Andres Freund wrote: > Hi, > > On 2025-02-13 10:52:31 +0100, Sébastien wrote: > > Introduce an INSERT FROZ

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-14 Thread Andres Freund
Hi, On 2025-02-13 10:52:31 +0100, Sébastien wrote: > Introduce an INSERT FROZEN feature to bypass vacuum processing for > large-scale cold data imports, reducing the impact on system performance > post-import. For large imports, migrations and major version upgrades. > Business Use-case: > > When

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-14 Thread Greg Sabino Mullane
On Fri, Feb 14, 2025 at 1:41 AM Sébastien wrote: > I had huge problems on server 3 weeks after a 6 TB migration from other > DB. I think it's sad to rewrite all data twice. > You could always COPY FREEZE into a new table, then move any remaining rows, and finally rename the tables. Should be a w

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-13 Thread Sébastien
Thank you for your answer TOM. I'd like to have this option only for initial loading of huge amounts of data, where atomicity consistency is needed at all. Maybe an option on startup command just for initial import mode, would be nice. I had huge problems on server 3 weeks after a 6 TB migration f

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-13 Thread Greg Sabino Mullane
On Thu, Feb 13, 2025 at 9:45 AM Sébastien wrote: > This issue is particularly critical during database *migrations* or *version > upgrades*, where a full data reload is often necessary. Each time a major > PostgreSQL upgrade occurs, users must reimport large datasets, leading to > the same proble

Re: [Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-13 Thread Tom Lane
=?UTF-8?Q?S=C3=A9bastien?= writes: > Implementation details: >- A new INSERT FROZEN option could be introduced, similar to COPY FREEZE, >allowing direct insertion of tuples in a frozen state. >- This would likely require changes in heap storage logic to ensure >tuples are written

[Feature Request] INSERT FROZEN to Optimize Large Cold Data Imports and Migrations

2025-02-13 Thread Sébastien
One-line Summary: Introduce an INSERT FROZEN feature to bypass vacuum processing for large-scale cold data imports, reducing the impact on system performance post-import. For large imports, migrations and major version upgrades. Business Use-case: When importing massive datasets (e.g., 6-10TB) in