Re: Faster data load

2024-09-05 Thread Ron Johnson
On Fri, Sep 6, 2024 at 12:43 AM Lok P wrote: > Also during index creation we are trying to do it multiple partitions at > same time from multiple sessions.But seeing out of memory error in 5th or > 6th session. > Had that same problem during pg_restore. Reduced maintenance_work_mem and the prob

Re: Faster data load

2024-09-05 Thread Lok P
On Fri, 6 Sept, 2024, 9:20 am Muhammad Usman Khan, wrote: > Hi, > > You can use pg_partman. If your table is partitioned, you can manage > partitions in parallel by distributing the load across partitions > concurrently. Or you can use citus. It can be an excellent solution, > especially for hand

Re: Faster data load

2024-09-05 Thread Muhammad Usman Khan
Hi, You can use pg_partman. If your table is partitioned, you can manage partitions in parallel by distributing the load across partitions concurrently. Or you can use citus. It can be an excellent solution, especially for handling large data volumes and parallelizing data operations On Fri, 6 S

Re: Faster data load

2024-09-05 Thread Jeff Ross
On 9/5/24 14:14, Lok P wrote: Hi, We are having a requirement to create approx 50 billion rows in a partition table(~1 billion rows per partition, 200+gb size daily partitions) for a performance test. We are currently using ' insert into select.. From or ;' method . We have dropped all in

Re: Faster data load

2024-09-05 Thread Ron Johnson
On Thu, Sep 5, 2024 at 4:14 PM Lok P wrote: > Hi, > > We are having a requirement to create approx 50 billion rows in a > partition table(~1 billion rows per partition, 200+gb size daily > partitions) for a performance test. We are currently using ' insert into > select.. From or transformed q

Faster data load

2024-09-05 Thread Lok P
Hi, We are having a requirement to create approx 50 billion rows in a partition table(~1 billion rows per partition, 200+gb size daily partitions) for a performance test. We are currently using ' insert into select.. From or ;' method . We have dropped all indexes and constraints First and then

Re: question on audit columns

2024-09-05 Thread Vincent Veyron
On Wed, 4 Sep 2024 18:19:47 +0530 yudhi s wrote: Hi, > In postgres database , we have all the tables with audit columns like > created_by_user, created_timestamp,updated_by_user, updated_timestamp. So > we have these fields that were supposed to be populated by the time at > which the insert/upd