Re: Advice on efficiently logging outputs to PostgreSQL

2024-10-15 Thread Christophe Pettus
> On Oct 15, 2024, at 07:17, Dominique Devienne wrote: > Am I worrying too much? :) Probably. :-) The main things I'd worry about is: 1. What's the ratio of log lines to database updates? You want this to be as high as usefully possible, since in effect you are doing write amplification by

Advice on efficiently logging outputs to PostgreSQL

2024-10-15 Thread Dominique Devienne
I have an existing heavy ETL that serially loads tons of data to PostgreSQL. This is done using a CLI tool, processing one project after another. I'd like to parallelize / distribute the work, which I could do from my CLI tool, but 1) that would be confined to a single machine, and 2) we'd like to