Hi,
Performance Considerations
Avro files are smaller due to compression so needing less I/O time.
whereas CSV files are simpler but larger in size so read/write will need
more time.
COPY command works very well with CSV files whereas ETL process is
required for handling Avro.
Regards,
Muhammad Ikram
On Fri, Jul 5, 2024 at 3:03 PM Josef Šimánek <josef.sima...@gmail.com>
wrote:
> pá 5. 7. 2024 v 11:08 odesílatel sud <suds1...@gmail.com> napsal:
> >
> > Hello all,
> >
> > Its postgres database. We have option of getting files in csv and/or in
> avro format messages from another system to load it into our postgres
> database. The volume will be 300million messages per day across many files
> in batches.
> >
> > My question was, which format should we chose in regards to faster data
> loading performance ? and if any other aspects to it also should be
> considered apart from just loading performance?
>
> We are able to load ~300 million rows per one day using CSV and COPY
> functions (
> https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND).
>
>
>
--
Muhammad Ikram
