Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-11 Thread Frits Jalvingh
Hi Alvaro, I did not try that, to be honest. I am using a single prepared statement so that the database needs to parse it only once. All executes then use the batched parameters. I will try this later on, but I wonder whether having to reparse the statement every time compared to one prepared sta

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-10 Thread Frits Jalvingh
I think binary is worse.. according to the postgres documentation: The binary format option causes all data to be stored/read as binary format rather than as text. It is somewhat faster than the text and CSV formats, but a binary-format file is less portable across machine architectures and Postgr

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-10 Thread Frits Jalvingh
On Sat, Jun 10, 2017 at 12:08 AM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > Would you mind sharing the source code of your benchmark? > The source code for the several tests, plus the numbers collected so far, can be found at: https://etc.to/confluence/display/~admjal/PostgreSQL+p

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
2017 at 5:33 PM Kenneth Marshall wrote: > On Fri, Jun 09, 2017 at 03:22:35PM +0000, Frits Jalvingh wrote: > > Hi Babu, > > > > That was all already done, as it is common practice for JDBC. Your > > parameter was added to the code that already did all that - and worked &

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
Hi Babu, That was all already done, as it is common practice for JDBC. Your parameter was added to the code that already did all that - and worked brilliantly there ;) >

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
[mailto: > pgsql-performance-ow...@postgresql.org] *On Behalf Of *Frits Jalvingh > *Sent:* Friday, June 09, 2017 7:55 AM > *To:* Sunkara, Amrutha; pgsql-performance@postgresql.org > > > *Subject:* Re: [PERFORM] Improving PostgreSQL insert performance > > > > I am not doing

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
Hi Babu, No, I did not, and the effect is quite great: Inserted 100 rows in 2535 milliseconds, 394477.3175542406 rows per second Inserted 100 rows in 2553 milliseconds, 391696.0438699569 rows per second compared to (without your parameter): Inserted 100 rows in 7643 milliseconds, 130

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
Hi Kenneth, I tried unlogged before, but as long as the commit interval is long it had no discerning effect that I could see.

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
I am not doing anything special I guess. I am adding the results of the tests and the programs I'm using to the following page: https://etc.to/confluence/display/~admjal/PostgreSQL+performance+tests The copy example, in Java, is at the end. All of the examples use trivial data and the same data.

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
mance hit with the JDBC driver itself (as the stored procedure is a lot faster), so I can look into that. But even after that there is quite a gap.. Regards, Frits On Fri, Jun 9, 2017 at 4:33 PM Scott Marlowe wrote: > On Fri, Jun 9, 2017 at 7:56 AM, Frits Jalvingh wrote: > > Hi Ke

Re: [PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
etschmer wrote: > > > > > > Am 09.06.2017 um 15:04 schrieb Frits Jalvingh: > > >Hi all, > > > > > >I am trying to improve the runtime of a big data warehouse > > >application. One significant bottleneck found was insert > > >performanc

[PERFORM] Improving PostgreSQL insert performance

2017-06-09 Thread Frits Jalvingh
Hi all, I am trying to improve the runtime of a big data warehouse application. One significant bottleneck found was insert performance, so I am investigating ways of getting Postgresql to insert data faster. I ran several tests on a fast machine to find out what performs best, and compared the re

[PERFORM] Postgres delete performance problem

2012-06-25 Thread Frits Jalvingh
Hi, I have a Java application that tries to synchronize tables in two databases (remote source to local target). It does so by removing all constraints, then it compares table contents row by row, inserts missing rows and deletes "extra" rows in the target database. Delete performance is incredibl