Re: [PERFORM] Postgres bulk insert/ETL performance on high speed servers - test results

2016-09-07 Thread Mike Sofen
From: Jim Nasby [mailto:jim.na...@bluetreble.com] Sent: Wednesday, September 07, 2016 12:22 PM On 9/4/16 7:34 AM, Mike Sofen wrote: > You raise a good point. However, other disk activities involving > large data (like backup/restore and pure large table copying), on both > platforms, do not

Re: [PERFORM] Postgres bulk insert/ETL performance on high speed servers - test results

2016-09-07 Thread Jim Nasby
On 9/4/16 7:34 AM, Mike Sofen wrote: You raise a good point. However, other disk activities involving large data (like backup/restore and pure large table copying), on both platforms, do not seem to support that notion. I did have both our IT department and Cisco turn on instrumentation for my

Re: [PERFORM] Postgres bulk insert/ETL performance on high speed servers - test results

2016-09-04 Thread Mike Sofen
From: Claudio Freire Sent: Friday, September 02, 2016 1:27 PM On Thu, Sep 1, 2016 at 11:30 PM, Mike Sofen < mso...@runbox.com> wrote: > It's obvious the size of the batch exceeded the AWS server memory, > resulting in a profoundly slower processing time. This was a

Re: [PERFORM] Postgres bulk insert/ETL performance on high speed servers - test results

2016-09-02 Thread Claudio Freire
On Thu, Sep 1, 2016 at 11:30 PM, Mike Sofen wrote: > PASS 2: > Process: Transform/Load (all work local to the server - read, > transform, write as a single batch) > Num Source Rows: 10,554,800 (one batch from just a single source table > going to a single target table) > Avg Rowcount Com

[PERFORM] Postgres bulk insert/ETL performance on high speed servers - test results

2016-09-01 Thread Mike Sofen
High level summary: server ram has a significant impact on batch processing performance (no surprise), and AWS processing can largely compete with local servers IF the AWS network connection is optimized. With the recent threads about insert performance (and performance in general), I thought I'd