Re: Performance of INSERT into temporary tables using psqlODBC driver

[Sergei Kornilov]

Hello

> The data to be inserted into temporary tables is obtained from one or more
> queries run earlier and the data is available as a vector of strings.
You can not use "insert into temp_table select /*anything you wish*/" statement?
Or even insert .. select ... returning if you need receive data to application?

> If I need to use COPY FROM command, then the application would need to create 
> a
> file with the data to be inserted
You can not using "copy from stdin" statement?

regards, Sergei

Re: Performance of INSERT into temporary tables using psqlODBC driver

[padusuma]

Hello Sergei,
>> The data to be inserted into temporary tables is obtained from one or
>> more 
>> queries run earlier and the data is available as a vector of strings. 
>You can not use "insert into temp_table select /*anything you wish*/"
statement? 
>Or even insert .. select ... returning if you need receive data to
application? 
Unfortunately, the existing functionality in our application is in such a
manner that the data returned from one or more SELECT queries is processed
by server business logic and filtered, and the filtered data is then
inserted into the temporary tables. This is the reason I could not use
insert into ... select ... or insert ... select ... returning statements.

>> If I need to use COPY FROM command, then the application would need to
>> create a 
>> file with the data to be inserted 
>You can not using "copy from stdin" statement?
Thank you for suggesting the usage of "copy from stdin". I am not sure how
to pass the values to be inserted as input for "COPY FROM STDIN" statement
from my application based on psqlODBC driver. Can someone point me to an
example or suggest how to pass data from a client application to "COPY FROM
STDIN" statement?
Thanks.



--
Sent from: 
http://www.postgresql-archive.org/PostgreSQL-performance-f2050081.html

Re: Performance of INSERT into temporary tables using psqlODBC driver

[Tim Cross]

padusuma  writes:

>>We are inserting large numbers (millions) of rows into a postgres
>>database from a Javascript application and found using the COPY command
>>was much, much faster than doing regular inserts (even with multi-insert
>>commit). If you can do this using the driver you are using, that will
>>give you the largest performance boost.
>
> The data to be inserted into temporary tables is obtained from one or more
> queries run earlier and the data is available as a vector of strings. If I
> need to use COPY FROM command, then the application would need to create a
> file with the data to be inserted and the file needs to be readable by the
> user running database server process, which may not be always possible
> unless the application is running on the same host. I think this approach
> may not be feasible for our application.
>

OK, that does make a difference. If your data is already in the
database, COPY is not going to help you much.

> I have increased the value for /temp_buffers/ server parameter from the
> default 8 MB to 128 MB. However, this change did not affect the INSERT time
> for temporary tables.

It isn't clear why you create vectors of strings rather than just select
into or something similar.

There are no 'quick fixes' which can be applied without real analysis of
the system. However, based on the limited information available, you may
want to consider -

- Increase work_mem to reduce use of temp files. Need it to be 2 to 3
  times largest temp file (but use common sense)

- Tweak wal checkpoint parameters to prevent wal checkpoints occurring
  too frequently. Note that there is a play off here between frequency
  of checkpoints and boot time after a crash. Fewer wal checkpoints will
  usually improve performance, but recovery time is longer.

- Verify your inserts into temporary tables is the bottleneck and not
  the select from existing data (explain plan etc and adjust indexes
  accordingly).

How effectively you can increase insert times will depend on what the
memory and cpu profile of the system is. More memory, less use of temp
files, faster system, so spend a bit of time to make sure your system is
configured to squeeze as much out of that RAM as you can!

--
Tim Cross