Re: [GENERAL] bulk loading table via join of 2 large staging tables

2014-01-03 Thread Seb
On Tue, 31 Dec 2013 02:23:30 +, Brent Wood wrote: > This should help... In each temporary table convert the time parts to > a timestamp, then create an index on each of these, then join on the > timestamp. [...] Thank you, these were very useful suggestions, and lead to a very efficient ta

Re: [GENERAL] bulk loading table via join of 2 large staging tables

2013-12-31 Thread Brent Wood
This should help... In each temporary table convert the time parts to a timestamp, then create an index on each of these, then join on the timestamp. ALTER table mmc add column timer timestamp without time zone; UPDATE mmc set timer = (utc_year || '-' || utc_month || '-' || utc_day || ' ' ||

Re: [GENERAL] bulk loading table via join of 2 large staging tables

2013-12-30 Thread Andrew Sullivan
On Mon, Dec 30, 2013 at 07:53:06PM -0600, Seb wrote: > Given that the process involves a full join, I'm not sure I can do this > in chunks (say breaking down the files into smaller pieces). Any > suggestions would be greatly appreciated. First, what I would probably do is merge the two files outsi

Re: [GENERAL] bulk loading table via join of 2 large staging tables

2013-12-30 Thread David Johnston
Quick thoughts: On both tables: Convert your date-time varchar fields into a single epoch/integer field. Create an index of that epoch/integer field. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/bulk-loading-table-via-join-of-2-large-staging-tables-tp57