Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread Paul A Jungwirth
Hi Eugene, > Now I need to import the patch into the database, and produce another file as > - if the passed "series" field exists in the database, then return ID:series > - otherwise insert a new row to the table and generate new ID and return > ID:series > for each row in the source file. I th

Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread Eugene Dzhurinsky
On Sun, Feb 15, 2015 at 06:48:36PM +0100, Francisco Olarte wrote: > One thing that strikes me is you are either at the beginning of your usage > of this or you have A LOT of already present lines in the path ( I mean, > path has one fourth of the lines of dictionary ). Which is the case? Right now

Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread Francisco Olarte
Hi Eugene: On Sun, Feb 15, 2015 at 6:36 PM, Eugene Dzhurinsky wrote: > ​...​ > > Since the "dictionary" already has an index on the "series", it seems that > patch_data doesn't need to have any index here. > ​ > At this point "patch_data" needs to get an index on "already_exists = > false"

Fwd: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread Francisco Olarte
Forwarding to the list, as I forgot ( as usual, think I did the same with my original message ). Sorry for the duplicates. -- Forwarded message -- From: Francisco Olarte Date: Sun, Feb 15, 2015 at 6:48 PM Subject: Re: [GENERAL] Import large data set into a table and resolve duplica

Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread Eugene Dzhurinsky
On Sun, Feb 15, 2015 at 10:00:50AM -0600, John McKown wrote: > UPDATE patch_data SET already_exists=((SELECT TRUE FROM dictionary WHERE > dictionary.series = patch_data.series)); Since the "dictionary" already has an index on the "series", it seems that patch_data doesn't need to have any index he

Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread Eugene Dzhurinsky
On Sun, Feb 15, 2015 at 01:06:02PM +0100, Francisco Olarte wrote: > You state below 200k rows, 50k lines per path. That is not huge unless > "series" really big, is it? series data is in between of 100-4096 chars > 1.- Get the patches into a ( temp ) table, using something like \copy, call > this

Re: [GENERAL] partial "on-delete set null" constraint

2015-02-15 Thread Rafal Pietrak
W dniu 03.02.2015 o 04:44, Jim Nasby pisze: On 1/3/15 2:49 AM, Rafal Pietrak wrote: [---] But an application could do -a successfull scenario with expected result--- testvm=# UPDATE mailboxes SET username = null; UPDATE 1 testvm=# DELETE FROM mail

Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread John McKown
OOPS, I forgot to mention in the SELECT generating the output file that the e'\t' generates a "tab" character. You likely already know this, but I like to make my posts as self contained as possible. On Sun, Feb 15, 2015 at 10:00 AM, John McKown wrote: > On Sat, Feb 14, 2015 at 3:54 PM, Eugene D

Re: [GENERAL] Import large data set into a table and resolve duplicates?

2015-02-15 Thread John McKown
On Sat, Feb 14, 2015 at 3:54 PM, Eugene Dzhurinsky wrote: > Hello! > > I have a huge dictionary table with series data generated by a third-party > service. The table consists of 2 columns > > - id : serial, primary key > - series : varchar, not null, indexed > > From time to time I need to apply