Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Christopher Kings-Lynne
Actually, there are really only a few errors people want to trap I imagine: - CHECK constraints (all handled in ExecConstraints) - Duplicate keys - Foreign key violations (all handled by triggers) Rather than worry about all the events we can't safely trap, how about we simply deal with the hand

Re: [HACKERS] Practical error logging for very large COPY statements

2005-11-22 Thread Christopher Kings-Lynne
Seems similar to the pgloader project on pgfoundry.org. It is similar and good, but I regard that as a workaround rather than the way forward. Yes, your way would be rad :) ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, ple

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Martijn van Oosterhout
On Tue, Nov 22, 2005 at 12:16:00PM -0500, Tom Lane wrote: > I think the distinction you are proposing between constraint errors > and datatype errors is entirely artificial. Who's to say what is a > constraint error and what is a datatype error, especially when you > start thinking about cases lik

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > > It would be nice to be able to have the former loaded into an actual table > > where it can be queried and perhaps fixed and reloaded. > > > The latter clearly cannot. > > Sure it can --- you just have to dump it as raw text (or perhaps bytea, > as someon

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Tom Lane
Greg Stark <[EMAIL PROTECTED]> writes: > I think that's precisely the point here though. There are basically two > categories of errors: > 1) Data that can be parsed and loaded but generates some sort of constraint >violation such as a UNIQUE violation, foreign key violation, or other >con

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Greg Stark
Tom Lane <[EMAIL PROTECTED]> writes: > Martijn van Oosterhout writes: > > On Tue, Nov 22, 2005 at 09:58:44AM -0500, Tom Lane wrote: > >> The general problem that needs to be solved is "trap any error that > >> occurs during attempted insertion of a COPY row, and instead of aborting > >> the copy,

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Martijn van Oosterhout
On Tue, Nov 22, 2005 at 10:45:50AM -0500, Tom Lane wrote: > Martijn van Oosterhout writes: > > Actually, there are really only a few errors people want to trap I > > imagine: > > You've forgotten bad data, eg "foo" in an integer field, or an > untranslatable multibyte character. The bad-data pro

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Tom Lane
Martijn van Oosterhout writes: > On Tue, Nov 22, 2005 at 09:58:44AM -0500, Tom Lane wrote: >> The general problem that needs to be solved is "trap any error that >> occurs during attempted insertion of a COPY row, and instead of aborting >> the copy, record the data and the error message someplace

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Martijn van Oosterhout
On Tue, Nov 22, 2005 at 09:58:44AM -0500, Tom Lane wrote: > The general problem that needs to be solved is "trap any error that > occurs during attempted insertion of a COPY row, and instead of aborting > the copy, record the data and the error message someplace else". Seen > in that light, implem

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > I have committed the sin of omission again. > Duplicate row violation is the big challenge, but not the only function > planned. Formatting errors occur much more frequently, so yes we'd want > to log all of that too. And yes, it would be done in the way y

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Simon Riggs
On Mon, 2005-11-21 at 19:05 -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > Flow of control would be to: > > > locate page of index where value should go > > lock index block > > _bt_check_unique, but don't error > > if violation then insert row into ERRORT

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Simon Riggs
On Tue, 2005-11-22 at 10:00 +0800, Christopher Kings-Lynne wrote: > Seems similar to the pgloader project on pgfoundry.org. It is similar and good, but I regard that as a workaround rather than the way forward. Best Regards, Simon Riggs ---(end of broadcast)

Re: [HACKERS] Practical error logging for very large COPY

2005-11-22 Thread Simon Riggs
On Mon, 2005-11-21 at 19:38 -0500, Andrew Dunstan wrote: > > Tom Lane wrote: > > >Simon Riggs <[EMAIL PROTECTED]> writes: > > > > > >>What I'd like to do is add an ERRORTABLE clause to COPY. The main > >>problem is how we detect a duplicate row violation, yet prevent it from > >>aborting the tr

Re: [HACKERS] Practical error logging for very large COPY statements

2005-11-21 Thread Christopher Kings-Lynne
Seems similar to the pgloader project on pgfoundry.org. Chris Simon Riggs wrote: If you've ever loaded 100 million rows, you'll know just how annoying it is to find that you have a duplicate row somewhere in there. Experience shows that there is always one, whatever oath the analyst swears befo

Re: [HACKERS] Practical error logging for very large COPY statements

2005-11-21 Thread Andrew Dunstan
Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: What I'd like to do is add an ERRORTABLE clause to COPY. The main problem is how we detect a duplicate row violation, yet prevent it from aborting the transaction. If this only solves the problem of duplicate keys, and not any

Re: [HACKERS] Practical error logging for very large COPY statements

2005-11-21 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > What I'd like to do is add an ERRORTABLE clause to COPY. The main > problem is how we detect a duplicate row violation, yet prevent it from > aborting the transaction. If this only solves the problem of duplicate keys, and not any other kind of COPY error,