Re: [HACKERS] Optimising Foreign Key checks

2013-06-10 Thread Noah Misch
On Mon, Jun 10, 2013 at 09:05:40AM +0100, Simon Riggs wrote: > Your earlier comments argue that it is OK to make an early check. The > above seems to argue the opposite, not sure. I'll attempt to summarize. If we execute a traditional error-throwing FK check any earlier than we execute it today,

Re: [HACKERS] Optimising Foreign Key checks

2013-06-10 Thread Simon Riggs
On 10 June 2013 07:06, Noah Misch wrote: > On Sun, Jun 09, 2013 at 10:51:43AM +0100, Simon Riggs wrote: >> On 9 June 2013 02:12, Noah Misch wrote: >> > On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: >> >> On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: >> >> > Likewise; I don't s

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Noah Misch
On Sun, Jun 09, 2013 at 10:51:43AM +0100, Simon Riggs wrote: > On 9 June 2013 02:12, Noah Misch wrote: > > On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: > >> On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: > >> > Likewise; I don't see why we couldn't perform an optimistic check A

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Simon Riggs
On 9 June 2013 14:59, Greg Stark wrote: > On Sun, Jun 9, 2013 at 10:51 AM, Simon Riggs wrote: >> AFAICS there are weird cases where changing the way FKs execute will >> change the way complex trigger applications will execute. I don't see >> a way to avoid that other than "do nothing". Currently,

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Greg Stark
On Sun, Jun 9, 2013 at 10:51 AM, Simon Riggs wrote: > AFAICS there are weird cases where changing the way FKs execute will > change the way complex trigger applications will execute. I don't see > a way to avoid that other than "do nothing". Currently, we execute the > checks following the normal

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Andres Freund
On 2013-06-01 09:41:13 +0100, Simon Riggs wrote: > FK checks can be expensive, especially when loading large volumes of > data into an existing table or partition. A couple of ideas for > improving performance are discussed here: Another idea would be to optimize away the row level locks if we hav

Re: [HACKERS] Optimising Foreign Key checks

2013-06-09 Thread Simon Riggs
On 9 June 2013 02:12, Noah Misch wrote: > On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: >> On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: >> > Likewise; I don't see why we couldn't perform an optimistic check ASAP and >> > schedule a final after-statement check when an early che

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: > On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: > > Likewise; I don't see why we couldn't perform an optimistic check ASAP and > > schedule a final after-statement check when an early check fails. That > > changes performance charac

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Robert Haas
On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: > This does appear to specify FK timing semantics like PostgreSQL gives today. > Namely, it does not permit a FK-induced error when later actions of the query > that prompted the check could possibly remedy the violation. Yeah. Standard or no sta

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 09:39:14PM +0100, Simon Riggs wrote: > On 8 June 2013 15:30, Noah Misch wrote: > > On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: > >> 2. Don't store FK events in the after trigger queue at all, but apply > >> them as we go. That solves problems2 and 3. That

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Simon Riggs
On 8 June 2013 15:30, Noah Misch wrote: > On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: >> > On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: >> >> For clarity the 4 problems are >> >> 1. SQL execution overhead >> >> 2. Memory usage >> >> 3. Memory scrolling >> >> 4. Loc

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: > > On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: > >> For clarity the 4 problems are > >> 1. SQL execution overhead > >> 2. Memory usage > >> 3. Memory scrolling > >> 4. Locking overhead, specifically FPWs and WAL records

Re: [HACKERS] Optimising Foreign Key checks

2013-06-05 Thread Hannu Krosing
On 06/05/2013 11:37 AM, Greg Stark wrote: > On Sat, Jun 1, 2013 at 9:41 AM, Simon Riggs wrote: >> COMMIT; >> The inserts into order_line repeatedly execute checks against the same >> ordid. Deferring and then de-duplicating the checks would optimise the >> transaction. >> >> Proposal: De-duplicate

Re: [HACKERS] Optimising Foreign Key checks

2013-06-05 Thread Greg Stark
On Sat, Jun 1, 2013 at 9:41 AM, Simon Riggs wrote: > COMMIT; > The inserts into order_line repeatedly execute checks against the same > ordid. Deferring and then de-duplicating the checks would optimise the > transaction. > > Proposal: De-duplicate multiple checks against same value. This would >

Re: [HACKERS] Optimising Foreign Key checks

2013-06-04 Thread Simon Riggs
On 4 June 2013 01:54, Noah Misch wrote: > On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: >> For clarity the 4 problems are >> 1. SQL execution overhead >> 2. Memory usage >> 3. Memory scrolling >> 4. Locking overhead, specifically FPWs and WAL records from FK checks >> probably in th

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Noah Misch
On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: > For clarity the 4 problems are > 1. SQL execution overhead > 2. Memory usage > 3. Memory scrolling > 4. Locking overhead, specifically FPWs and WAL records from FK checks > probably in that order or thereabouts. > > The above is why I

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Simon Riggs
On 3 June 2013 19:41, Jim Nasby wrote: > On 6/2/13 4:45 AM, Simon Riggs wrote: >>> >>> >Will this add too much cost where it doesn't help? I don't know what to >>> >predict there. There's the obvious case of trivial transactions with no >>> > more >>> >than one referential integrity check per FK

Re: [HACKERS] Optimising Foreign Key checks

2013-06-03 Thread Jim Nasby
On 6/2/13 4:45 AM, Simon Riggs wrote: >Will this add too much cost where it doesn't help? I don't know what to >predict there. There's the obvious case of trivial transactions with no more >than one referential integrity check per FK, but there's also the case of a >transaction with many FK che

Re: [HACKERS] Optimising Foreign Key checks

2013-06-02 Thread Simon Riggs
On 1 June 2013 21:27, Noah Misch wrote: > On Sat, Jun 01, 2013 at 09:41:13AM +0100, Simon Riggs wrote: >> FK checks can be expensive, especially when loading large volumes of >> data into an existing table or partition. A couple of ideas for >> improving performance are discussed here: >> >> 1. Us

Re: [HACKERS] Optimising Foreign Key checks

2013-06-01 Thread Noah Misch
On Sat, Jun 01, 2013 at 09:41:13AM +0100, Simon Riggs wrote: > FK checks can be expensive, especially when loading large volumes of > data into an existing table or partition. A couple of ideas for > improving performance are discussed here: > > 1. Use Case: Bulk loading > COPY pgbench_accounts;

[HACKERS] Optimising Foreign Key checks

2013-06-01 Thread Simon Riggs
FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here: 1. Use Case: Bulk loading COPY pgbench_accounts; --> references pgbench_branches with many repeated values Proposal: Tran