Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Noah Misch
On Thu, Jun 16, 2011 at 04:53:41PM +0100, Simon Riggs wrote: > On Thu, Jun 16, 2011 at 3:47 PM, Noah Misch wrote: > > Thanks. ?We still hit a conflict when btpo.xact == RecentGlobalXmin and the > > standby has a transaction older than any master transaction. ?This happens > > because the tests at

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Simon Riggs
On Thu, Jun 16, 2011 at 4:53 PM, Simon Riggs wrote: > I agree with your suggested fix. Please ignore the previous patch, which was sent in error. Here's the fix. I'll apply this tomorrow morning if we all still agree. --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL De

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Simon Riggs
On Thu, Jun 16, 2011 at 3:47 PM, Noah Misch wrote: > On Thu, Jun 16, 2011 at 12:02:47AM +0100, Simon Riggs wrote: >> On Tue, Jun 14, 2011 at 5:28 AM, Noah Misch wrote: >> > On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: >> >> On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas >> >> wrot

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Noah Misch
On Thu, Jun 16, 2011 at 12:02:47AM +0100, Simon Riggs wrote: > On Tue, Jun 14, 2011 at 5:28 AM, Noah Misch wrote: > > On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: > >> On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas wrote: > >> > On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch wrote: >

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-15 Thread Simon Riggs
On Tue, Jun 14, 2011 at 5:28 AM, Noah Misch wrote: > On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: >> On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas wrote: >> > On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch wrote: >> >> Assuming that conclusion, I do think it's worth starting >> >> wi

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Noah Misch
On Mon, Jun 13, 2011 at 04:41:11PM +0100, Simon Riggs wrote: > On Thu, Jun 9, 2011 at 10:38 PM, Noah Misch wrote: > > On Fri, Apr 22, 2011 at 11:10:34AM -0400, Noah Misch wrote: > >> In an attempt to resuscitate this thread, here's my own shot at that. > >> ?Apologies > >> in advance if it's just

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Noah Misch
On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: > On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas wrote: > > On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch wrote: > >> Assuming that conclusion, I do think it's worth starting > >> with something simple, even if it means additional bloat on

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Simon Riggs
On Thu, Jun 9, 2011 at 10:38 PM, Noah Misch wrote: > On Fri, Apr 22, 2011 at 11:10:34AM -0400, Noah Misch wrote: >> On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: >> > On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: >> > > On 12.03.2011 12:40, Noah Misch wrote: >>

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Simon Riggs
On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas wrote: > On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch wrote: >> I fully agree.  That said, if this works on the standby, we may as well also >> use >> it opportunistically on the master, to throttle bloat. > > As long as the performance cost is de mini

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-12 Thread Robert Haas
On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch wrote: > I fully agree.  That said, if this works on the standby, we may as well also > use > it opportunistically on the master, to throttle bloat. As long as the performance cost is de minimis, I agree. >> At any rate, if taking a cleanup lock on th

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-12 Thread Noah Misch
On Sun, Jun 12, 2011 at 12:15:29AM -0400, Robert Haas wrote: > On Sat, Jun 11, 2011 at 11:40 PM, Noah Misch wrote: > > We currently achieve that wait-free by first marking the page with the next > > available xid and then reusing it when that mark (btpo.xact) predates the > > oldest running xid (R

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-11 Thread Robert Haas
On Sat, Jun 11, 2011 at 11:40 PM, Noah Misch wrote: > We currently achieve that wait-free by first marking the page with the next > available xid and then reusing it when that mark (btpo.xact) predates the > oldest running xid (RecentXmin).  (At the moment, I'm failing to work out why > this is OK

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-11 Thread Noah Misch
Hi Robert, On Sat, Jun 11, 2011 at 08:55:28PM -0400, Robert Haas wrote: > On Fri, Apr 22, 2011 at 11:10 AM, Noah Misch wrote: > > On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: > >> On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: > >> > On 12.03.2011 12:40, Noah M

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-11 Thread Robert Haas
On Fri, Apr 22, 2011 at 11:10 AM, Noah Misch wrote: > On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: >> On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: >> > On 12.03.2011 12:40, Noah Misch wrote: >> >> The installation that inspired my original report recently upgr

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-09 Thread Noah Misch
On Fri, Apr 22, 2011 at 11:10:34AM -0400, Noah Misch wrote: > On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: > > On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: > > > On 12.03.2011 12:40, Noah Misch wrote: > > >> The installation that inspired my original report rec

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-04-22 Thread Noah Misch
On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: > On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: > > On 12.03.2011 12:40, Noah Misch wrote: > >> The installation that inspired my original report recently upgraded from > >> 9.0.1 > >> to 9.0.3, and your fix did sign

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-15 Thread Noah Misch
On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: > On 12.03.2011 12:40, Noah Misch wrote: >> The installation that inspired my original report recently upgraded from >> 9.0.1 >> to 9.0.3, and your fix did significantly decrease its conflict frequency. >> The >> last several co

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-14 Thread Heikki Linnakangas
On 12.03.2011 12:40, Noah Misch wrote: The installation that inspired my original report recently upgraded from 9.0.1 to 9.0.3, and your fix did significantly decrease its conflict frequency. The last several conflicts I have captured involve XLOG_BTREE_REUSE_PAGE records. (FWIW, the index has g

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-12 Thread Noah Misch
On Thu, Dec 09, 2010 at 09:48:25AM +, Simon Riggs wrote: > On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: > > On 29.11.2010 08:10, Noah Misch wrote: > > > I have a hot_standby system and use it to bear the load of various > > > reporting > > > queries that take 15-60 minutes each

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-10 Thread Heikki Linnakangas
On 10.12.2010 19:55, Noah Misch wrote: On Thu, Dec 09, 2010 at 09:48:25AM +, Simon Riggs wrote: On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: Seems reasonable. HeapTupleHeaderAdvanceLatestRemovedXid() will need similar treatment. Actually, btree_xlog_delete_get_latestRemovedX

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-12-10 Thread Noah Misch
On Thu, Dec 09, 2010 at 09:48:25AM +, Simon Riggs wrote: > On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: > > On 29.11.2010 08:10, Noah Misch wrote: > > > I have a hot_standby system and use it to bear the load of various > > > reporting > > > queries that take 15-60 minutes each

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-12-09 Thread Simon Riggs
On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: > On 29.11.2010 08:10, Noah Misch wrote: > > I have a hot_standby system and use it to bear the load of various reporting > > queries that take 15-60 minutes each. In an effort to avoid long pauses in > > recovery, I set a vacuum_defer_c

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-12-03 Thread Heikki Linnakangas
On 29.11.2010 08:10, Noah Misch wrote: I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a vacuum_defer_cleanup_age constituting roughly three hours of the master's transactions.

[HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-11-28 Thread Noah Misch
I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a vacuum_defer_cleanup_age constituting roughly three hours of the master's transactions. Even so, I kept seeing recovery pause f