Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-06 Thread Tom Lane
Andres Freund writes: > On 2014-02-06 16:35:07 -0500, Tom Lane wrote: >> Thoughts? > Let's let it stew a while in master, there certainly are enough > subtleties around this that I'd hesitate to add it to a point release in > the not too far away future. Doesn't seem that urgent to me. But after

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-06 Thread Tom Lane
I wrote: > (BTW, I see that the CLOBBER_CACHE_ALWAYS members of the buildfarm are > unhappy with the IsTransactionState check in RelationIdGetRelation. > Will look at that ... but it seems to be in initdb which breaks a lot > of these rules anyway, so I think it's probably not significant.) So wha

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-06 Thread Andres Freund
On 2014-02-06 16:35:07 -0500, Tom Lane wrote: > > I wonder though, if we couldn't just stop doing the > > RelationReloadIndexInfo() for nailed indexes. > > No; you're confusing nailed indexes with mapped indexes. There are nailed > indexes that aren't on mapped catalogs, see the load_critical_ind

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-06 Thread Tom Lane
Andres Freund writes: > On 2014-02-02 15:16:45 -0500, Tom Lane wrote: >> I've been thinking about this for the past little while, and I believe >> that it's probably okay to have RelationClearRelation leave the relcache >> entry un-rebuilt, but with rd_isvalid = false so it will be rebuilt when >>

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-05 Thread Tom Lane
Andres Freund writes: > On 2014-02-05 14:07:29 -0500, Tom Lane wrote: >> I stuck such an Assert in ScanPgRelation, and verified that it doesn't >> break any existing regression tests --- although of course the above >> test case still fails, and now even without declaring an index. >> >> Barring

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-05 Thread Andres Freund
On 2014-02-05 14:07:29 -0500, Tom Lane wrote: > I stuck such an Assert in ScanPgRelation, and verified that it doesn't > break any existing regression tests --- although of course the above > test case still fails, and now even without declaring an index. > > Barring objections I'll go commit that

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-05 Thread Noah Misch
On Wed, Feb 05, 2014 at 02:07:29PM -0500, Tom Lane wrote: > Of course, a direct system catalog scan is certainly no safer in an > aborted transaction than the one that catcache.c is refusing to do. > Therefore, in my opinion, relcache.c ought also to be doing an > Assert(IsTransactionState()), at l

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-05 Thread Tom Lane
Noah Misch writes: > The following test case reliably hits this new assertion: > create table t (c int); > begin; > create index on t (c); > savepoint q; > insert into t values (1); > select 1/0; As of commit ac8bc3b6e4, this test case no longer triggers the assertion, because it depends on the

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-02 Thread Andres Freund
On 2014-02-02 15:16:45 -0500, Tom Lane wrote: > Andres Freund writes: > > On February 2, 2014 5:52:22 PM CET, Tom Lane wrote: > >> More to the point, changing the Assert so it doesn't fire > >> doesn't do one damn thing to ameliorate the fact that cache reload > >> during transaction abort is wro

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-02 Thread Tom Lane
Andres Freund writes: > On February 2, 2014 5:52:22 PM CET, Tom Lane wrote: >> More to the point, changing the Assert so it doesn't fire >> doesn't do one damn thing to ameliorate the fact that cache reload >> during transaction abort is wrong and unsafe. > And, as upthread, I still don't think

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-02 Thread Andres Freund
On February 2, 2014 5:52:22 PM CET, Tom Lane wrote: >Andres Freund writes: >> On 2014-01-31 16:41:33 -0500, Bruce Momjian wrote: >>> Is there any plan to commit this? > >> IMO it has to be applied. Tom objected on the grounds that cache >> invalidation has to be fixed properly but that's a major

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-02 Thread Tom Lane
Andres Freund writes: > On 2014-01-31 16:41:33 -0500, Bruce Momjian wrote: >> Is there any plan to commit this? > IMO it has to be applied. Tom objected on the grounds that cache > invalidation has to be fixed properly but that's a major restructuring > of code that worked this way for a long tim

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-02-02 Thread Andres Freund
On 2014-01-31 16:41:33 -0500, Bruce Momjian wrote: > On Mon, Oct 7, 2013 at 03:02:36PM -0400, Robert Haas wrote: > > On Fri, Oct 4, 2013 at 3:20 PM, Andres Freund > > wrote: > > > On 2013-10-04 15:15:36 -0400, Robert Haas wrote: > > >> Andres, are you (or is anyone) going to try to fix this asse

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2014-01-31 Thread Bruce Momjian
On Mon, Oct 7, 2013 at 03:02:36PM -0400, Robert Haas wrote: > On Fri, Oct 4, 2013 at 3:20 PM, Andres Freund wrote: > > On 2013-10-04 15:15:36 -0400, Robert Haas wrote: > >> Andres, are you (or is anyone) going to try to fix this assertion failure? > > > > I think short term replacing it by IsTran

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-10-07 Thread Andres Freund
On 2013-10-07 15:02:36 -0400, Robert Haas wrote: > On Fri, Oct 4, 2013 at 3:20 PM, Andres Freund wrote: > > On 2013-10-04 15:15:36 -0400, Robert Haas wrote: > >> Andres, are you (or is anyone) going to try to fix this assertion failure? > > > > I think short term replacing it by IsTransactionOrTra

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-10-07 Thread Robert Haas
On Fri, Oct 4, 2013 at 3:20 PM, Andres Freund wrote: > On 2013-10-04 15:15:36 -0400, Robert Haas wrote: >> Andres, are you (or is anyone) going to try to fix this assertion failure? > > I think short term replacing it by IsTransactionOrTransactionBlock() is > the way to go. Entirely restructuring

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-10-04 Thread Andres Freund
On 2013-10-04 15:15:36 -0400, Robert Haas wrote: > Andres, are you (or is anyone) going to try to fix this assertion failure? I think short term replacing it by IsTransactionOrTransactionBlock() is the way to go. Entirely restructuring how cache invalidation in the abort path works is not a small

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-10-04 Thread Robert Haas
On Fri, Aug 9, 2013 at 11:17 PM, Andres Freund wrote: > On 2013-08-09 14:11:46 -0400, Tom Lane wrote: >> Andres Freund writes: >> > On 2013-08-08 09:27:24 -0400, Robert Haas wrote: >> >> How can it be safe to try to read catalogs if the transaction is aborted? >> >> > Well. It isn't. At least not

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-09 Thread Andres Freund
On 2013-08-09 14:11:46 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2013-08-08 09:27:24 -0400, Robert Haas wrote: > >> How can it be safe to try to read catalogs if the transaction is aborted? > > > Well. It isn't. At least not in general. The specific case triggered > > here though are

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-09 Thread Noah Misch
On Fri, Aug 09, 2013 at 02:11:46PM -0400, Tom Lane wrote: > Cache invalidation during abort should *not* lead to any attempt to > immediately revalidate the cache. No amount of excuses will make that > okay. I have not looked to see just what the path of control is in this > particular case, but

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-09 Thread Tom Lane
Andres Freund writes: > On 2013-08-08 09:27:24 -0400, Robert Haas wrote: >> How can it be safe to try to read catalogs if the transaction is aborted? > Well. It isn't. At least not in general. The specific case triggered > here though are cache invalidations being processed which can lead to > th

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-09 Thread Noah Misch
On Tue, Aug 06, 2013 at 09:06:59AM +0200, Andres Freund wrote: > On 2013-08-05 13:09:31 -0400, Noah Misch wrote: > > When we call AtEOSubXact_Inval() or AtEOXact_Inval() with a relation still > > open, we can potentially get a relcache rebuild and therefore a syscache > > lookup as shown above. Co

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-09 Thread Andres Freund
On 2013-08-09 09:05:51 -0400, Robert Haas wrote: > On Thu, Aug 8, 2013 at 1:28 PM, Andres Freund wrote: > > Well. It isn't. At least not in general. The specific case triggered > > here though are cache invalidations being processed which can lead to > > the catalog being read (pretty crummy, but

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-09 Thread Robert Haas
On Thu, Aug 8, 2013 at 1:28 PM, Andres Freund wrote: > Well. It isn't. At least not in general. The specific case triggered > here though are cache invalidations being processed which can lead to > the catalog being read (pretty crummy, but not easy to get rid > of). That's actually safe since bef

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-08 Thread Andres Freund
On 2013-08-08 09:27:24 -0400, Robert Haas wrote: > On Tue, Aug 6, 2013 at 3:06 AM, Andres Freund wrote: > > The reason for that is that when we do the AtEO(Sub)?Xact_Inval(), we've > > already done a RecordTransactionAbort(true|false) and > > CurrentTransactionState->state = TRANS_ABORT. So the vi

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-08 Thread Robert Haas
On Tue, Aug 6, 2013 at 3:06 AM, Andres Freund wrote: > The reason for that is that when we do the AtEO(Sub)?Xact_Inval(), we've > already done a RecordTransactionAbort(true|false) and > CurrentTransactionState->state = TRANS_ABORT. So the visibility routines > have enough information to consider r

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-06 Thread Andres Freund
On 2013-08-05 13:09:31 -0400, Noah Misch wrote: > On Fri, Jul 12, 2013 at 11:42:23AM +0200, Andres Freund wrote: > > On 2013-07-11 15:09:45 -0400, Tom Lane wrote: > > > It never has been, and never will be, allowed to call the catcache code > > > without being in a transaction. What do you think w

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-05 Thread Andres Freund
On 2013-08-05 13:09:31 -0400, Noah Misch wrote: > On Fri, Jul 12, 2013 at 11:42:23AM +0200, Andres Freund wrote: > > On 2013-07-11 15:09:45 -0400, Tom Lane wrote: > > > It never has been, and never will be, allowed to call the catcache code > > > without being in a transaction. What do you think w

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-08-05 Thread Noah Misch
On Fri, Jul 12, 2013 at 11:42:23AM +0200, Andres Freund wrote: > On 2013-07-11 15:09:45 -0400, Tom Lane wrote: > > It never has been, and never will be, allowed to call the catcache code > > without being in a transaction. What do you think will happen if the > > requested row isn't in cache? A t

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-07-15 Thread Robert Haas
On Fri, Jul 12, 2013 at 5:42 AM, Andres Freund wrote: > I'd like to add an Assert like in the attached patch making sure we're > in a transaction. Otherwise it's far too easy not to hit an error during > development because everything is cached - and syscache usage isn't > always obvious from the

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-07-12 Thread Andres Freund
Hi, On 2013-07-11 11:53:57 -0700, Jeff Davis wrote: > On Thu, 2013-07-11 at 10:28 +0200, Andres Freund wrote: > > There doesn't seem be an explicitly stated rule that we cannot use the > > syscaches outside of a transaction - but effectively that's required > > atm. > > Aren't there other things

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-07-11 Thread Tom Lane
Andres Freund writes: > since the mvcc catalog patch has gone in we require all users of > systable_* to be in a valid transaction since the snapshot is copied via > CopySnapshot() in RegisterSnapshot(). It never has been, and never will be, allowed to call the catcache code without being in a tr

Re: [HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-07-11 Thread Jeff Davis
On Thu, 2013-07-11 at 10:28 +0200, Andres Freund wrote: > There doesn't seem be an explicitly stated rule that we cannot use the > syscaches outside of a transaction - but effectively that's required > atm. Aren't there other things that already required that before the MVCC catalog snapshot patch

[HACKERS] mvcc catalo gsnapshots and TopTransactionContext

2013-07-11 Thread Andres Freund
Hi, since the mvcc catalog patch has gone in we require all users of systable_* to be in a valid transaction since the snapshot is copied via CopySnapshot() in RegisterSnapshot(). Which we call in systable_beginscan(). CopySnapshot() allocates the copied snapshot in TopTransactionContext. There d