Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Tomas Vondra
On 01/29/2018 03:17 PM, Simon Riggs wrote: > On 29 January 2018 at 14:13, Tomas Vondra > wrote: > >> 4) inspect the new row (which we still have in reorderbuffer) >> >> 5) Kabooom! The row has column "c" which we don't see in the catalog. > > We don't use caches? Why does a cache miss cause i

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Simon Riggs
On 29 January 2018 at 14:13, Tomas Vondra wrote: > 4) inspect the new row (which we still have in reorderbuffer) > > 5) Kabooom! The row has column "c" which we don't see in the catalog. We don't use caches? Why does a cache miss cause it to explode? -- Simon Riggshttp://www.2n

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Tomas Vondra
On 01/29/2018 02:49 PM, Simon Riggs wrote: > On 29 January 2018 at 13:34, Tomas Vondra > wrote: > >> The important detail is that we only really care >> about aborts in transactions that modified catalogs in some way (e.g. by >> doing DDL). But we can safely decode (and stream) changes up to t

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Simon Riggs
On 29 January 2018 at 13:34, Tomas Vondra wrote: > The important detail is that we only really care > about aborts in transactions that modified catalogs in some way (e.g. by > doing DDL). But we can safely decode (and stream) changes up to the > point when the catalogs get modified, so we can do

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Tomas Vondra
Hi, On 01/29/2018 11:23 AM, Simon Riggs wrote: > On 29 January 2018 at 07:15, Nikhil Sontakke wrote: > >>> Having this as responsibility of plugin sounds interesting. It >>> certainly narrows the scope for which we need to solve the abort >>> issue. For 2PC that may be okay as we need to somehow

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Simon Riggs
On 23 January 2018 at 19:17, Petr Jelinek wrote: > I am not sure if this helps streaming use-case though as > there is not going to be any external transaction management involved there. So, I think we need some specific discussion of what to do in that case. Streaming happens only with big tra

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-29 Thread Simon Riggs
On 29 January 2018 at 07:15, Nikhil Sontakke wrote: >> Having this as responsibility of plugin sounds interesting. It certainly >> narrows the scope for which we need to solve the abort issue. For 2PC >> that may be okay as we need to somehow interact with transaction manager >> as Simon noted. I

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-28 Thread Nikhil Sontakke
Hi, > Having this as responsibility of plugin sounds interesting. It certainly > narrows the scope for which we need to solve the abort issue. For 2PC > that may be okay as we need to somehow interact with transaction manager > as Simon noted. I am not sure if this helps streaming use-case though

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-23 Thread Petr Jelinek
On 23/01/18 16:01, Nikhil Sontakke wrote: >> I must be missing something in this discussion, cos I don't see any >> problems with this "other option". >> >> Surely we prepare the 2PCxact and then it is persistent. Yes, >> potentially for an unbounded period of time. And it is (usually) up to >> the

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-23 Thread Nikhil Sontakke
> I must be missing something in this discussion, cos I don't see any > problems with this "other option". > > Surely we prepare the 2PCxact and then it is persistent. Yes, > potentially for an unbounded period of time. And it is (usually) up to > the XA manager to resolve that. 2PC interacts with

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-23 Thread Simon Riggs
On 22 January 2018 at 20:03, Petr Jelinek wrote: > On 22/01/18 20:21, Robert Haas wrote: >>> The other option would be to make sure 2PC decoding/tx streaming does >>> not read aborted transaction but that would mean locking the transaction >>> every time we give control to output plugin. Given th

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-22 Thread Petr Jelinek
On 22/01/18 20:21, Robert Haas wrote: > On Mon, Jan 22, 2018 at 10:40 AM, Petr Jelinek > wrote: >> I think this is the only real problem from your list for logical >> decoding catalog snapshots. But it's indeed quite a bad one. Is there >> something preventing us to remove the assumption that the

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-22 Thread Robert Haas
On Mon, Jan 22, 2018 at 10:40 AM, Petr Jelinek wrote: > I think this is the only real problem from your list for logical > decoding catalog snapshots. But it's indeed quite a bad one. Is there > something preventing us to remove the assumption that the CTID of T1 is > garbage nobody cares about? I

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-22 Thread Petr Jelinek
Hi, On 20/01/18 00:52, Robert Haas wrote: > On Fri, Jan 19, 2018 at 5:19 PM, Tomas Vondra > wrote: >> Regarding the HOT issue - I have to admit I don't quite see why A2 >> wouldn't be reachable through the index, but that's likely due to my >> limited knowledge of the HOT internals. > > The inde

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Robert Haas
On Fri, Jan 19, 2018 at 5:19 PM, Tomas Vondra wrote: > Regarding the HOT issue - I have to admit I don't quite see why A2 > wouldn't be reachable through the index, but that's likely due to my > limited knowledge of the HOT internals. The index entries only point to the root tuple in the HOT chai

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Tomas Vondra
On 01/19/2018 08:33 PM, Robert Haas wrote: > On Fri, Jan 19, 2018 at 2:16 PM, Tomas Vondra > wrote: >> I think an important piece of this puzzle is that we only really care >> about catalog changes made in a transaction that aborts after doing some >> additional changes, with that catalog tuple

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Robert Haas
On Fri, Jan 19, 2018 at 2:16 PM, Tomas Vondra wrote: > I think an important piece of this puzzle is that we only really care > about catalog changes made in a transaction that aborts after doing some > additional changes, with that catalog tuple in place. Because only then > we actually need that

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Tomas Vondra
On 01/19/2018 06:54 PM, Robert Haas wrote: > On Tue, Dec 26, 2017 at 9:21 AM, Nikhil Sontakke > wrote: >> The main issue here is that HeapTupleSatisfiesVacuum *assumes* that >> rows belonging to an aborted transaction are not visible to anyone >> else. > > One problem here is that if a transacti

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Robert Haas
On Tue, Dec 26, 2017 at 9:21 AM, Nikhil Sontakke wrote: > The main issue here is that HeapTupleSatisfiesVacuum *assumes* that > rows belonging to an aborted transaction are not visible to anyone > else. One problem here is that if a transaction aborts, it might have done so after inserting or upd

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Tomas Vondra
Hi, On 01/19/2018 03:34 PM, Simon Riggs wrote: > On 26 December 2017 at 14:21, Nikhil Sontakke wrote: > >> With logical decoding, there might arise a case that such a row, if it >> belongs to a system catalog table or even a user catalog table >> (https://git.postgresql.org/gitweb/?p=postgresql.

Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2018-01-19 Thread Simon Riggs
On 26 December 2017 at 14:21, Nikhil Sontakke wrote: > With logical decoding, there might arise a case that such a row, if it > belongs to a system catalog table or even a user catalog table > (https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=66abc2608c7c00fcd449e00a9e23f13f02e6

Logical Decoding and HeapTupleSatisfiesVacuum assumptions

2017-12-26 Thread Nikhil Sontakke
Hi, As of today, in the existing code in HeapTupleSatisfiesVacuum, if a row is inserted via an aborted transaction, then it's deemed to be dead immediately and can be cleaned up for re-use by HOT or vacuum, etc. With logical decoding, there might arise a case that such a row, if it belongs to a s