On Tue, Oct 29, 2013 at 11:43 AM, Andres Freund wrote:
>> I think modifying GetNewRelFileNode() is attacking the problem from
>> the wrong end. The point is that when a table is dropped, that fact
>> can be communicated to the same machine machinery that's been tracking
>> the CTID->CTID mappings
On 2013-10-29 11:28:44 -0400, Robert Haas wrote:
> On Tue, Oct 29, 2013 at 10:47 AM, Andres Freund
> wrote:
> > On 2013-10-28 11:54:31 -0400, Robert Haas wrote:
> >> > There's one snag I currently can see, namely that we actually need to
> >> > prevent that a formerly dropped relfilenode is getti
On Tue, Oct 29, 2013 at 10:47 AM, Andres Freund wrote:
> On 2013-10-28 11:54:31 -0400, Robert Haas wrote:
>> > There's one snag I currently can see, namely that we actually need to
>> > prevent that a formerly dropped relfilenode is getting reused. Not
>> > entirely sure what the best way for that
On 2013-10-28 11:54:31 -0400, Robert Haas wrote:
> > There's one snag I currently can see, namely that we actually need to
> > prevent that a formerly dropped relfilenode is getting reused. Not
> > entirely sure what the best way for that is.
>
> I'm not sure in detail, but it seems to me that thi
On Mon, Oct 28, 2013 at 12:17 PM, Andres Freund wrote:
>> In general, I don't think waiting on an XID is sufficient because a
>> process can acquire a heavyweight lock without having an XID. Perhaps
>> use the VXID instead?
>
> But decoding doesn't care about transactions that haven't "used" an X
On 2013-10-28 12:04:01 -0400, Robert Haas wrote:
> On Fri, Oct 25, 2013 at 8:14 AM, Andres Freund wrote:
> > I wonder if this is isn't maybe sufficient. Yes, it can deadlock, but
> > that's already the case for VACUUM FULLs of system tables, although less
> > likely. And it will be detected/handl
On Fri, Oct 25, 2013 at 8:14 AM, Andres Freund wrote:
> So, I thought about this for some more and I think I've a partial
> solution to the problem.
>
> The worst thing about deadlocks that occur in the above is that they
> could be the VACUUM FULL waiting for the "restart LSN"[1] of a decoding
>
On Fri, Oct 25, 2013 at 7:57 AM, Andres Freund wrote:
>> However, I'm leery about the idea of using a relation fork for this.
>> I'm not sure whether that's what you had it mind, but it gives me the
>> willies. First, it adds distributed overhead to the system, as
>> previously discussed; and sec
On 2013-10-21 16:15:58 +0200, Andres Freund wrote:
> > I don't think I understand exactly what you have in mind for (2); can
> > you elaborate? I have always thought that having a
> > WaitForDecodingToCatchUp() primitive was a good way of handling
> > changes that were otherwise too difficult to t
On 2013-10-24 10:59:21 -0400, Robert Haas wrote:
> On Tue, Oct 22, 2013 at 2:13 PM, Andres Freund wrote:
> > On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
> >> On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund
> >> wrote:
> >> >> That strikes me as a flaw in the implementation rather than the id
On Tue, Oct 22, 2013 at 2:13 PM, Andres Freund wrote:
> On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
>> On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund
>> wrote:
>> >> That strikes me as a flaw in the implementation rather than the idea.
>> >> You're presupposing a patch where the necessary i
On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
> On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund wrote:
> >> That strikes me as a flaw in the implementation rather than the idea.
> >> You're presupposing a patch where the necessary information is
> >> available in WAL yet you don't make use of it
On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund wrote:
>> It seems to me that you have to think of the CTID map as tied to a
>> relfilenode; if you try to use one relfilenode's map with a different
>> relfilenode, it's obviously not going to work. So don't do that.
>
> It has to be tied to relfile
On 2013-10-22 11:59:35 -0400, Robert Haas wrote:
> >> So I have a new idea for handling this problem, which seems obvious in
> >> retrospect. What if we make the VACUUM FULL or CLUSTER log the old
> >> CTID -> new CTID mappings? This would only need to be done for
> >> catalog tables, and maybe c
On 2013-10-22 19:25:31 +0300, Heikki Linnakangas wrote:
> On 22.10.2013 19:23, Andres Freund wrote:
> >On 2013-10-22 19:19:19 +0300, Heikki Linnakangas wrote:
> >>On 22.10.2013 19:12, Andres Freund wrote:
> >>>On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
> 4) Store both (cmin, cmax) for c
On Tue, Oct 22, 2013 at 12:25 PM, Heikki Linnakangas
wrote:
> Or just hand over a copy of the combocid map to the worker, along with the
> snapshot. Seems a lot simpler than this wide cids business..
Yes, that's what Noah and I talked about doing. Or possibly even
making the map into a hash tabl
On 22.10.2013 19:23, Andres Freund wrote:
On 2013-10-22 19:19:19 +0300, Heikki Linnakangas wrote:
On 22.10.2013 19:12, Andres Freund wrote:
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of deliverin
On 2013-10-22 19:19:19 +0300, Heikki Linnakangas wrote:
> On 22.10.2013 19:12, Andres Freund wrote:
> >On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
> >>4) Store both (cmin, cmax) for catalog tuples.
> >
> >BTW: That would have the nice side-effect of delivering the basis of
> >what you need t
On 22.10.2013 19:12, Andres Freund wrote:
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of delivering the basis of
what you need to do parallel sort in a transaction that previously has
performed DDL.
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
> 4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of delivering the basis of
what you need to do parallel sort in a transaction that previously has
performed DDL.
Currently you cannot do anything in parall
On Tue, Oct 22, 2013 at 11:02 AM, Andres Freund wrote:
> On 2013-10-22 10:52:48 -0400, Robert Haas wrote:
>> On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund
>> wrote:
>> > So. As it turns out that solution isn't sufficient in the face of VACUUM
>> > FULL and mixed DML/DDL transaction that have no
On 2013-10-22 10:52:48 -0400, Robert Haas wrote:
> On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund wrote:
> > So. As it turns out that solution isn't sufficient in the face of VACUUM
> > FULL and mixed DML/DDL transaction that have not yet been decoded.
> >
> > To reiterate, as published it works l
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund wrote:
> So. As it turns out that solution isn't sufficient in the face of VACUUM
> FULL and mixed DML/DDL transaction that have not yet been decoded.
>
> To reiterate, as published it works like:
> For every modification of catalog tuple (insert, mul
Hi,
On 2013-10-21 21:36:02 +0200, Andres Freund wrote:
> I think this approach would have lower maintenance overhead in
> comparison to the previous solution of Handling CommandIds because it's
> actually much simpler.
Btw, I think the new approach would allow for *easier* modifications
about fu
On 2013-10-21 14:50:54 -0400, Robert Haas wrote:
> On Mon, Oct 21, 2013 at 2:27 PM, Andres Freund wrote:
> >> I think it's a complete no-go. Consider, e.g., the comment for
> >> MaxTupleAttributeNumber, which you've blithely falsified. Even if you
> >> update the comment and the value, I'm not i
On Mon, Oct 21, 2013 at 2:27 PM, Andres Freund wrote:
>> I think it's a complete no-go. Consider, e.g., the comment for
>> MaxTupleAttributeNumber, which you've blithely falsified. Even if you
>> update the comment and the value, I'm not inspired by the idea of
>> subtracting 32 from that number
On 2013-10-21 14:22:17 -0400, Robert Haas wrote:
> On Mon, Oct 21, 2013 at 1:52 PM, Andres Freund wrote:
> >> > In my opinion, (4) is too ugly to consider. I think that if we start
> >> > playing games like this, we're opening up the doors to lots of subtle
> >> > bugs and future architectural pa
On Mon, Oct 21, 2013 at 1:52 PM, Andres Freund wrote:
>> > In my opinion, (4) is too ugly to consider. I think that if we start
>> > playing games like this, we're opening up the doors to lots of subtle
>> > bugs and future architectural pain that will be with us for many, many
>> > years to come
On 2013-10-21 09:32:12 -0400, Robert Haas wrote:
> On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund wrote:
> > I know of the following solutions:
> > 1) Don't allow VACUUM FULL on catalog tables if wal_level = logical.
> > 2) Make VACUUM FULL prevent DDL and then wait till all changestreams
> >h
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund wrote:
> I know of the following solutions:
> 1) Don't allow VACUUM FULL on catalog tables if wal_level = logical.
> 2) Make VACUUM FULL prevent DDL and then wait till all changestreams
>have decoded up to the current point.
> 3) don't delete the
On 2013-10-14 09:36:03 -0400, Robert Haas wrote:
> > I thought and implemented that in the beginning. Unfortunately it's not
> > enough :(. That's probably the issue that took me longest to understand
> > in this patchseries...
> >
> > Combocids can only fix the case where a transaction actually ha
On Tue, Oct 15, 2013 at 7:09 AM, Robert Haas wrote:
> Let's try that again.
>
> User: Wow, that sounds great. How do I use it?
> Hacker: Well, currently, the output gets dumped as a series of text
> files that are designed to be parsed using a scripting language. We
> have sample parsers written
On Tue, Oct 15, 2013 at 10:09:05AM -0400, Robert Haas wrote:
> On Tue, Oct 15, 2013 at 9:17 AM, Andres Freund wrote:
> User: So, what's new in PostgreSQL 9.4?
> Hacker: Well, now we have logical replication!
> User: Why is that cool?
> Hacker: Well, streaming replication is awesome for HA, but it
On 10/15/2013 07:56 AM, Andres Freund wrote:
>>> Well, just providing the C API + an example in a first step didn't work
>>> > > out too badly for FDWs. I am pretty sure that once released there will
>>> > > soon be extensions for it on PGXN or whatever for special usecases.
>> >
>> > I suspect so
On Tue, Oct 15, 2013 at 10:48 AM, Andres Freund wrote:
>> > What about columns like:
>> > * action B|I|U|D|C
>>
>> BEGIN and COMMIT?
>
> That's B and C, yes. You'd rather not have them? When would you replay
> the commit without an explicit message telling you to?
No, BEGIN and COMMIT sounds good
On 2013-10-15 11:02:39 -0400, Robert Haas wrote:
> >> If the plugin interface isn't rich enough to provide a convenient way
> >> to avoid that, then it needs to be fixed so that it is, because it
> >> will be a common requirement.
> >
> > Oh, it surely is possibly to avoid repeating it. The output
On Tue, Oct 15, 2013 at 11:02:39AM -0400, Robert Haas wrote:
> >> goals may be in conflict; we'll have to pick something.
> >
> > Note that parsing COPYs is a major PITA from most languages...
> >
> > Perhaps we should make the default output json instead? With every
> > action terminated by a null
On Tue, Oct 15, 2013 at 10:56 AM, Andres Freund wrote:
>> > That means you allow trivial remote code execution since you could try
>> > to load system() or something else that's available in every shared
>> > object. Now you can argue that that's OK since we have special checks
>> > for replicatio
On 2013-10-15 10:34:53 -0400, Robert Haas wrote:
> On Tue, Oct 15, 2013 at 10:27 AM, Andres Freund
> wrote:
> >> I think part of the problem may be that you're using the library name
> >> to identify the output plugin. I'm not excited about that design.
> >> For functions, you give the function
On 2013-10-15 10:15:14 -0400, Robert Haas wrote:
> On Tue, Oct 15, 2013 at 9:47 AM, Andres Freund wrote:
> > On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
> >> If we go for CSV I think we should put the entire primary key as one
> >> column (containing all the columns) and the entire row anot
On Tue, Oct 15, 2013 at 10:27 AM, Andres Freund wrote:
>> I think part of the problem may be that you're using the library name
>> to identify the output plugin. I'm not excited about that design.
>> For functions, you give the function a name and that is a pointer to
>> where to actually find th
On 2013-10-15 10:09:05 -0400, Robert Haas wrote:
> On Tue, Oct 15, 2013 at 9:17 AM, Andres Freund wrote:
> > It allows you to use the shared libary both as a normal extension loaded
> > via shared_preload_library or adhoc and as an output plugin which seems
> > like a sensible goal.
> > We could h
On 2013-10-15 10:20:55 -0400, Robert Haas wrote:
> > For multi-master / conflict resolution you may also want all old
> > values to make sure that they have not changed on target.
>
> The patch as proposed doesn't make that information available. If you
> want that to be an option, now would be t
On Tue, Oct 15, 2013 at 10:09 AM, Hannu Krosing wrote:
>> I don't see how you can fail to know what "all" is.
> We instinctively know what "all" is - as in the famous case of buddhist
> ordering a
> hamburger - "Make me All wit Everything" :) - but the requirements of
> different replications sys
On Tue, Oct 15, 2013 at 9:47 AM, Andres Freund wrote:
> On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
>> If we go for CSV I think we should put the entire primary key as one
>> column (containing all the columns) and the entire row another.
>
> What about columns like:
> * action B|I|U|D|C
B
On 10/15/2013 02:47 PM, Andres Freund wrote:
> On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
>> If we go for CSV I think we should put the entire primary key as one
>> column (containing all the columns) and the entire row another.
just use JSON :)
>> What about columns like:
>> * action B|I|U
On 10/15/2013 01:42 PM, Robert Haas wrote:
> On Mon, Oct 14, 2013 at 9:51 AM, Andres Freund wrote:
>>> Well, I just think relying on specific symbol names in the .so file is
>>> kind of unfortunate. It means that, for example, you can't have
>>> multiple output plugins provided by a single .so.
On Tue, Oct 15, 2013 at 9:17 AM, Andres Freund wrote:
> It allows you to use the shared libary both as a normal extension loaded
> via shared_preload_library or adhoc and as an output plugin which seems
> like a sensible goal.
> We could have a single _PG_init_output_plugin() symbol that fills in
On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
> If we go for CSV I think we should put the entire primary key as one
> column (containing all the columns) and the entire row another.
What about columns like:
* action B|I|U|D|C
* xid
* timestamp
* tablename
* key name
* key column names
* k
On 2013-10-15 08:49:26 -0400, Robert Haas wrote:
> On Mon, Oct 14, 2013 at 5:07 PM, Andres Freund wrote:
> > So, see the attatched benchmark skript. I've always done using a disk
> > bound and a memory bound (using eatmydata, preventing fsyncs) run.
> >
> > * unpatched run, wal_level = hot_standby
On 2013-10-15 08:42:20 -0400, Robert Haas wrote:
> On Mon, Oct 14, 2013 at 9:51 AM, Andres Freund wrote:
> >> Well, I just think relying on specific symbol names in the .so file is
> >> kind of unfortunate. It means that, for example, you can't have
> >> multiple output plugins provided by a sing
On Mon, Oct 14, 2013 at 5:07 PM, Andres Freund wrote:
> So, see the attatched benchmark skript. I've always done using a disk
> bound and a memory bound (using eatmydata, preventing fsyncs) run.
>
> * unpatched run, wal_level = hot_standby, eatmydata
> * unpatched run, wal_level = hot_standby
>
>
On Mon, Oct 14, 2013 at 9:51 AM, Andres Freund wrote:
>> Well, I just think relying on specific symbol names in the .so file is
>> kind of unfortunate. It means that, for example, you can't have
>> multiple output plugins provided by a single .so. And in general I
>> think it's something that we
On 2013-10-14 15:51:14 +0200, Andres Freund wrote:
> > > It'd probably not hurt to redo those benchmarks to make sure...
> >
> > Yes, I think it would be good to characterize it more precisely than
> > "a bit", so people know what to expect.
>
> A "bit" was below the 3% range for loops of adding
On 2013-10-14 09:36:03 -0400, Robert Haas wrote:
> On Fri, Oct 11, 2013 at 12:57 PM, Andres Freund
> wrote:
> >> I don't see any need for SQL syntax. I was just thinking that the
> >> _PG_init function could fill in a structure and then call
> >> RegisterLogicalReplicationOutputPlugin(&mystruct)
On Fri, Oct 11, 2013 at 12:57 PM, Andres Freund wrote:
>> I don't see any need for SQL syntax. I was just thinking that the
>> _PG_init function could fill in a structure and then call
>> RegisterLogicalReplicationOutputPlugin(&mystruct).
>
> Hm. We can do that, but what'd be the advantage of tha
Hi,
On 2013-10-11 09:08:43 -0400, Robert Haas wrote:
> On Thu, Oct 10, 2013 at 7:11 PM, Andres Freund wrote:
> > On 2013-10-09 14:49:46 -0400, Robert Haas wrote:
> >> I spent some time looking at the sample plugin (patch 9/12). Here are
> >> some review comments:
> >>
> >> - I think that the dec
On Thu, Oct 10, 2013 at 7:11 PM, Andres Freund wrote:
> Hi Robert,
>
> On 2013-10-09 14:49:46 -0400, Robert Haas wrote:
>> I spent some time looking at the sample plugin (patch 9/12). Here are
>> some review comments:
>>
>> - I think that the decoding plugin interface should work more like the
>>
Hi Robert,
On 2013-10-09 14:49:46 -0400, Robert Haas wrote:
> I spent some time looking at the sample plugin (patch 9/12). Here are
> some review comments:
>
> - I think that the decoding plugin interface should work more like the
> foreign data wrapper interface. Instead of using pg_dlsym to l
On Mon, Sep 30, 2013 at 6:44 PM, Andres Freund wrote:
> The series from friday was a bit too buggy - obviously I was too
> tired. So here's a new one:
>
> * fix pg_recvlogical makefile (Thanks Steve)
> * fix two commits not compiling properly without later changes (Thanks Kevin)
> * keep track of
On 2013-10-07 09:56:11 -0400, Steve Singer wrote:
> On 10/03/2013 04:00 PM, Andres Freund wrote:
> >Ok, there were a couple of bugs because I thought mxacts wouldn't need to
> >be supported. So far your testcase doesn't crash the database anymore - it
> >spews some internal errors though, so I am n
On 10/03/2013 04:00 PM, Andres Freund wrote:
Ok, there were a couple of bugs because I thought mxacts wouldn't need
to be supported. So far your testcase doesn't crash the database
anymore - it spews some internal errors though, so I am not sure if
it's entirely fixed for you. Thanks for testin
On 2013-10-03 13:03:07 -0400, Steve Singer wrote:
> On 10/03/2013 12:38 PM, Andres Freund wrote:
> >Does your code use SELECT FOR UPDATE/SHARE on system or treat_as_catalog
> >tables? Greetings, Andres Freund
>
> Yes.
> It declares sl_table and sl_sequence and sl_set as catalog.
>
> It does a
> S
On 10/03/2013 12:38 PM, Andres Freund wrote:
Does your code use SELECT FOR UPDATE/SHARE on system or
treat_as_catalog tables? Greetings, Andres Freund
Yes.
It declares sl_table and sl_sequence and sl_set as catalog.
It does a
SELECT ..
from @NAMESPACE@.sl_table T, @NAMESPACE@.sl_set S
On 2013-10-01 16:11:47 -0400, Steve Singer wrote:
> On 09/30/2013 06:44 PM, Andres Freund wrote:
> >Hi,
> >
> >The series from friday was a bit too buggy - obviously I was too
> >tired. So here's a new one:
>
> With this series I've also noticed
> #2 0x007741a7 in ExceptionalCondition (
>
On 09/30/2013 06:44 PM, Andres Freund wrote:
Hi,
The series from friday was a bit too buggy - obviously I was too
tired. So here's a new one:
With this series I've also noticed
#2 0x007741a7 in ExceptionalCondition (
conditionName=conditionName@entry=0x7c2908 "!(!(tuple->t_infomas
66 matches
Mail list logo
