Re: Column Filtering in Logical Replication

2021-12-13 Thread Alvaro Herrera
On 2021-Dec-13, Alvaro Herrera wrote: > Hmm, I messed up the patch file I sent. Here's the complete patch. Actually, this requires even a bit more mess than this to be really complete if we want to be strict about it. The reason is that, with the patch I just posted, we're creati

Re: Column Filtering in Logical Replication

2021-12-14 Thread Alvaro Herrera
On 2021-Dec-13, Alvaro Herrera wrote: > I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in > the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass > value. Looking now to confirm. After working on this a little bit more, I realized that this is a

Re: Column Filtering in Logical Replication

2021-12-14 Thread Alvaro Herrera
On 2021-Dec-14, Tomas Vondra wrote: > Yeah. I think it's not clear if this should behave more like an index or a > view. When an indexed column gets dropped we simply drop the index. But if > you drop a column referenced by a view, we fail with an error. I think we > should handle this more like a

Re: Add id's to various elements in protocol.sgml

2021-12-14 Thread Alvaro Herrera
On 2021-Dec-05, Brar Piening wrote: > When working with the Frontend/Backend Protocol implementation in Npgsql > and discussing things with the team, I often struggle with the fact that > you can't set deep links to individual message formats in the somewhat > lengthy html docs pages. > > The att

Re: Add id's to various elements in protocol.sgml

2021-12-15 Thread Alvaro Herrera
On 2021-Dec-15, Brar Piening wrote: > On Dec 14, 2021 at 20:47, Alvaro Herrera wrote: > > > > Hmm, I think we tend to avoid xreflabels; see > > https://www.postgresql.org/message-id/8315c0ca-7758-8823-fcb6-f37f9413e...@2ndquadrant.com > > Ok, thank you for the h

Re: logical decoding and replication of sequences

2021-12-15 Thread Alvaro Herrera
Looking at 0003, On 2021-Dec-14, Tomas Vondra wrote: > diff --git a/doc/src/sgml/ref/alter_publication.sgml > b/doc/src/sgml/ref/alter_publication.sgml > index bb4ef5e5e22..4d166ad3f9c 100644 > --- a/doc/src/sgml/ref/alter_publication.sgml > +++ b/doc/src/sgml/ref/alter_publication.sgml > @@ -31

Re: Column Filtering in Logical Replication

2021-12-16 Thread Alvaro Herrera
On 2021-Dec-16, houzj.f...@fujitsu.com wrote: > The patch ensures all columns of RT are in column list when CREATE/ALTER > publication, but it seems doesn't prevent user from changing the replica > identity or dropping the index used in replica identity. Do we also need to > check those cases ? Y

Re: Confused comment about drop replica identity index

2021-12-16 Thread Alvaro Herrera
On 2021-Dec-15, Michael Paquier wrote: > On Tue, Dec 14, 2021 at 07:10:49PM +0530, Ashutosh Bapat wrote: > > This code in RelationGetIndexList() is not according to that comment. > > > >if (replident == REPLICA_IDENTITY_DEFAULT && OidIsValid(pkeyIndex)) > > relation->rd_replidindex =

Re: Column Filtering in Logical Replication

2021-12-16 Thread Alvaro Herrera
On 2021-Dec-14, Tomas Vondra wrote: > 7) There's a couple places doing this > > if (att_map != NULL && > !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, >att_map) && > !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, >

Re: row filtering for logical replication

2021-12-16 Thread Alvaro Herrera
Kindly do not change the mode of src/backend/parser/gram.y. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-16 Thread Alvaro Herrera
On 2021-Dec-15, Melanie Plageman wrote: > I noticed after changing the docs on the "bgwriter" target for > pg_stat_reset_shared to say "checkpointer", that it still said "bgwriter" in > src/backend/po/ko.po > src/backend/po/it.po > ... > I presume these are automatically updated with some in

Re: Column Filtering in Logical Replication

2021-12-17 Thread Alvaro Herrera
On 2021-Dec-17, Rahila Syed wrote: > > This means that we need to support changing the column list of a > > table in a publication. I'm looking at implementing some form of > > ALTER PUBLICATION for that. > > I think right now the patch contains support only for ALTER > PUBLICATION.. ADD TABLE w

Re: Column Filtering in Logical Replication

2021-12-17 Thread Alvaro Herrera
So I've been thinking about this as a "security" item (you can see my comments to that effect sprinkled all over this thread), in the sense that if a publication "hides" some column, then the replica just won't get access to it. But in reality that's mistaken: the filtering that this patch impleme

Re: Column Filtering in Logical Replication

2021-12-17 Thread Alvaro Herrera
On 2021-Dec-17, Tomas Vondra wrote: > On 12/17/21 22:07, Alvaro Herrera wrote: > > So I've been thinking about this as a "security" item (you can see my > > comments to that effect sprinkled all over this thread), in the sense > > that if a publication "

Re: Column Filtering in Logical Replication

2021-12-18 Thread Alvaro Herrera
On 2021-Dec-18, Tomas Vondra wrote: > On 12/18/21 02:34, Alvaro Herrera wrote: > > On 2021-Dec-17, Tomas Vondra wrote: > > > > If the server has a *separate* security mechanism to hide the > > > > columns (per-column privs), it is that feature that will protect

Re: Addition of --no-sync to pg_upgrade for test speedup

2021-12-20 Thread Alvaro Herrera
On 2021-Dec-16, Michael Paquier wrote: > In pg_upgrade, we let the flush happen with initdb --sync-only, based > on the binary path of the new cluster, so I think that we are not > going to miss any test coverage by skipping that. There was one patch of mine with breakage that only manifested in

Re: psql format output

2021-12-20 Thread Alvaro Herrera
On 2021-Dec-20, Tom Lane wrote: > -[ RECORD 1 > ]---+--- > Schema | pg_catalog > Name| pg_copy_logical_replication_slot > Result data type| record > Argumen

Re: PublicationActions - use bit flags.

2021-12-20 Thread Alvaro Herrera
On 2021-Dec-20, Peter Eisentraut wrote: > I don't see why this is better. It just makes the code longer and adds more > punctuation and reduces type safety. Removing one palloc is I think the most important consequence ... probably not a big deal though. I think we could change the memcpy calls

Re: minor gripe about lax reloptions parsing for views

2021-12-24 Thread Alvaro Herrera
On 2021-Dec-21, Mark Dilger wrote: > Rebased patch attached: These tests are boringly repetitive. Can't we have something like a nested loop, with AMs on one and reloptions on the other, where each reloption is tried on each AM and an exception block to report the failure or success for each cas

Re: row filtering for logical replication

2021-12-26 Thread Alvaro Herrera
On 2021-Dec-26, Euler Taveira wrote: > On Sat, Dec 25, 2021, at 1:20 AM, Amit Kapila wrote: > > On Fri, Dec 24, 2021 at 11:04 AM Peter Smith wrote: > > > > > > So, IMO the PG docs wording for this part should be relaxed a bit. > > > > > > e.g. > > > BEFORE: > > > + A nullable column in the WHER

Re: Add Boolean node

2021-12-27 Thread Alvaro Herrera
On 2021-Dec-27, Peter Eisentraut wrote: > This patch adds a new node type Boolean, to go alongside the "value" nodes > Integer, Float, String, etc. This seems appropriate given that Boolean > values are a fundamental part of the system and are used a lot. I like the idea. I'm surprised that the

Re: Column Filtering in Logical Replication

2021-12-27 Thread Alvaro Herrera
Determining that an array has a NULL element seems convoluted. I ended up with this query, where comparing the result of array_positions() with an empty array does that. If anybody knows of a simpler way, or any situations in which this fails, I'm all ears. with published_cols as ( selec

Re: Column Filtering in Logical Replication

2021-12-27 Thread Alvaro Herrera
On 2021-Dec-27, Tom Lane wrote: > Alvaro Herrera writes: > > Determining that an array has a NULL element seems convoluted. I ended > > up with this query, where comparing the result of array_positions() with > > an empty array does that. If anybody knows of

Re: Foreign key joins revisited

2021-12-28 Thread Alvaro Herrera
On 2021-Dec-27, Joel Jacobson wrote: > >On Mon, Dec 27, 2021, at 15:48, Isaac Morland wrote: > >I thought the proposal was to give the FK constraint name. > >However, if the idea now is to allow leaving that out also if there > >is only one FK, then that's also OK as long as people understand it

Re: Column Filtering in Logical Replication

2021-12-28 Thread Alvaro Herrera
OK, getting closer now. I've fixed the code to filter them column list during the initial sync, and added some more tests for code that wasn't covered. There are still some XXX comments. The one that bothers me most is the lack of an implementation that allows changing the column list in a publi

Re: Column Filtering in Logical Replication

2021-12-29 Thread Alvaro Herrera
On 2021-Dec-28, Alvaro Herrera wrote: > There are still some XXX comments. The one that bothers me most is the > lack of an implementation that allows changing the column list in a > publication without having to remove the table from the publication > first. OK, I made some prog

Re: Column Filtering in Logical Replication

2021-12-30 Thread Alvaro Herrera
On 2021-Dec-29, Alvaro Herrera wrote: > This new stuff is not yet finished. For example I didn't refactor > handling of REPLICA IDENTITY, so the new command does not correctly > check everything, such as the REPLICA IDENTITY FULL stuff. Also, no > tests have been added yet. I

Re: Column Filtering in Logical Replication

2021-12-30 Thread Alvaro Herrera
Chile — https://www.EnterpriseDB.com/ >From dd2515ee7e0b37f82c76edc4fe890bb7be1abb3e Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Thu, 30 Dec 2021 19:49:31 -0300 Subject: [PATCH v14 1/2] Avoid use of DEFELEM enum in AlterPublicationStmt This allows to add new values for future function

Re: Adding CI to our tree

2021-12-31 Thread Alvaro Herrera
On 2021-Dec-30, Andres Freund wrote: > On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > > I plan to push this after another cycle of tests passing (and driving > > over the bay bridge...) unless I hear protests. > > Pushed. > > Marked CF entry as committed. I tried it using the column filte

Re: Column Filtering in Logical Replication

2022-01-03 Thread Alvaro Herrera
rrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/ "La persona que no quería pecar / estaba obligada a sentarse en duras y empinadas sillas/ desprovistas, por cierto de blandos atenuantes" (Patricio Vogel) >From ad2766290e7011481813ce24c1947bff7041521

Re: Column Filtering in Logical Replication

2022-01-03 Thread Alvaro Herrera
On 2022-Jan-03, Justin Pryzby wrote: > Yes, I know both paths are hit now that it uses GetBool. > > What I'm wondering is why tests didn't fail when one path wasn't hit - when it > said am_partition=DatumGetChar(); if (!am_partition){} Ah! > I suppose it's because the am_partition=true case cor

Re: [PATCH] pg_stat_toast v0.4

2022-01-03 Thread Alvaro Herrera
On 2022-Jan-03, Gunnar "Nick" Bluth wrote: > Am 03.01.22 um 17:50 schrieb Justin Pryzby: > > Soon you will think this is fun :) > > As long as you're happy with plain patches like the attached one, I may ;-) Well, with a zero-byte patch, not very much ... BTW why do you number with a "0." pref

Re: Remove inconsistent quotes from date_part error

2022-01-03 Thread Alvaro Herrera
On 2022-Jan-03, Tom Lane wrote: > Attached v3 restores that distinction, and makes some other small > tweaks. (I found that there were actually a couple of spots in > date.c that got it backwards, so admittedly this is a fine point > that not everybody is on board with. But let's make it consist

Re: [PATCH] pg_stat_toast v0.4

2022-01-03 Thread Alvaro Herrera
On 2022-Jan-03, Gunnar "Nick" Bluth wrote: > 9:38 $ git format-patch PGDG/master -v5 -o .. > ../v5-0001-ping-pong-of-thougths.patch > ../v5-0002-ping-pong-of-thougths.patch > ../v5-0003-adds-some-debugging-messages-in-toast_helper.c.patch > ... Hmm, in such cases I would suggest to create a sepa

Re: [PATCH] pg_stat_toast v6

2022-01-03 Thread Alvaro Herrera
Overall I think this is a good feature to have; assessing the need for compression is important for tuning, so +1 for the goal of the patch. I didn't look into the patch carefully, but here are some minor comments: On 2022-Jan-03, Gunnar "Nick" Bluth wrote: > @@ -229,7 +230,9 @@ toast_tuple_try_

Re: SKIP LOCKED assert triggered

2022-01-03 Thread Alvaro Herrera
On 2021-Dec-01, Simon Riggs wrote: > On Wed, 1 Dec 2021 at 14:33, Bossart, Nathan wrote: > > > > On 11/12/21, 8:56 AM, "Simon Riggs" wrote: > > > The combination of these two statements in a transaction hits an > > > Assert in heapam.c at line 4770 on REL_14_STABLE > > > > I've been unable to re

Re: SKIP LOCKED assert triggered

2022-01-04 Thread Alvaro Herrera
On 2022-Jan-03, Alvaro Herrera wrote: > What I don't understand is why hasn't this been reported already: this > bug is pretty old. My only explanation is that nobody runs sufficiently- > concurrent load with SKIP LOCKED in assert-enabled builds. Pushed, thanks Simon for rep

Re: SKIP LOCKED assert triggered

2022-01-04 Thread Alvaro Herrera
On 2022-Jan-04, Alvaro Herrera wrote: > On 2022-Jan-03, Alvaro Herrera wrote: > > > What I don't understand is why hasn't this been reported already: this > > bug is pretty old. My only explanation is that nobody runs sufficiently- > > concurrent load with SK

Re: row filtering for logical replication

2022-01-05 Thread Alvaro Herrera
BTW I think it's not great to commit with the presented split. We would have non-trivial short-lived changes for no good reason (0002 in particular). I think this whole series should be a single patch, with the commit message being a fusion of messages explaining in full what the functional chang

Re: a misbehavior of partition row movement (?)

2022-01-05 Thread Alvaro Herrera
Pushed 0001. I had to adjust the query used in pg_dump; you changed the attribute name in the query used for pg15, but left unchanged the one for older branches, so pg_dump failed for all branches other than 15. Also, psql's describe.c required a small tweak to a version number test. https://gith

Re: Column Filtering in Logical Replication

2022-01-06 Thread Alvaro Herrera
On 2022-Jan-06, Amit Kapila wrote: > On Mon, Jan 3, 2022 at 8:01 PM Alvaro Herrera wrote: > > > > fetch_remote_table_info() > { > .. > + appendStringInfo(&cmd, > + " SELECT pg_catalog.unnest(prattrs)\n" > +

Re: Column Filtering in Logical Replication

2022-01-06 Thread Alvaro Herrera
On 2022-Jan-06, Amit Kapila wrote: > Considering this, don't we need to deal with "For All Tables" and "For > All Tables In Schema .." Publications in this query? The row filter > patch deal with such cases. The row filter patch handles the NULL case > via C code which makes the query relatively s

Re: a misbehavior of partition row movement (?)

2022-01-06 Thread Alvaro Herrera
On 2022-Jan-06, Amit Langote wrote: > On Thu, Jan 6, 2022 at 7:27 AM Alvaro Herrera wrote: > > I have pushed it thinking that we would not backpatch any of this fix. > > However, after running the tests and realizing that I didn't need an > > initdb for either patch, I

Re: Deduplicate min restart_lsn calculation code

2022-01-06 Thread Alvaro Herrera
On 2022-Jan-06, Bharath Rupireddy wrote: > Hi, > > It seems like the two functions ReplicationSlotsComputeRequiredLSN and > ReplicationSlotsComputeLogicalRestartLSN more or less does the same > thing which makes me optimize (saving 40 LOC) it as attached. I'm > pretty much okay if it gets rejecte

Re: reporting TID/table with corruption error

2022-01-10 Thread Alvaro Herrera
On 2022-Jan-10, Andrey Borodin wrote: > 3. The tuple seems to be updated in a high-contention concurrency > trigger function, autovacuum keeks in ~20-30 seconds after the message > in logs Hmm, I bet this is related. > [ 2022-01-10 09:07:17.671 MSK [unknown],,_s,310759,XX001 ]:ERROR:

Re: Column Filtering in Logical Replication

2022-01-10 Thread Alvaro Herrera
ot; (William Blake) >From e907582e4cd249850dc3784fa4e21b8c1448e99f Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Mon, 6 Sep 2021 10:34:29 -0300 Subject: [PATCH v16] Support column lists for logical replication of tables Add the capability of specifying a column list for individual tables as part of a publicati

Re: Column Filtering in Logical Replication

2022-01-10 Thread Alvaro Herrera
On 2022-Jan-07, Peter Eisentraut wrote: > ATExecReplicaIdentity(): Regarding the question of how to handle > REPLICA_IDENTITY_NOTHING: I see two ways to do this. Right now, the > approach is that the user can set the replica identity freely, and we > decide later based on that what we can replica

Re: a misbehavior of partition row movement (?)

2022-01-11 Thread Alvaro Herrera
On 2022-Jan-11, Amit Langote wrote: > As for the fix to make cross-partition updates work correctly with > foreign keys, I just realized it won't work for the users' existing > foreign keys, because the parent table's triggers that are needed for > the fix to work would not be present. Were you t

Re: Column Filtering in Logical Replication

2022-01-11 Thread Alvaro Herrera
On 2022-Jan-10, Alvaro Herrera wrote: > Hmm. So you're saying that we should only raise errors about the column > list if we are publishing UPDATE or DELETE, but otherwise let the > replica identity be anything. OK, I'll see if I can come up with a > reasonable set of

Re: row filtering for logical replication

2022-01-11 Thread Alvaro Herrera
I just looked at 0002 because of Justin Pryzby's comment in the column filtering thread, and realized that the pgoutput row filtering has a very strange API, which receives both heap tuples and slots; and we seem to convert to and from slots in seemingly unprincipled ways. I don't think this is go

Re: Column Filtering in Logical Replication

2022-01-11 Thread Alvaro Herrera
On 2022-Jan-11, Alvaro Herrera wrote: > On 2022-Jan-10, Alvaro Herrera wrote: > > > Hmm. So you're saying that we should only raise errors about the column > > list if we are publishing UPDATE or DELETE, but otherwise let the > > replica identity be anything. OK, I

Re: Column Filtering in Logical Replication

2022-01-12 Thread Alvaro Herrera
On 2022-Jan-11, Justin Pryzby wrote: > Is there any coordination between the "column filter" patch and the "row > filter" patch ? Not beyond the grammar, which I tested. > Are they both on track for PG15 ? I think they're both on track, yes. > Has anybody run them together ? Not me. > I have

Re: is ErrorResponse possible on Sync?

2022-01-12 Thread Alvaro Herrera
On 2022-Jan-12, Andrei Matei wrote: > If Sync itself cannot fail, then what is this > sentence really saying: "This parameterless message (ed. Sync) causes the > backend to close the current transaction if it's not inside a BEGIN/COMMIT > transaction block (“close” meaning to commit if no error, o

Re: row filtering for logical replication

2022-01-13 Thread Alvaro Herrera
> /* > + * Only 3 publication actions are used for row filtering ("insert", "update", > + * "delete"). See RelationSyncEntry.exprstate[]. > + */ > +typedef enum RowFilterPubAction > +{ > + PUBACTION_INSERT, > + PUBACTION_UPDATE, > + PUBACTION_DELETE, > + NUM_ROWFILTER_PUBACTIONS /

Re: Column Filtering in Logical Replication

2022-01-14 Thread Alvaro Herrera
On 2022-Jan-14, Amit Kapila wrote: > 1. Replica Identity handling: Currently the column filter patch gives > an error during create/alter subscription if the specified column list > is invalid (Replica Identity columns are missing). It also gives an > error if the user tries to change the replica

Re: libpq debug log

2021-03-17 Thread Alvaro Herrera
Hello In pqTraceOutputString(), you can use the return value from fprintf to move the cursor -- no need to count chars. I still think that the message-type specific functions should print the message type, rather than having the string arrays. -- Álvaro Herrera Valdivia, Chile "La gente v

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-17 Thread Alvaro Herrera
On 2021-Mar-15, Alvaro Herrera wrote: > Here's a fixup patch to do it that way. I tried running the commands > you showed and one of them immediately dies with the new error message; > I can't cause the bogus constraint to show up anymore. Actually, that was a silly fix t

Re: Key management with tests

2021-03-18 Thread Alvaro Herrera
Patch 10 uses the term "WAL-skip relations". What does that mean? Is it "relations that are not WAL-logged"? I suppose we already have a term for this; I'm not sure it's a good idea to invent a different term that is only used in this new place. -- Álvaro Herrera39°

Re: Key management with tests

2021-03-18 Thread Alvaro Herrera
On 2021-Mar-18, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > > it "relations that are not WAL-logged"? I suppose we already have a > > term for this;

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-19 Thread Alvaro Herrera
39°49'30"S 73°17'W "Java is clearly an example of money oriented programming" (A. Stepanov) >From ecec91fc4744ddd9a231754afdf10ec09742f8d9 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Fri, 19 Mar 2021 10:52:26 -0300 Subject: [PATCH] Fix for gener

Re: support for MERGE

2021-03-19 Thread Alvaro Herrera
On 2021-Mar-19, David Steele wrote: > Since it does not appear this is being worked on for PG14 perhaps we should > close it in this CF and then reopen it when a patch is available? No, please move it to the next CF. Thanks -- Álvaro Herrera Valdivia, Chile

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Alvaro Herrera
Hmm, if I use configure --with-lz4, I get this: checking whether to build with LZ4 support... yes checking for liblz4... no configure: error: Package requirements (liblz4) were not met: No package 'liblz4' found Consider adjusting the PKG_CONFIG_PATH envir

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Alvaro Herrera
On 2021-Mar-19, Robert Haas wrote: > > Regarding your point, that does look like clutter. We don't annotate > > the dump with a storage clause unless it's non-default, so probably we > > should do the same thing here. I think I gave Dilip bad advice here... > > Here's a patch for that. It's a lit

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Alvaro Herrera
On 2021-Mar-19, Robert Haas wrote: > On Fri, Mar 19, 2021 at 10:11 AM Dilip Kumar wrote: > > Also added a test case for vacuum full to recompress the data. > > I committed the core patch (0003) with a bit more editing. Let's see > what the buildfarm thinks. I updated the coverage script to use

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Alvaro Herrera
On 2021-Mar-19, Robert Haas wrote: > Here's one from me that tries to make the handling of the LZ4 stuff > more like what we already do for zlib, but I'm not sure if it's > correct, or if it's what everyone wants. This one seems to behave as expected (Debian 10, with and without liblz4-dev). --

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Alvaro Herrera
On 2021-Mar-19, Robert Haas wrote: > On Fri, Mar 19, 2021 at 6:22 PM Alvaro Herrera > wrote: > > (At least, for binary upgrade surely you must make sure to apply the > > correct setting regardless of defaults on either system). > > It's not critical from a system

Re: [HACKERS] Custom compression methods (./configure)

2021-03-20 Thread Alvaro Herrera
On 2021-Mar-20, Justin Pryzby wrote: > On Fri, Mar 19, 2021 at 05:35:58PM -0300, Alvaro Herrera wrote: > > Hmm, if I use configure --with-lz4, I get this: > > > > checking whether to build with LZ4 support... yes > > checking for liblz4... no > &

Re: support for MERGE

2021-03-21 Thread Alvaro Herrera
On 2021-Mar-21, Magnus Hagander wrote: > On Fri, Mar 19, 2021 at 6:03 PM Bruce Momjian wrote: > > > > Let's get someone to go into the "database" and adjust it. ;-) > > Yeah, we probably can clean that up on the database side :) Thank you! > But what is it we *want* it to do? That is, what sh

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-21 Thread Alvaro Herrera
On 2021-Mar-21, Justin Pryzby wrote: > On Fri, Mar 19, 2021 at 10:57:37AM -0300, Alvaro Herrera wrote: > > > Also, it "fails to avoid" adding duplicate constraints: > > > > > > Check constraints: > > > "c" CHECK (i IS NOT NULL AND

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-21 Thread Alvaro Herrera
On 2021-Mar-19, Alvaro Herrera wrote: > diff --git a/src/backend/utils/cache/partcache.c > b/src/backend/utils/cache/partcache.c > index 0fe4f55b04..6dfa3fb4a8 100644 > --- a/src/backend/utils/cache/partcache.c > +++ b/src/backend/utils/cache/partcache.c >

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-21 Thread Alvaro Herrera
On 2021-Mar-21, Justin Pryzby wrote: > On Sun, Mar 21, 2021 at 03:01:15PM -0300, Alvaro Herrera wrote: > > > But note that it doesn't check if an existing constraint "implies" the new > > > constraint - maybe it should. > > > > Hm, I'm not

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-21 Thread Alvaro Herrera
On 2021-Mar-21, Justin Pryzby wrote: > On Sun, Mar 21, 2021 at 03:22:00PM -0300, Alvaro Herrera wrote: > > > So if we do that on DETACH, what would happen on ATTACH? > > Do you mean what happens to the constraint that was already there ? > Nothing, since it's not ou

Re: WIP: BRIN multi-range indexes

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-22, Tomas Vondra wrote: > I don't know what's the right fix, but it seems like this patch has > nothing to do with it. If we want to move the opclasses into an > extension, we can comment out that one (cidr/inet) case for now. I don't know what would be a good reason to define the opc

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-23 Thread Alvaro Herrera
So I was about ready to get these patches pushed, when I noticed that in REPEATABLE READ isolation mode it is possible to insert rows violating an FK referencing the partition that is being detached. I'm not sure what is a good solution to this problem. The problem goes like this: /* setup */

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-23, Alvaro Herrera wrote: > So I was about ready to get these patches pushed, when I noticed that in > REPEATABLE READ isolation mode it is possible to insert rows violating > an FK referencing the partition that is being detached. I'm not sure > what is a good

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-22, Bruce Momjian wrote: > --- a/doc/src/sgml/ref/explain.sgml > +++ b/doc/src/sgml/ref/explain.sgml > @@ -136,8 +136,10 @@ ROLLBACK; >the output column list for each node in the plan tree, schema-qualify >table and function names, always label variables in expressions

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-23 Thread Alvaro Herrera
I'm coming around to the idea that the fact that you can cancel the wait phase of DETACH CONCURRENTLY creates quite a disaster, and it's not easy to get away from it. The idea that REPEATABLE READ mode means that you now see detached partitions as if they were in normal condition, is completely at

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-23, James Coleman wrote: > On Tue, Mar 23, 2021 at 1:46 AM Fujii Masao > wrote: > > Therefore for now what we've not reached the consensus is what message > > should be logged at PM_STARTUP. I'm thinking it's better to log > > "the database system is starting up" in that case becaus

Re: WIP: BRIN multi-range indexes

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-23, Tomas Vondra wrote: > FWIW there's yet another difference between the current BRIN opclass > definition, compared to what CREATE OPERATOR CLASS would do. Or more > precisely, how we'd define opfamily for the cross-type cases (integer, > float and timestamp cases). > > AFAICS we do

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-24, Fujii Masao wrote: > On 2021/03/24 1:20, Alvaro Herrera wrote: > > Please note that PM_STARTUP mode is very very short-lived. It only > > starts happening when postmaster launches the startup process, and > > before the startup process begins WAL

Re: [PATCH] pg_permissions

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-23, Joel Jacobson wrote: > On Thu, Mar 11, 2021, at 08:00, Joel Jacobson wrote: > > 0004-pg_permissions-and-pg_ownerships.patch > > Having gotten some hands-on experience of these views for a while, > I notice I quite often want to check the ownerships/permissions > for some specific

Re: [PATCH] pg_permissions

2021-03-23 Thread Alvaro Herrera
On 2021-Mar-08, Joel Jacobson wrote: > $ dropuser test > dropuser: error: removal of role "test" failed: ERROR: role "test" cannot be > dropped because some objects depend on it > DETAIL: 1 object in database joel > > Hmmm. I wonder which 1 object that could be? BTW the easiest way to find ou

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-24, Fujii Masao wrote: > On 2021/03/24 5:59, Tom Lane wrote: > > Alvaro Herrera writes: > > > FATAL: the database system is starting up > > > DETAIL: WAL is being applied to recover from a system crash. > > > or > > > DETAIL: The sy

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-22, Bruce Momjian wrote: > diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat > index e259531f60..9550de0798 100644 > --- a/src/include/catalog/pg_proc.dat > +++ b/src/include/catalog/pg_proc.dat > @@ -5249,9 +5249,9 @@ >proname => 'pg_stat_get_activity'

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-21, Andres Freund wrote: > We currently also fetch the full stats in places like autovacuum.c. Where we > don't need repeated access to be consistent - we even explicitly force the > stats to be re-read for every single table that's getting vacuumed. > > Even if we to just cache alrea

Re: multi-install PostgresNode

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-24, Dagfinn Ilmari Mannsåker wrote: > I think it would be even neater having a method that returns the desired > environment and then have the other methods do: > > local %ENV = $self->_get_install_env(); Hmm, is it possible to integrate PGHOST and PGPORT handling into this too

psql lacking clearerr()

2021-03-24 Thread Alvaro Herrera
psql seems to never call clearerr() on its output file. So if it gets an error while printing a result, it'll show could not print result table: Success after each and every result, even though the output file isn't in error state anymore. It seems that the simplest fix is just to do clearerr()

Re: multi-install PostgresNode

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-24, Andrew Dunstan wrote: > > On 3/24/21 9:23 AM, Andrew Dunstan wrote: > > On 3/24/21 8:29 AM, Alvaro Herrera wrote: > > If we're going to do that we probably shouldn't special case any > > particular settings, but simply take any extra arguments

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-24, Julien Rouhaud wrote: > From e08c9d5fc86ba722844d97000798de868890aba3 Mon Sep 17 00:00:00 2001 > From: Bruce Momjian > Date: Mon, 22 Mar 2021 17:43:23 -0400 > Subject: [PATCH v20 2/3] Expose queryid in pg_stat_activity and > src/backend/executor/execMain.c | 9 ++

Re: multi-install PostgresNode

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-24, Andrew Dunstan wrote: > OK, like this? Yeah, looks good! > +# Private routine to return a copy of the environment with the PATH and > (DY)LD_LIBRARY_PATH > +# correctly set when there is an install path set for the node. > +# Routines that call Postgres binaries need to call thi

Re: PoC/WIP: Extended statistics on expressions

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-24, Justin Pryzby wrote: > On Wed, Mar 24, 2021 at 05:15:46PM +, Dean Rasheed wrote: > > Hmm, I think "univariate" and "multivariate" are pretty ubiquitous, > > when used to describe statistics. You could use "single-column" and > > "multi-column", but then "column" isn't really r

Re: Refactor SSL test framework to support multiple TLS libraries

2021-03-24 Thread Alvaro Herrera
On 2021-Mar-25, Daniel Gustafsson wrote: > Attached is a v2 which addresses the comments raised on the main NSS thread, > as > well as introduces named parameters for the server cert function to make the > test code easier to read. I don't like this patch. I think your SSL::Server::OpenSSL and

Re: [PATCH] pg_permissions

2021-03-25 Thread Alvaro Herrera
On 2021-Mar-25, Joel Jacobson wrote: > pg_shdepend doesn't contain the aclitem info though, > so it won't work for pg_permissions if we want to expose > privilege_type, is_grantable and grantor. Ah, of course -- the only way to obtain the acl columns is by going through the catalogs individually,

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-25 Thread Alvaro Herrera
On 2021-Mar-23, Alvaro Herrera wrote: > I think a possible solution to this problem is that the "detach" flag in > pg_inherits is not a boolean anymore, but an Xid (or maybe two Xids). > Not sure exactly which Xid(s) yet, and I'm not sure what are the exact > rules, b

document lock obtained for FKs during detach

2021-03-25 Thread Alvaro Herrera
logarithmic while I wasn't looking?" http://smylers.hates-software.com/2005/09/08/1995c749.html >From 98c0fe1d22333ea4ec27256f9c4dda98b1166b65 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Thu, 25 Mar 2021 14:53:21 -0300 Subject: [PATCH] Document lock obtained during partitio

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-25 Thread Alvaro Herrera
I added that test as promised, and I couldn't find any problems, so I have pushed it. Thanks! -- Álvaro Herrera Valdivia, Chile

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-25 Thread Alvaro Herrera
On 2020-Nov-30, Justin Pryzby wrote: > On Tue, Nov 03, 2020 at 08:56:06PM -0300, Alvaro Herrera wrote: > > * On the first transaction (the one that marks the partition as > > detached), the partition is locked with ShareLock rather than > > ShareUpdateExclusiveLock. This

Re: [PATCH] pg_permissions

2021-03-26 Thread Alvaro Herrera
On 2021-Mar-26, Joel Jacobson wrote: > On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: > > I wonder what performance will be like with lots o' objects. > > I guess pg_get_acl() would need to be implemented using a switch(classid) > with 36 cases (one for each class)? No, we have a generalized

Re: [PATCH] pg_permissions

2021-03-26 Thread Alvaro Herrera
On 2021-Mar-26, Joel Jacobson wrote: > On Fri, Mar 26, 2021, at 11:30, Alvaro Herrera wrote: > > On 2021-Mar-26, Joel Jacobson wrote: > > > > > On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: > > > > > > I wonder what performance will be like

Re: WIP: BRIN multi-range indexes

2021-03-26 Thread Alvaro Herrera
On 2021-Mar-26, Tomas Vondra wrote: > I've pushed both the bloom and minmax-multi indexes today. Congratulations! I think this reimplementation of the minmax opclass infrastructure makes BRIN much more useful (read: actually usable). -- Álvaro Herrera Valdivia, Chile

<    1   2   3   4   5   6   7   8   9   10   >