Re: Assertion failure with replication origins and PREPARE TRANSACTIOn

2021-12-13 Thread Michael Paquier
On Mon, Dec 13, 2021 at 04:30:36PM +0900, Masahiko Sawada wrote: > Why do we check if replorigin_session_origin_lsn is not invalid data > only when PREPARE TRANSACTION? Well, it does not matter for the case of PREPARE TRANSACTION, does it? we would include values for the the origin LSN and timesta

WIN32 pg_import_system_collations

2021-12-13 Thread Juan José Santamaría Flecha
I want to propose an implementation of pg_import_system_collations() for WIN32 using EnumSystemLocalesEx() [1], which is available from Windows Server 2008 onwards. The patch includes a test emulating that of collate.linux.utf8, but for Windows-1252. The main difference is that it doesn't have the

Re: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread Amit Kapila
On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com wrote: > Few questions and comments: 1. The pg_stat_subscription_workers view will contain one row per subscription worker on which errors have occurred, for workers applying logical replication changes an

Re: Remove pg_strtouint64(), use strtoull() directly

2021-12-13 Thread Peter Eisentraut
On 10.12.21 16:25, Tom Lane wrote: Our experience with the variable size of "long" has left a sufficiently bad taste in my mouth that I'm not enthused about adding hard-wired assumptions that "long long" is identical to int64. So this seems like it's going in the wrong direction, and giving up p

Re: Optionally automatically disable logical replication subscriptions on error

2021-12-13 Thread vignesh C
On Mon, Dec 6, 2021 at 4:22 PM osumi.takami...@fujitsu.com wrote: > > On Monday, December 6, 2021 1:16 PM Greg Nancarrow > wrote: > > On Sat, Dec 4, 2021 at 12:20 AM osumi.takami...@fujitsu.com > > wrote: > > > > > > Hi, I've made a new patch v11 that incorporated suggestions described > > abov

Re: Assertion failure with replication origins and PREPARE TRANSACTIOn

2021-12-13 Thread Amit Kapila
On Mon, Dec 13, 2021 at 2:00 PM Michael Paquier wrote: > > On Mon, Dec 13, 2021 at 04:30:36PM +0900, Masahiko Sawada wrote: > > Why do we check if replorigin_session_origin_lsn is not invalid data > > only when PREPARE TRANSACTION? > > Well, it does not matter for the case of PREPARE TRANSACTION,

Re: isolationtester: add session name to application name

2021-12-13 Thread Michael Paquier
On Fri, Dec 10, 2021 at 05:20:52PM -0800, Andres Freund wrote: > These days isolationtester.c already prefixes log output with the session > name. How about doing the same for application_name? It's a *tad* more > complicated than I'd like because isolationtester.c currently doesn't know the > name

Re: Assertion failure with replication origins and PREPARE TRANSACTIOn

2021-12-13 Thread Michael Paquier
On Mon, Dec 13, 2021 at 03:46:55PM +0530, Amit Kapila wrote: > This is my understanding as well. I think here the point of Sawada-San > is why to have additional for replorigin_session_origin_lsn in prepare > code path? I think the way you have it in your patch is correct as > well but it is probab

Re: Question about 001_stream_rep.pl recovery test

2021-12-13 Thread Michael Paquier
On Fri, Dec 10, 2021 at 01:44:40PM -0800, David Zhang wrote: > Inside the test script `src/test/recovery/t/001_stream_rep.pl`, a comment at > line 30 says `my_backup` is "not mandatory", > >  30 # Take backup of standby 1 (not mandatory, but useful to check if >  31 # pg_basebackup works on a stan

RE: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread osumi.takami...@fujitsu.com
On Monday, December 13, 2021 6:19 PM Amit Kapila wrote: > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > wrote: > > Few questions and comments: Thank you for your comments ! > > 1. > The pg_stat_subscription_workers view will > contain > one row per s

Re: Probable memory leak with ECPG and AIX

2021-12-13 Thread talk to ben
Hi, (I work with Guillaume on this case.) On Sun, Dec 12, 2021 at 8:34 AM Noah Misch wrote: > That almost certainly means he's using a 32-bit binary with the default > heap > size. To use more heap on AIX, build 64-bit or override the heap size. > For > example, "env LDR_CNTRL=MAXDATA=0x80

Re: using extended statistics to improve join estimates

2021-12-13 Thread Tomas Vondra
On 11/22/21 02:23, Justin Pryzby wrote: Your regression tests include two errors, which appear to be accidental, and fixing the error shows that this case is being estimated poorly. +-- try combining with single-column (and single-expression) statistics +DROP STATISTICS join_test_2; +ERROR: sta

Re: [PATCH] pg_stat_toast

2021-12-13 Thread Gunnar "Nick" Bluth
Am 13.12.21 um 00:41 schrieb Andres Freund: Hi, On 2021-12-13 00:00:23 +0100, Gunnar "Nick" Bluth wrote: Regarding stats size; it adds one PgStat_BackendToastEntry (PgStat_BackendAttrIdentifier + PgStat_ToastCounts, should be 56-64 bytes or something in that ballpark) per TOASTable attribute, I

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Masahiko Sawada
On Mon, Dec 13, 2021 at 1:04 PM Amit Kapila wrote: > > On Mon, Dec 13, 2021 at 8:28 AM Masahiko Sawada wrote: > > > > On Sat, Dec 11, 2021 at 3:29 PM Amit Kapila wrote: > > > > > > 3. > > > + * Also, we don't skip receiving the changes in streaming cases, > > > since we decide > > > + * whether

Re: daitch_mokotoff module

2021-12-13 Thread Dag Lem
Please find attached an updated patch, with the following fixes: * Replaced remaining malloc/free with palloc/pfree. * Made "make check" pass. * Updated notes on other implementations. Best regards Dag Lem diff --git a/contrib/Makefile b/contrib/Makefile index 87bf87ab90..5ea729 100644 ---

Re: extended stats on partitioned tables

2021-12-13 Thread Justin Pryzby
On Sun, Dec 12, 2021 at 10:29:39PM +0100, Tomas Vondra wrote: > > In your 0003 patch, the "if inh: break" isn't removed from > > examine_variable(), > > but the corresponding thing is removed everywhere else. > > Ah, you're right. And it wasn't updated in the 0002 patch either - it > should do th

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Robert Haas
On Fri, Dec 10, 2021 at 2:03 PM Bossart, Nathan wrote: > Well, I haven't had a chance to look at your patch, and my patch set > still only has handling for CheckPointSnapBuild() and > RemovePgTempFiles(), but I thought I'd share what I have anyway. I > split it into 5 patches: > > 0001 - Adds a n

Re: extended stats on partitioned tables

2021-12-13 Thread Justin Pryzby
On Sun, Dec 12, 2021 at 11:23:19PM +0100, Tomas Vondra wrote: > On 12/12/21 22:32, Justin Pryzby wrote: > > On Sun, Dec 12, 2021 at 05:17:10AM +0100, Tomas Vondra wrote: > > > The one thing bugging me a bit is that the regression test checks only a > > > GROUP BY query. It'd be nice to add queries

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-13 Thread Robert Haas
On Thu, Dec 9, 2021 at 2:32 AM Maciek Sakrejda wrote: > > Considering the vanishingly small number of actual complaints we've > > seen about this, that sounds ridiculously over-engineered. > > A documentation example should be sufficient. > > I don't know if this will tip the scales, but I'd like

Re: WIP: WAL prefetch (another approach)

2021-12-13 Thread Robert Haas
On Fri, Nov 26, 2021 at 9:47 PM Tom Lane wrote: > Yeah ... on the one hand, that machine has shown signs of > hard-to-reproduce flakiness, so it's easy to write off the failures > I saw as hardware issues. On the other hand, the flakiness I've > seen has otherwise manifested as kernel crashes, wh

Re: daitch_mokotoff module

2021-12-13 Thread Tomas Vondra
On 12/13/21 14:38, Dag Lem wrote: Please find attached an updated patch, with the following fixes: * Replaced remaining malloc/free with palloc/pfree. * Made "make check" pass. * Updated notes on other implementations. Thanks, looks interesting. A couple generic comments, based on a quick co

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-13 Thread Dilip Kumar
On Tue, Dec 7, 2021 at 11:11 AM Justin Pryzby wrote: > > You added this to pg_stat_activity, which already has a lot of fields. > We talked a few months ago about not adding more fields that weren't commonly > used. > https://www.postgresql.org/message-id/flat/20210426191811.sp3o77doinphyjhu%40al

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-13 Thread Dilip Kumar
On Mon, Dec 13, 2021 at 8:34 AM Ashutosh Sharma wrote: > > + /* > +* If the relation is from the default tablespace then we need to > +* create it in the destinations db's default tablespace. Otherwise, > +* we need to create in the same tablespace as it is in the so

Re: Multi-Column List Partitioning

2021-12-13 Thread Ashutosh Sharma
Hi, Is this okay? postgres=# CREATE TABLE t1 (a int, b int) PARTITION BY LIST ( a, a, a ); CREATE TABLE postgres=# CREATE TABLE t1_1 PARTITION OF t1 FOR VALUES IN ((1, 2, 3), (4, 5, 6)); CREATE TABLE postgres=# \d t1 Partitioned table "public.t1" Column | Type | Collation | Nulla

Re: Documenting when to retry on serialization failure

2021-12-13 Thread Robert Haas
On Thu, Dec 9, 2021 at 7:43 AM Simon Riggs wrote: > I had a conversation with Kevin Grittner about retry some years back > and it seemed clear that the application should re-execute application > logic from the beginning, rather than just slavishly re-execute the > same SQL. But that is not docume

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2021-12-13 Thread Shruthi Gowda
On Mon, Dec 6, 2021 at 10:14 AM Sadhuprasad Patro wrote: > > On Tue, Oct 26, 2021 at 6:55 PM Shruthi Gowda wrote: > > > > > > I have revised the patch w.r.t the way 'create_storage' is interpreted > > in heap_create() along with some minor changes to preserve the DBOID > > patch. > > > > Hi Shrut

Re: Remove pg_strtouint64(), use strtoull() directly

2021-12-13 Thread Tom Lane
Peter Eisentraut writes: > On 10.12.21 16:25, Tom Lane wrote: >> Our experience with the variable size of "long" has left a sufficiently >> bad taste in my mouth that I'm not enthused about adding hard-wired >> assumptions that "long long" is identical to int64. So this seems like >> it's going i

Re: should we enable log_checkpoints out of the box?

2021-12-13 Thread Robert Haas
On Thu, Dec 9, 2021 at 11:02 AM Bharath Rupireddy wrote: > May I know if the v2 patch (attached at [1] in this thread) is still > of interest? > > CF entry is here - https://commitfest.postgresql.org/36/3401/ > > [1] - > https://www.postgresql.org/message-id/CALj2ACU9cK4pCzcqvey71F57PTPsdxtUGmfUn

Re: daitch_mokotoff module

2021-12-13 Thread Andrew Dunstan
On 12/13/21 09:26, Tomas Vondra wrote: > On 12/13/21 14:38, Dag Lem wrote: >> Please find attached an updated patch, with the following fixes: >> >> * Replaced remaining malloc/free with palloc/pfree. >> * Made "make check" pass. >> * Updated notes on other implementations. >> > > Thanks, looks i

Re: RecoveryInProgress() has critical side effects

2021-12-13 Thread Robert Haas
On Tue, Dec 7, 2021 at 5:55 PM Robert Haas wrote: > On Sat, Dec 4, 2021 at 7:44 PM Michael Paquier wrote: > > My main worry here is that this changes slightly the definition of > > doPageWrites across stable branches at the end of recovery as there > > could be records generated there. Note that

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2021-12-13 Thread Shruthi Gowda
On Mon, Dec 6, 2021 at 11:25 PM Robert Haas wrote: > > On Sun, Dec 5, 2021 at 11:44 PM Sadhuprasad Patro wrote: > > 3. > > @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > > */ > > pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); > > > > - do >

Re: Remove pg_strtouint64(), use strtoull() directly

2021-12-13 Thread Robert Haas
On Mon, Dec 13, 2021 at 9:44 AM Tom Lane wrote: > Yeah, exactly. That seems like a natural evolution: > short -> 2 bytes > int -> 4 bytes > long -> 8 bytes > long long -> 16 bytes > so I'm surprised that vendors haven't done that already instead > of inventing hack

Re: speed up verifying UTF-8

2021-12-13 Thread John Naylor
On Fri, Dec 10, 2021 at 2:33 PM Heikki Linnakangas wrote: > I had another look at this now. Looks good, just a few minor comments below: Thanks for reviewing! I've attached v25 to address your points. > This function assumes that the input len is a multiple of 8. There's an > assertion for that

Re: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread vignesh C
On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com wrote: > > On Monday, December 6, 2021 11:27 PM vignesh C wrote: > > Thanks for the updated patch, few comments: > Thank you for your review ! > > > 1) We can keep the documentation similar to mention the count includes both > > table syn

Re: Remove pg_strtouint64(), use strtoull() directly

2021-12-13 Thread Tom Lane
Robert Haas writes: > I really am glad they haven't. I think it's super-annoying that we > need hacks like UINT64_FORMAT all over the place. I think it was a > mistake not to nail down the size that each type is expected to be in > the original C standard, Well, mumble. One must remember that wh

Re: [PATCH]Comment improvement in publication.sql

2021-12-13 Thread vignesh C
On Wed, Dec 8, 2021 at 11:07 AM tanghy.f...@fujitsu.com wrote: > > On Wednesday, December 8, 2021 1:49 PM, vignesh C wrote: > > > The patch no longer applies, could you post a rebased patch. > > Thanks for your kindly reminder. Attached a rebased patch. > Some changes in v4 patch has been fixed b

Re: Remove pg_strtouint64(), use strtoull() directly

2021-12-13 Thread Robert Haas
On Mon, Dec 13, 2021 at 10:46 AM Tom Lane wrote: > Robert Haas writes: > > I really am glad they haven't. I think it's super-annoying that we > > need hacks like UINT64_FORMAT all over the place. I think it was a > > mistake not to nail down the size that each type is expected to be in > > the or

speed up text_position() for utf-8

2021-12-13 Thread John Naylor
Hi, Commit 9556aa01c69 (Use single-byte Boyer-Moore-Horspool search even with multibyte encodings), was a speed improvement for the majority of cases, but when the match was toward the end of the string, the slowdown in text_position_get_match_pos() was noticeable. It was found that there was a lo

Re: daitch_mokotoff module

2021-12-13 Thread Tomas Vondra
On 12/13/21 16:05, Andrew Dunstan wrote: On 12/13/21 09:26, Tomas Vondra wrote: On 12/13/21 14:38, Dag Lem wrote: Please find attached an updated patch, with the following fixes: * Replaced remaining malloc/free with palloc/pfree. * Made "make check" pass. * Updated notes on other implementat

Re: WIN32 pg_import_system_collations

2021-12-13 Thread Juan José Santamaría Flecha
On Mon, Dec 13, 2021 at 9:41 AM Juan José Santamaría Flecha < juanjo.santama...@gmail.com> wrote: Per path tester. > Regards, > > Juan José Santamaría Flecha > v2-0001-WIN32-pg_import_system_collations.patch Description: Binary data

conchuela has some SSL issues

2021-12-13 Thread Robert Haas
Hi, The last three buildfarm runs on conchuela show a failure in initdb: Shared object "libssl.so.48" not found, required by "libldap_r-2.4.so.2" It seems likely to me that this is a machine configuration issue rather than the result of some recent change in PostgreSQL, because the first failure

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Justin Pryzby
On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote: > On Fri, Dec 10, 2021 at 2:03 PM Bossart, Nathan wrote: > > Well, I haven't had a chance to look at your patch, and my patch set > > still only has handling for CheckPointSnapBuild() and > > RemovePgTempFiles(), but I thought I'd share

Re: pg_dump versus ancient server versions

2021-12-13 Thread Tom Lane
I wrote: > Anyway, it seems like there's some consensus that 9.2 is a good > stopping place for today. I'll push forward with > (1) back-patching as necessary to make 9.2 and up build cleanly > on the platforms I have handy; I've done as much as I plan to do in that direction. As of the respecti

Re: pg_dump versus ancient server versions

2021-12-13 Thread Robert Haas
On Mon, Dec 13, 2021 at 12:23 PM Tom Lane wrote: > I've done as much as I plan to do in that direction. As of the > respective branch tips, I see clean builds and check-world > results with minimal configure options in all branches back to 9.2 > on Fedora 35 (gcc 11.2.1) and macOS Monterey (Apple

Re: Column Filtering in Logical Replication

2021-12-13 Thread Alvaro Herrera
On 2021-Dec-10, Alvaro Herrera wrote: > Actually it's not so easy to implement. So I needed to add "sub object id" support for pg_publication_rel objects in pg_depend / dependency.c. What I have now is partial (the describe routines need patched) but it's sufficient to show what's needed. In es

Re: Column Filtering in Logical Replication

2021-12-13 Thread Alvaro Herrera
Hmm, I messed up the patch file I sent. Here's the complete patch. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "Doing what he did amounts to sticking his fingers under the hood of the implementation; if he gets his fingers burnt, it's his problem." (Tom Lan

Re: Add spin_delay() implementation for Arm in s_lock.h

2021-12-13 Thread Blake, Geoff
Hi Tom, > What did you test exactly? Tested 3 benchmark configurations on an m6g.16xlarge (Graviton2, 64 cpus, 256GB RAM) I set the scale factor to consume about 1/3 of 256GB and the other parameters in the next line. pgbench setup: -F 90 -s 5622 -c 256 Pgbench select-only w/ patch 662804 tp

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 5:54 AM, "Robert Haas" wrote: > I don't know whether this kind of idea is good or not. Thanks for chiming in. I have an almost-complete patch set that I'm hoping to post to the lists in the next couple of days. > One thing we've seen a number of times now is that entrusting the sam

Re: using extended statistics to improve join estimates

2021-12-13 Thread Tomas Vondra
On 11/6/21 11:03, Andy Fan wrote: Hi Tomas: This is the exact patch I want, thanks for the patch! Good to hear. On Thu, Oct 7, 2021 at 3:33 AM Tomas Vondra wrote: 3) estimation by join pairs At the moment, the estimates are calculated for pairs of relations, so for example given a quer

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 9:20 AM, "Justin Pryzby" wrote: > On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote: >> Another issue is that we don't want to increase the number of >> processes without bound. Processes use memory and CPU resources and if >> we run too many of them it becomes a burden on t

Re: isolationtester: add session name to application name

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 19:46:34 +0900, Michael Paquier wrote: > On Fri, Dec 10, 2021 at 05:20:52PM -0800, Andres Freund wrote: > > These days isolationtester.c already prefixes log output with the session > > name. How about doing the same for application_name? It's a *tad* more > > complicated than I'

Re: Add client connection check during the execution of the query

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 17:51:00 +1300, Thomas Munro wrote: > On Sat, Dec 11, 2021 at 7:09 PM Thomas Munro wrote: > > On Sat, Dec 11, 2021 at 6:11 PM Andres Freund wrote: > > > Yuck. Is there really no better way to deal with this? What kind of > > > errors is > > > this trying to handle transparentl

Re: isolationtester: add session name to application name

2021-12-13 Thread Tom Lane
Andres Freund writes: > On 2021-12-13 19:46:34 +0900, Michael Paquier wrote: >> +1 for the idea. Maybe it could be backpatched? > Not entirely trivially - the changes have some dependencies on other changes > (e.g. b1907d688, more on 741d7f104, but that was backpatched). I guess we > could backp

Re: Commitfest 2021-11 Patch Triage - Part 3

2021-12-13 Thread Mark Dilger
> On Nov 11, 2021, at 2:12 PM, Daniel Gustafsson wrote: > > 3223: Delegating superuser tasks to new security roles > == > There is a lot of ongoing discussion on this one, but skimming the thread it's > not really clear (to me) where it leave

Re: port conflicts when running tests concurrently on windows.

2021-12-13 Thread Andres Freund
Hi, On 2021-12-10 10:22:13 +0100, Peter Eisentraut wrote: > On 09.12.21 19:41, Andres Freund wrote: > > Withhttps://github.com/anarazel/postgres/commit/046203741803da863f6129739fd215f8a32ec357 > > all tests pass. pg_regress requires PG_REGRESS_SOCK_DIR because it checks > > for > > TMPDIR, but wi

Re: bugfix: invalid bit/varbit input causes the log file to be unreadable

2021-12-13 Thread Huansong Fu
Hi, We recently saw a similar issue in v12 and wondered why the corresponding fix for v14 (https://github.com/postgres/postgres/commit/16e3ad5d143) was not backported to v13 and before. The commit message did mention that this fix might have problem with translatable string messages - would yo

Re: extended stats on partitioned tables

2021-12-13 Thread Tomas Vondra
On 12/13/21 14:48, Justin Pryzby wrote: > On Sun, Dec 12, 2021 at 10:29:39PM +0100, Tomas Vondra wrote: >>> In your 0003 patch, the "if inh: break" isn't removed from >>> examine_variable(), >>> but the corresponding thing is removed everywhere else. >> >> Ah, you're right. And it wasn't updated i

Re: conchuela has some SSL issues

2021-12-13 Thread Andrew Dunstan
On 12/13/21 11:48, Robert Haas wrote: > Hi, > > The last three buildfarm runs on conchuela show a failure in initdb: > > Shared object "libssl.so.48" not found, required by "libldap_r-2.4.so.2" > > It seems likely to me that this is a machine configuration issue > rather than the result of some r

Re: isolationtester: add session name to application name

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 13:57:52 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2021-12-13 19:46:34 +0900, Michael Paquier wrote: > >> +1 for the idea. Maybe it could be backpatched? > > > Not entirely trivially - the changes have some dependencies on other changes > > (e.g. b1907d688, more on

Re: bugfix: invalid bit/varbit input causes the log file to be unreadable

2021-12-13 Thread Tom Lane
Huansong Fu writes: > We recently saw a similar issue in v12 and wondered why the corresponding fix > for v14 (https://github.com/postgres/postgres/commit/16e3ad5d143) was not > backported to v13 and before. The commit message did mention that this fix > might have problem with translatable str

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Robert Haas
On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan wrote: > > But against all that, if these tasks are slowing down checkpoints and > > that's avoidable, that seems pretty important too. Interestingly, I > > can't say that I've ever seen any of these things be a problem for > > checkpoint or startup

Re: extended stats on partitioned tables

2021-12-13 Thread Tomas Vondra
On 12/13/21 14:53, Justin Pryzby wrote: > On Sun, Dec 12, 2021 at 11:23:19PM +0100, Tomas Vondra wrote: >> On 12/12/21 22:32, Justin Pryzby wrote: >>> On Sun, Dec 12, 2021 at 05:17:10AM +0100, Tomas Vondra wrote: The one thing bugging me a bit is that the regression test checks only a GRO

Re: WIN32 pg_import_system_collations

2021-12-13 Thread Thomas Munro
On Tue, Dec 14, 2021 at 5:29 AM Juan José Santamaría Flecha wrote: > On Mon, Dec 13, 2021 at 9:41 AM Juan José Santamaría Flecha > wrote: > Per path tester. Hi Juan José, I haven't tested yet but +1 for the feature. I guess the API didn't exist at the time collation support was added. +/

Re: Granting SET and ALTER SYSTE privileges for GUCs

2021-12-13 Thread Andrew Dunstan
On 11/23/21 21:14, Mark Dilger wrote: > >> On Nov 23, 2021, at 8:07 AM, Robert Haas wrote: >> >> It's my impression that information_schema is a child of the SQL >> standard, and that inventions specific to PG go in pg_catalog. >> >> Also, I think the user-facing name for GUCs is "settings". > T

Re: Granting SET and ALTER SYSTE privileges for GUCs

2021-12-13 Thread Mark Dilger
> On Dec 13, 2021, at 12:56 PM, Andrew Dunstan wrote: > > This patch had bit-rotted slightly, and I was attempting to remedy it. I have that already, and getting ready to post. Give me a few minutes and I'll repost. > However, I got a failure running the TAP tests because of this change: >

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2021-12-13 Thread Robert Haas
On Mon, Dec 13, 2021 at 9:40 AM Shruthi Gowda wrote: > > I am reviewing another patch > > "v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg" as well > > and will provide the comments soon if any... I spent much of today reviewing 0001. Here's an updated version, so far only lightly teste

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
Hi, Attached is an updated version of the CI patches. An example of a test run with the attached version of this https://cirrus-ci.com/build/6501998521483264 I again included the commit allowing crash dumps to be collected on windows, but I don't think it can be merged as-is, and should be left f

Re: Column Filtering in Logical Replication

2021-12-13 Thread Alvaro Herrera
On 2021-Dec-13, Alvaro Herrera wrote: > Hmm, I messed up the patch file I sent. Here's the complete patch. Actually, this requires even a bit more mess than this to be really complete if we want to be strict about it. The reason is that, with the patch I just posted, we're creating a new type o

Re: Adding CI to our tree

2021-12-13 Thread Justin Pryzby
On Mon, Dec 13, 2021 at 01:12:23PM -0800, Andres Freund wrote: > Hi, > > Attached is an updated version of the CI patches. An example of a test run > with the attached version of this > https://cirrus-ci.com/build/6501998521483264 sudo is used exactly twice; maybe it's not needed at all ? > +tas

Re: Add client connection check during the execution of the query

2021-12-13 Thread Thomas Munro
On Tue, Dec 14, 2021 at 7:53 AM Andres Freund wrote: > On 2021-12-13 17:51:00 +1300, Thomas Munro wrote: > > I tried that. It seems OK, and gets rid of the PG_FINALLY(), which is > > nice. Latches still have higher priority, and still have the fast > > return if already set and you asked for onl

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 6:30 AM, "Dilip Kumar" wrote: > On Tue, Dec 7, 2021 at 11:11 AM Justin Pryzby wrote: >> Since I think this field is usually not interesting to most users of >> pg_stat_activity, maybe this should instead be implemented as a function like >> pg_backend_get_subxact_status(pid). >> >> P

Re: make tuplestore helper function

2021-12-13 Thread Melanie Plageman
On Thu, Nov 18, 2021 at 1:24 PM Justin Pryzby wrote: > > On Thu, Nov 18, 2021 at 12:59:03PM -0500, Melanie Plageman wrote: > > On Mon, Nov 8, 2021 at 3:13 PM Justin Pryzby wrote: > > > On Mon, Nov 08, 2021 at 02:52:28PM -0500, Melanie Plageman wrote: > > > > On Tue, Nov 2, 2021 at 4:23 PM Justin

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 16:02:50 -0600, Justin Pryzby wrote: > On Mon, Dec 13, 2021 at 01:12:23PM -0800, Andres Freund wrote: > > Hi, > > > > Attached is an updated version of the CI patches. An example of a test run > > with the attached version of this > > https://cirrus-ci.com/build/6501998521483264

Re: row filtering for logical replication

2021-12-13 Thread Peter Smith
On Thu, Dec 9, 2021 at 1:37 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, December 8, 2021 7:52 PM Ajin Cherian > > On Tue, Dec 7, 2021 at 5:36 PM Peter Smith wrote: > > > > > > We were mid-way putting together the next v45* when your latest > > > attachment was posted over the weekend. So

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 12:37 PM, "Robert Haas" wrote: > On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan wrote: >> > But against all that, if these tasks are slowing down checkpoints and >> > that's avoidable, that seems pretty important too. Interestingly, I >> > can't say that I've ever seen any of these

Re: row filtering for logical replication

2021-12-13 Thread Peter Smith
On Tue, Dec 7, 2021 at 5:48 PM tanghy.f...@fujitsu.com wrote: > ... > Thanks for looking into it. > > I have another problem with your patch. The document says: > > ... If the subscription has several publications in > + which the same table has been published with different filters, those > +

Re: Adding CI to our tree

2021-12-13 Thread Tom Lane
Andres Freund writes: > On 2021-12-13 16:02:50 -0600, Justin Pryzby wrote: >> sudo is used exactly twice; maybe it's not needed at all ? > The macos one is needed, but the freebsd one indeed isn't. I'm with Justin on this one. I would view a script trying to mess with /cores as a hostile act.

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 18:14:52 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2021-12-13 16:02:50 -0600, Justin Pryzby wrote: > >> sudo is used exactly twice; maybe it's not needed at all ? > > > The macos one is needed, but the freebsd one indeed isn't. > > I'm with Justin on this one. I wou

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2021-12-13 Thread Andres Freund
Hi, On 2021-10-02 23:34:38 -0400, Tom Lane wrote: > Andrew Dunstan writes: > > On 10/2/21 5:03 PM, Tom Lane wrote: > >> IIUC, the only problem for a non-updated animal would be that it'd > >> run the test twice? Or would it actually fail? If the latter, > >> we'd need to sit on the patch rather

RE: parallel vacuum comments

2021-12-13 Thread tanghy.f...@fujitsu.com
On Monday, December 13, 2021 2:12 PM Masahiko Sawada wrote: > > On Mon, Dec 13, 2021 at 2:09 PM Amit Kapila wrote: > > > > On Mon, Dec 13, 2021 at 10:33 AM Masahiko Sawada > wrote: > > > > > > On Fri, Dec 10, 2021 at 9:08 PM Amit Kapila > wrote: > > > > > > > > On Thu, Dec 9, 2021 at 6:05 PM

Re: Assertion failure with replication origins and PREPARE TRANSACTIOn

2021-12-13 Thread Michael Paquier
On Mon, Dec 13, 2021 at 07:53:43PM +0900, Michael Paquier wrote: > Well, I don't think that it is a big deal one way or the other, as > we'd finish with InvalidXLogRecPtr for the LSN and 0 for the timestamp > anyway. If both of you feel that just removing the assertion rather > than adding an extr

RE: parallel vacuum comments

2021-12-13 Thread houzj.f...@fujitsu.com
On Tuesday, December 14, 2021 10:11 AM Tang, Haiying wrote: > On Monday, December 13, 2021 2:12 PM Masahiko Sawada > wrote: > > I've attached the patch. I've just moved some functions back but not > > done other changes. > > > > Thanks for your patch. > > I tested your patch and tried some cases

Re: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread Amit Kapila
On Mon, Dec 13, 2021 at 5:48 PM osumi.takami...@fujitsu.com wrote: > > On Monday, December 13, 2021 6:19 PM Amit Kapila > wrote: > > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > > wrote: > > > > Few questions and comments: > Thank you for your comments ! > > > ==

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Amit Kapila
On Mon, Dec 13, 2021 at 6:55 PM Masahiko Sawada wrote: > > On Mon, Dec 13, 2021 at 1:04 PM Amit Kapila wrote: > > > > On Mon, Dec 13, 2021 at 8:28 AM Masahiko Sawada > > wrote: > > > > > > > > > > 4. > > > > + * Also, one might think that we can skip preparing the skipped > > > > transaction.

Re: parallel vacuum comments

2021-12-13 Thread Amit Kapila
On Tue, Dec 14, 2021 at 7:40 AM tanghy.f...@fujitsu.com wrote: > > On Monday, December 13, 2021 2:12 PM Masahiko Sawada > wrote: > > > > On Mon, Dec 13, 2021 at 2:09 PM Amit Kapila wrote: > > > > > > On Mon, Dec 13, 2021 at 10:33 AM Masahiko Sawada > > wrote: > > > > > > > > On Fri, Dec 10, 20

Re: Adding CI to our tree

2021-12-13 Thread Thomas Munro
On Tue, Dec 14, 2021 at 10:12 AM Andres Freund wrote: > Attached is an updated version of the CI patches. An example of a test run > with the attached version of this > https://cirrus-ci.com/build/6501998521483264 I've been pushing various versions of these patches into my own development branche

more descriptive message for process termination due to max_slot_wal_keep_size

2021-12-13 Thread Kyotaro Horiguchi
Hello. As complained in pgsql-bugs [1], when a process is terminated due to max_slot_wal_keep_size, the related messages don't mention the root cause for *the termination*. Note that the third message does not show for temporary replication slots. [pid=a] LOG: terminating process x to release r

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
Hi, On 2021-12-14 16:51:58 +1300, Thomas Munro wrote: > I'd better go and figure out how to fix cfbot when this lands... I assume it'd be: - stop adding the CI stuff - adjust links to CI tasks, appveyor wouldn't be used anymore - perhaps reference individual tasks from the cfbot page? > > I thi

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread vignesh C
On Fri, Dec 10, 2021 at 11:14 AM Masahiko Sawada wrote: > > On Thu, Dec 9, 2021 at 6:16 PM Amit Kapila wrote: > > > > On Thu, Dec 9, 2021 at 2:24 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Dec 9, 2021 at 11:47 AM Amit Kapila > > > wrote: > > > > > > > > I am thinking that we can start

Re: row filtering for logical replication

2021-12-13 Thread Amit Kapila
On Tue, Dec 14, 2021 at 4:44 AM Peter Smith wrote: > > On Tue, Dec 7, 2021 at 5:48 PM tanghy.f...@fujitsu.com > wrote: > > > > I think for "FOR ALL TABLE" publication(p2 in my case), table tbl should be > > treated as no filter, and table tbl should have no filter in subscription > > sub. Though

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2021-12-13 Thread Michael Paquier
On Mon, Dec 13, 2021 at 06:08:24PM -0800, Andres Freund wrote: > Seems like we might get away with making make -C contrib/pg_upgrade check and > vcregress.pl upgradecheck do nothing? You mean #contrib/#src/bin/# here, right? I don't think that we have any need to have "make -C" do nothing. For v

RE: Optionally automatically disable logical replication subscriptions on error

2021-12-13 Thread osumi.takami...@fujitsu.com
On Monday, December 13, 2021 6:57 PM vignesh C wrote: > On Mon, Dec 6, 2021 at 4:22 PM osumi.takami...@fujitsu.com > wrote: > > > > I've attached the new version v12. I appreciate your review. > Thanks for the updated patch, few comments: > 1) This is not required as it is not used in the calle

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Greg Nancarrow
On Tue, Dec 14, 2021 at 3:23 PM vignesh C wrote: > > While the worker is skipping one of the skip transactions specified by > the user and immediately if the user specifies another skip > transaction while the skipping of the transaction is in progress this > new value will be reset by the worker

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Dilip Kumar
On Fri, Dec 3, 2021 at 12:12 PM Masahiko Sawada wrote: > > Skipping a whole transaction by specifying xid would be a good start. > Ideally, we'd like to automatically skip only operations within the > transaction that fail but it seems not easy to achieve. If we allow > specifying operations and/o

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2021-12-13 Thread Andres Freund
On 2021-12-14 14:31:24 +0900, Michael Paquier wrote: > On Mon, Dec 13, 2021 at 06:08:24PM -0800, Andres Freund wrote: > > Seems like we might get away with making make -C contrib/pg_upgrade check > > and > > vcregress.pl upgradecheck do nothing? > > You mean #contrib/#src/bin/# here, right? I do

Re: Commitfest 2021-11 Patch Triage - Part 3

2021-12-13 Thread Jeff Davis
On Mon, 2021-12-13 at 11:20 -0800, Mark Dilger wrote: > - The logical replication subscription patch needs to consider RLS. > (Jeff Davis) > > This is implemented but I still need to update the documentation > before > posting. We also discussed how much of the insert path we want t

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Dilip Kumar
On Tue, Dec 14, 2021 at 8:20 AM Amit Kapila wrote: > > On Mon, Dec 13, 2021 at 6:55 PM Masahiko Sawada wrote: > > In streaming cases, we don’t know when stream-commit or stream-abort > > comes and another conflict could occur on the subscription in the > > meanwhile. But given that (we expect) t

RE: pg_get_publication_tables() output duplicate relid

2021-12-13 Thread houzj.f...@fujitsu.com
On Sat, Nov 20, 2021 7:31 PM Amit Kapila wrote: > On Fri, Nov 19, 2021 at 10:58 AM Amit Kapila > wrote: > > > > On Fri, Nov 19, 2021 at 7:19 AM Amit Langote > wrote: > > > > > > The problematic case is attaching the partition *after* the > > > subscriber has already marked the root parent as syn