Re: Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-03-14 Thread Peter Eisentraut
On 13.03.23 14:19, Daniel Gustafsson wrote: On 2 Mar 2023, at 15:44, Tom Lane wrote: Peter Eisentraut writes: I think an error message like "unexpected null value in system cache %d column %d" is sufficient. Since these are "can't happen" errors, we don't need to spend too much extra ef

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread Peter Smith
Thanks for the review! On Mon, Mar 13, 2023 at 6:19 PM vignesh C wrote: ... > Few comments: > 1) Can we move the macros along with the other macros present in this > file, just above this structure, similar to the macros added for > txn_flags: > /* Toplevel transaction for this subxact (N

Re: ICU locale validation / canonicalization

2023-03-14 Thread Peter Eisentraut
On 13.03.23 16:31, Jeff Davis wrote: What we had discussed a while ago in one of these threads is that ICU before version 54 do not support keyword lists, and we have custom code to do that parsing ourselves, but we don't have code to do the same for language tags.  Therefore, if I understand thi

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread Önder Kalacı
Hi Shi Yu, > in RemoteRelContainsLeftMostColumnOnIdx(): > > + if (indexInfo->ii_NumIndexAttrs < 1) > + return false; > > Did you see any cases that the condition is true? I think there is at > least one > column in the index. If so, we can use an Assert(). > Actually, it was

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread Önder Kalacı
Hi Amit, all Amit Kapila , 14 Mar 2023 Sal, 09:50 tarihinde şunu yazdı: > On Mon, Mar 13, 2023 at 7:46 PM Önder Kalacı > wrote: > > > > Attaching v47. > > > > I have made the following changes in the attached patch (a) removed > the function IsIdxSafeToSkipDuplicates() and used the check direct

Re: ICU 54 and earlier are too dangerous

2023-03-14 Thread Peter Eisentraut
On 14.03.23 01:26, Tom Lane wrote: Unless someone has a better idea, I think we need to bump the minimum required ICU version to 55. That would solve the issue in v16 and later, but those using old versions of ICU and old versions of postgres would still be vulnerable to these kinds of typos. ..

Re: logical decoding and replication of sequences, take 2

2023-03-14 Thread John Naylor
I tried a couple toy examples with various combinations of use styles. Three with "automatic" reading from sequences: create table test(i serial); create table test(i int GENERATED BY DEFAULT AS IDENTITY); create table test(i int default nextval('s1')); ...where s1 has some non-default parameter

Re: [PATCH] Add CANONICAL option to xmlserialize

2023-03-14 Thread Jim Jones
v4 attached fixes an encoding issue at the xml_parse call. It now uses GetDatabaseEncoding(). Best, Jim From 3ff8e7bd9a9e43194d834ba6b125841539d5df1c Mon Sep 17 00:00:00 2001 From: Jim Jones Date: Mon, 6 Mar 2023 14:08:54 +0100 Subject: [PATCH v4] Add CANONICAL format to xmlserialize This patc

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-03-14 Thread Bharath Rupireddy
On Tue, Mar 7, 2023 at 11:14 PM Nathan Bossart wrote: > > On Tue, Mar 07, 2023 at 12:39:13PM +0530, Bharath Rupireddy wrote: > > On Tue, Mar 7, 2023 at 3:30 AM Nathan Bossart > > wrote: > >> Is it possible to memcpy more than a page at a time? > > > > It would complicate things a lot there; the

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread Amit Kapila
On Tue, Mar 14, 2023 at 12:48 PM Önder Kalacı wrote: >> >> 2. >> +# make sure that the subscriber has the correct data after the update UPDATE >> >> "update UPDATE" seems to be a typo. >> > > thanks, fixed > >> >> 3. >> +# now, drop the index with the expression, and re-create index on column >>

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread Önder Kalacı
Amit Kapila , 14 Mar 2023 Sal, 11:59 tarihinde şunu yazdı: > On Tue, Mar 14, 2023 at 12:48 PM Önder Kalacı > wrote: > >> > >> 2. > >> +# make sure that the subscriber has the correct data after the update > UPDATE > >> > >> "update UPDATE" seems to be a typo. > >> > > > > thanks, fixed > > > >> >

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread vignesh C
On Tue, 14 Mar 2023 at 14:36, Önder Kalacı wrote: > > > Amit Kapila , 14 Mar 2023 Sal, 11:59 tarihinde şunu > yazdı: >> >> On Tue, Mar 14, 2023 at 12:48 PM Önder Kalacı wrote: >> >> >> >> 2. >> >> +# make sure that the subscriber has the correct data after the update >> >> UPDATE >> >> >> >> "u

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread Önder Kalacı
> > > > Thanks for the updated patch. > Few minor comments: > 1) The extra line break after IsIndexOnlyOnExpression function can be > removed: > removed > > > 2) Generally we don't terminate with "." for single line comments > + > + /* > + * Simple case, we already have a primary key or a replic

Re: [Proposal] Allow pg_dump to include all child tables with the root table

2023-03-14 Thread Stéphane Tachoires
Hi Gilles, V5 is ok (aside for LLVM 14 deprecated warnings, but that's another problem) with meson compile and meson test on Ubuntu 20.04.2. Code fits well and looks standart, --help explain what it does clearly, and documentation is ok (but as a Français, I'm not an expert in English). Stéphane

Re: [Proposal] Allow pg_dump to include all child tables with the root table

2023-03-14 Thread stephane tachoires
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed V5 is ok (aside for LLVM 14 deprecated warnings, but that's a

Re: lz4 --rm on Ubuntu 18.04 (Add LZ4 compression to pg_dump)

2023-03-14 Thread Christoph Berg
Re: Tomas Vondra > and I don't think there's a good place to inject the 'rm' so I ended up > adding a 'cleanup_cmd' right after 'compress_cmd'. But it seems a bit > strange / hacky. Maybe there's a better way? Does the file need to be removed at all? Could we leave it there and have "make clean" r

Re: [Proposal] Allow pg_dump to include all child tables with the root table

2023-03-14 Thread Gilles Darold
Le 14/03/2023 à 10:50, stephane tachoires a écrit : The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed V5 is o

Re: Allow logical replication to copy tables in binary format

2023-03-14 Thread Melih Mutlu
Hi, Attached v13. Peter Smith , 14 Mar 2023 Sal, 03:07 tarihinde şunu yazdı: > Here are some review comments for patch v12-0001 > Thanks for reviewing. I tried to make explanations in docs better according to your comments. What do you think? Amit Kapila , 14 Mar 2023 Sal, 06:17 tarihinde şun

Re: Add LZ4 compression in pg_dump

2023-03-14 Thread gkokolatos
--- Original Message --- On Monday, March 13th, 2023 at 10:47 PM, Tomas Vondra wrote: > > > Change pg_fatal() to an assertion+comment; > > > Yeah, that's reasonable. I'd even ditch the assert/comment, TBH. We > could add such protections against "impossible" stuff to a zillion ot

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-14 Thread John Naylor
I wrote: > > > Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this

Re: pg_dump/pg_restore: Fix stdin/stdout handling of custom format on Win32

2023-03-14 Thread Daniel Watzinger
I'm sorry I couldn't contribute to the discussion in time. The fix of the fstat() Win32 port looks good to me. I agree that there's a need for multiple fseek() ports to address the shortcomings of the MSVC functionality. The documentation event states that "on devices incapable of seeking, the ret

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread vignesh C
On Tue, 14 Mar 2023 at 12:37, Peter Smith wrote: > > Thanks for the review! > > On Mon, Mar 13, 2023 at 6:19 PM vignesh C wrote: > ... > > Few comments: > > 1) Can we move the macros along with the other macros present in this > > file, just above this structure, similar to the macros added for >

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread Amit Kapila
On Tue, Mar 14, 2023 at 12:37 PM Peter Smith wrote: > > Thanks for the review! > > On Mon, Mar 13, 2023 at 6:19 PM vignesh C wrote: > ... > > > 4) We check if txn->toptxn is not null twice here both in if condition > > and in the assignment, we could retain the assignment operation as > > earlier

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread Amit Kapila
On Tue, Mar 14, 2023 at 5:03 PM vignesh C wrote: > > On Tue, 14 Mar 2023 at 12:37, Peter Smith wrote: > > > > The same issue exists here too: > 1) > - if (toptxn != NULL && !rbtxn_has_catalog_changes(toptxn)) > + if (rbtxn_is_subtxn(txn)) > { > - toptxn->txn_flag

Fix fseek() detection of unseekable files on WIN32

2023-03-14 Thread Juan José Santamaría Flecha
Hello all, As highlighted in [1] fseek() might fail to error even when accessing unseekable streams. PFA a patch that checks the file type before the actual fseek(), so only supported calls are made. [1] https://www.postgresql.org/message-id/flat/b1448cd7-871e-20e3-8398-895e2d1d3...@gmail.com R

Re: pg_dump/pg_restore: Fix stdin/stdout handling of custom format on Win32

2023-03-14 Thread Juan José Santamaría Flecha
Please, don't top post. On Tue, Mar 14, 2023 at 12:30 PM Daniel Watzinger < daniel.watzin...@gmail.com> wrote: > I'm sorry I couldn't contribute to the discussion in time. The fix of the > fstat() Win32 port looks good to me. I agree that there's a need for > multiple fseek() ports to address the

Re: [PATCH] Add pretty-printed XML output option

2023-03-14 Thread Jim Jones
On 09.03.23 21:21, Tom Lane wrote: Peter Smith writes: The patch v19 LGTM. Another thing that's mildly irking me is that the current factorization of this code will result in xml_parse'ing the data twice, if you have both DOCUMENT and INDENT specified. We could consider avoiding that if we me

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-03-14 Thread Tomas Vondra
On 3/8/23 23:31, Matthias van de Meent wrote: > On Wed, 22 Feb 2023 at 14:14, Matthias van de Meent > wrote: >> >> On Wed, 22 Feb 2023 at 13:15, Tomas Vondra >> wrote: >>> >>> On 2/20/23 19:15, Matthias van de Meent wrote: Thanks. Based on feedback, attached is v2 of the patch, with as >>>

Re: meson: Non-feature feature options

2023-03-14 Thread Nazir Bilal Yavuz
Hi, On Wed, 22 Feb 2023 at 12:14, Peter Eisentraut wrote: > > On 21.02.23 17:32, Nazir Bilal Yavuz wrote: > >>> I like the second approach, with a 'uuid' feature option. As you wrote > >>> earlier, adding an 'auto' choice to a combo option doesn't work fully > >>> like a > >>> real feature opti

Re: lz4 --rm on Ubuntu 18.04 (Add LZ4 compression to pg_dump)

2023-03-14 Thread Tomas Vondra
On 3/14/23 11:34, Christoph Berg wrote: > Re: Tomas Vondra >> and I don't think there's a good place to inject the 'rm' so I ended up >> adding a 'cleanup_cmd' right after 'compress_cmd'. But it seems a bit >> strange / hacky. Maybe there's a better way? > > Does the file need to be removed at all

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-03-14 Thread Justin Pryzby
On Mon, Mar 13, 2023 at 10:42:59AM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Sun, Mar 12, 2023 at 06:25:13PM -0400, Tom Lane wrote: > >> I agree that adding such a field to IndexStmt would be a very bad idea. > >> However, adding another parameter to DefineIndex doesn't seem like a > >

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-03-14 Thread Matthias van de Meent
On Tue, 14 Mar 2023 at 14:49, Tomas Vondra wrote: > > > > On 3/8/23 23:31, Matthias van de Meent wrote: > > On Wed, 22 Feb 2023 at 14:14, Matthias van de Meent > > > > I think that the v4 patch solves all comments up to now; and > > considering that most of this patch was committed but then revert

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-03-14 Thread Tom Lane
Justin Pryzby writes: > The idea would be for: > 1) TOTAL to show the number of direct and indirect leaf partitions; > 2) update progress while building direct or indirect indexes; > 3) ATTACHing intermediate partitioned tables to increment by 0; > 4) ATTACHing a direct child should continue to in

Re: doc: mentioned CREATE+ATTACH PARTITION as an alternative to CREATE TABLE..PARTITION OF

2023-03-14 Thread Jakub Wartak
Hi, I've tested the attached patch by Justin and it applied almost cleanly to the master, but there was a tiny typo and make postgres-A4.pdf didn't want to run: Note that creating a partition using PARTITION OF => (note lack of closing literal) => Note that creating a partition using PARTITION OF

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-03-14 Thread Ilya Gladyshev
> 14 марта 2023 г., в 18:34, Justin Pryzby написал(а): > > On Mon, Mar 13, 2023 at 10:42:59AM -0400, Tom Lane wrote: >> Justin Pryzby writes: >>> On Sun, Mar 12, 2023 at 06:25:13PM -0400, Tom Lane wrote: I agree that adding such a field to IndexStmt would be a very bad idea. However

Re: Add LZ4 compression in pg_dump

2023-03-14 Thread gkokolatos
--- Original Message --- On Monday, March 13th, 2023 at 9:21 PM, Tomas Vondra wrote: > > > > > On 3/11/23 11:50, gkokola...@pm.me wrote: > > > --- Original Message --- > > On Saturday, March 11th, 2023 at 7:00 AM, Alexander Lakhin > > exclus...@gmail.com wrote: > > > > >

RE: Allow logical replication to copy tables in binary format

2023-03-14 Thread Takamichi Osumi (Fujitsu)
On Tuesday, March 14, 2023 8:02 PM Melih Mutlu wrote: > Attached v13. Hi, Thanks for sharing v13. Few minor review comments. (1) create_subscription.sgml + column types as opposed to text format. Even when this option is enabled, + only data types having binary send and recei

Re: Add LZ4 compression in pg_dump

2023-03-14 Thread Tomas Vondra
On 3/14/23 16:18, gkokola...@pm.me wrote: > ...> Would you mind me trying to come with a patch to address your points? > That'd be great, thanks. Please keep it split into smaller patches - two might work, with one patch for "cosmetic" changes and the other tweaking the API error-handling stuf

Re: ICU 54 and earlier are too dangerous

2023-03-14 Thread Jeff Davis
On Mon, 2023-03-13 at 18:13 -0700, Andres Freund wrote: > What non-error code is returned in the above example? When the collator for locale "asdf" is opened, the status is set to U_USING_DEFAULT_WARNING. That seemed very promising at first, but it's the same thing returned after opening most val

Re: Add LZ4 compression in pg_dump

2023-03-14 Thread Tomas Vondra
On 3/14/23 12:07, gkokola...@pm.me wrote: > > > --- Original Message --- > On Monday, March 13th, 2023 at 10:47 PM, Tomas Vondra > wrote: > > > >> >>> Change pg_fatal() to an assertion+comment; >> >> >> Yeah, that's reasonable. I'd even ditch the assert/comment, TBH. We >> could add

Re: ICU locale validation / canonicalization

2023-03-14 Thread Jeff Davis
On Tue, 2023-03-14 at 08:08 +0100, Peter Eisentraut wrote: > Another issue that came to mind:  Right now, you can, say, develop > SQL > schemas on a newer ICU version, say, your laptop, and then deploy > them > on a server running an older ICU version.  If we have a cutoff beyond > which we conve

Re: [PATCH] Add pretty-printed XML output option

2023-03-14 Thread Tom Lane
Jim Jones writes: > [ v22-0001-Add-pretty-printed-XML-output-option.patch ] I poked at this for awhile and ran into a problem that I'm not sure how to solve: it misbehaves for input with embedded DOCTYPE. regression=# SELECT xmlserialize(DOCUMENT '' as text indent); xmlserialize --

DROP DATABASE is interruptible

2023-03-14 Thread Andres Freund
Hi, Unfortunately DROP DATABASE does not hold interrupt over its crucial steps. If you e.g. set a breakpoint on DropDatabaseBuffers() and then do a signal SIGINT, we'll process that interrupt before the transaction commits. A later connect to that database ends with: 2023-03-14 10:22:24.443 PDT

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2023-03-14 Thread Gregory Stark (as CFM)
It looks like this needs a big rebase in fea-uth.c fe-auth-scram.c and fe-connect.c. Every hunk is failing which perhaps means the code you're patching has been moved or refactored?

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2023-03-14 Thread Tom Lane
"Gregory Stark (as CFM)" writes: > It looks like this needs a big rebase in fea-uth.c fe-auth-scram.c and > fe-connect.c. Every hunk is failing which perhaps means the code > you're patching has been moved or refactored? The cfbot is giving up after v14-0001-libpq-Run-pgindent-after-a9e9a9f32b3.p

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-03-14 Thread Gregory Stark (as CFM)
It does look like a rebase for meson.build would be helpful. I'm not marking it waiting on author just for meson.build conflicts but it would be perhaps more likely to be picked up by a committer if it's showing green in cfbot.

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2023-03-14 Thread Greg Stark
On Tue, 14 Mar 2023 at 13:59, Tom Lane wrote: > > "Gregory Stark (as CFM)" writes: > > It looks like this needs a big rebase in fea-uth.c fe-auth-scram.c and > > fe-connect.c. Every hunk is failing which perhaps means the code > > you're patching has been moved or refactored? > > The cfbot is giv

Re: [EXTERNAL] Re: Support load balancing in libpq

2023-03-14 Thread Gregory Stark (as CFM)
The pgindent run in b6dfee28f is causing this patch to need a rebase for the cfbot to apply it.

Re: pg_stat_statements and "IN" conditions

2023-03-14 Thread Gregory Stark (as CFM)
So I was seeing that this patch needs a rebase according to cfbot. However it looks like the review feedback you're looking for is more of design questions. What jumbling is best to include in the feature set and which is best to add in later patches. It sounds like you've gotten conflicting feedb

Re: Using each rel as both outer and inner for JOIN_ANTI

2023-03-14 Thread Gregory Stark (as CFM)
So what is the status of this patch? It looks like you received some feedback from Emre, Tom, Ronan, and Alvaro but it also looks like you responded to most or all of that. Are you still blocked waiting for feedback? Anything specific you need help with? Or is the patch ready for commit now? In w

Re: Privileges on PUBLICATION

2023-03-14 Thread Gregory Stark (as CFM)
FYI this looks like it needs a rebase due to a conflict in copy.c and an offset in pgoutput.c. Is there anything specific that still needs review or do you think you've handled all Peter's concerns? In particular, is there "a comprehensive description of what it is trying to do"? :)

Re: Sampling-based timing for EXPLAIN ANALYZE

2023-03-14 Thread Greg Stark
On Tue, 17 Jan 2023 at 14:52, Tomas Vondra wrote: > > On 1/17/23 19:46, Andres Freund wrote: > > > I think a "hybrid" explain mode might be worth thinking about. Use the > > "current" sampling method for the first execution of a node, and for the > > first > > few milliseconds of a query (or perh

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-14 Thread Nathan Bossart
On Tue, Mar 14, 2023 at 03:38:45PM +1300, Thomas Munro wrote: > On Tue, Mar 14, 2023 at 12:10 PM Nathan Bossart > wrote: >> > * NOTE: although the delay is specified in microseconds, the effective >> > - * resolution is only 1/HZ, or 10 milliseconds, on most Unixen. Expect >> > - * the requeste

Re: Raising the SCRAM iteration count

2023-03-14 Thread Gregory Stark (as CFM)
CFBot is failing with this test failure... I'm not sure if this just represents a timing dependency or a bad test or what? [09:44:49.937] --- stdout --- [09:44:49.937] # executing test in /tmp/cirrus-ci-build/build-32/testrun/authentication/001_password group authentication test 001_password [09:

Re: psql \watch 2nd argument: iteration count

2023-03-14 Thread Nathan Bossart
On Tue, Mar 14, 2023 at 01:58:59PM +0900, Kyotaro Horiguchi wrote: > + if (*opt_end) > + pg_log_error("\\watch: incorrect > interval value '%s'", opt); > + else if (errno == ERANGE) > +

Re: pg_stat_statements and "IN" conditions

2023-03-14 Thread Dmitry Dolgov
> On Tue, Mar 14, 2023 at 02:14:17PM -0400, Gregory Stark (as CFM) wrote: > So I was seeing that this patch needs a rebase according to cfbot. Yeah, folks are getting up to speed in with pgss improvements recently. Thanks for letting me know. > However it looks like the review feedback you're loo

Re: [PoC] Let libpq reject unexpected authentication requests

2023-03-14 Thread Jacob Champion
On Mon, Mar 13, 2023 at 10:39 PM Michael Paquier wrote: > 0001 was looking fine enough seen from here, so applied it after > tweaking a few comments. That's enough to cover most of the needs of > this thread. Thank you very much! > 0002 looks pretty simple as well, I think that's worth a look f

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-03-14 Thread Jacob Champion
On Tue, Mar 14, 2023 at 11:01 AM Gregory Stark (as CFM) wrote: > It does look like a rebase for meson.build would be helpful. I'm not > marking it waiting on author just for meson.build conflicts but it > would be perhaps more likely to be picked up by a committer if it's > showing green in cfbot.

Re: [Proposal] Allow pg_dump to include all child tables with the root table

2023-03-14 Thread Tom Lane
Gilles Darold writes: > Thanks Stepane, I've changed commit fest status to "Ready for committers". Pushed with some minor editing. regards, tom lane

Re: Making background psql nicer to use in tap tests

2023-03-14 Thread Daniel Gustafsson
> On 31 Jan 2023, at 01:00, Andres Freund wrote: > I've hacked some on this. I first tried to just introduce a few helper > functions in Cluster.pm, but that ended up being awkward. So I bit the bullet > and introduced a new class (in BackgroundPsql.pm), and made background_psql() > and interacti

Re: windows CI failing PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED

2023-03-14 Thread Thomas Munro
Here's a better version with more explicit comments about some details. It passes on CI. Planning to push this soon. From 0ac63e73313fffa19db22f19452c4bb08ea7bf14 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 14 Mar 2023 11:53:26 +1300 Subject: [PATCH v2] Fix waitpid() emulation on Wind

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-14 Thread Andres Freund
Hi, I just rebased a patch over commit 1f282c24e46 Author: Michael Paquier Date: 2023-03-13 13:03:29 +0900 Refactor and improve tests of pg_walinspect and got a test failure: https://cirrus-ci.com/task/5693041982308352 https://api.cirrus-ci.com/v1/artifact/task/5693041982308352/testrun

Re: Add pg_walinspect function with block info columns

2023-03-14 Thread Melanie Plageman
I'm excited to see that pg_get_wal_block_info() was merged. Thanks for working on this! Apologies for jumping back in here a bit late. I've been playing around with it and wanted to comment on the performance of JOINing to pg_get_wal_records_info(). On Tue, Mar 7, 2023 at 7:08 AM Matthias van de

Re: Track IO times in pg_stat_io

2023-03-14 Thread Andres Freund
Hi, On 2023-03-09 11:50:38 -0500, Melanie Plageman wrote: > On Tue, Mar 7, 2023 at 1:39 PM Andres Freund wrote: > > On 2023-03-06 11:30:13 -0500, Melanie Plageman wrote: > > > > As pgstat_bktype_io_stats_valid() is called only in Assert(), I think > > > > that would be a good idea > > > > to als

Re: Add pg_walinspect function with block info columns

2023-03-14 Thread Peter Geoghegan
On Tue, Mar 14, 2023 at 3:34 PM Melanie Plageman wrote: > After patching master to add in the columns from > pg_get_wal_records_info() which are not returned by > pg_get_wal_block_info() (except block_ref column of course), this query: > > SELECT COUNT(*) FROM pg_get_wal_block_info(:start_lsn, :

Re: [PATCH] Add pretty-printed XML output option

2023-03-14 Thread Jim Jones
On 14.03.23 18:40, Tom Lane wrote: Jim Jones writes: [ v22-0001-Add-pretty-printed-XML-output-option.patch ] I poked at this for awhile and ran into a problem that I'm not sure how to solve: it misbehaves for input with embedded DOCTYPE. regression=# SELECT xmlserialize(DOCUMENT '' as text in

Re: [PATCH] Add pretty-printed XML output option

2023-03-14 Thread Tom Lane
Jim Jones writes: > On 14.03.23 18:40, Tom Lane wrote: >> I poked at this for awhile and ran into a problem that I'm not sure >> how to solve: it misbehaves for input with embedded DOCTYPE. > The issue was the flag XML_SAVE_NO_EMPTY. It was forcing empty elements > to be serialized with start-en

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread Peter Smith
On Tue, Mar 14, 2023 at 10:43 PM Amit Kapila wrote: > > On Tue, Mar 14, 2023 at 12:37 PM Peter Smith wrote: > > > > Thanks for the review! > > > > On Mon, Mar 13, 2023 at 6:19 PM vignesh C wrote: > > ... > > > > > 4) We check if txn->toptxn is not null twice here both in if condition > > > and i

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread Peter Smith
On Tue, Mar 14, 2023 at 10:33 PM vignesh C wrote: > > On Tue, 14 Mar 2023 at 12:37, Peter Smith wrote: > > > > Thanks for the review! > > > > On Mon, Mar 13, 2023 at 6:19 PM vignesh C wrote: > > ... > > > Few comments: > > > 1) Can we move the macros along with the other macros present in this >

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

2023-03-14 Thread Melanie Plageman
Thanks for your interest in this patch! On Mon, Mar 13, 2023 at 8:38 AM Ants Aasma wrote: > > On Sat, 11 Mar 2023 at 16:55, Melanie Plageman > wrote: > > > > > On Tue, Feb 28, 2023 at 3:16 AM Bharath Rupireddy > > > wrote: > > > > > > > On Thu, Jan 12, 2023 at 6:06 AM Andres Freund > > > > wr

Re: Add pg_walinspect function with block info columns

2023-03-14 Thread Melanie Plageman
On Tue, Mar 14, 2023 at 6:57 PM Peter Geoghegan wrote: > > On Tue, Mar 14, 2023 at 3:34 PM Melanie Plageman > wrote: > > After patching master to add in the columns from > > pg_get_wal_records_info() which are not returned by > > pg_get_wal_block_info() (except block_ref column of course), this q

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 02:54:40PM -0700, Andres Freund wrote: > Object description > --- > function pg_get_wal_record_info(pg_lsn) > - function pg_get_wal_records_info(pg_lsn,pg_lsn) > function pg_get_wal_records_inf

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

2023-03-14 Thread Melanie Plageman
Thanks for the review! On Sat, Mar 11, 2023 at 2:16 PM Justin Pryzby wrote: > > On Sat, Mar 11, 2023 at 09:55:33AM -0500, Melanie Plageman wrote: > > Subject: [PATCH v3 2/3] use shared buffers when failsafe active > > > > + /* > > + * Assume the caller who allocated the m

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 10:35:43AM +0530, Bharath Rupireddy wrote: > My thoughts are simple here - how would one (an end user, not me and > not you) figure out how to get info/stats till the end of WAL? I'm > sure it would be difficult to find that out without looking at the > code or commit histor

Re: Making background psql nicer to use in tap tests

2023-03-14 Thread Andres Freund
Hi, On 2023-03-14 21:24:32 +0100, Daniel Gustafsson wrote: > > On 31 Jan 2023, at 01:00, Andres Freund wrote: > > > I've hacked some on this. I first tried to just introduce a few helper > > functions in Cluster.pm, but that ended up being awkward. So I bit the > > bullet > > and introduced a n

Re: psql \watch 2nd argument: iteration count

2023-03-14 Thread Kyotaro Horiguchi
At Tue, 14 Mar 2023 12:03:00 -0700, Nathan Bossart wrote in > On Tue, Mar 14, 2023 at 01:58:59PM +0900, Kyotaro Horiguchi wrote: > > + if (*opt_end) > > + pg_log_error("\\watch: incorrect > > interval value '%s'", opt); > > +

Re: psql \watch 2nd argument: iteration count

2023-03-14 Thread Michael Paquier
On Wed, Mar 15, 2023 at 10:19:28AM +0900, Kyotaro Horiguchi wrote: > I hesitated to propose such a level of simplification, but basically I > was alsothinking the same thing. Okay, fine by me to use one single message. I'd rather still keep the three tests, though, as they check the three conditi

Re: Add pg_walinspect function with block info columns

2023-03-14 Thread Peter Geoghegan
On Tue, Mar 14, 2023 at 5:34 PM Melanie Plageman wrote: > On Tue, Mar 14, 2023 at 6:57 PM Peter Geoghegan wrote: > > Why doesn't it already work like this? Why do we need a separate > > pg_get_wal_block_info() function at all? > > Well, I think if you only care about the WAL record-level informat

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-14 Thread Andres Freund
Hi, On 2023-03-15 09:56:10 +0900, Michael Paquier wrote: > On Tue, Mar 14, 2023 at 02:54:40PM -0700, Andres Freund wrote: > > Object description > > --- > > function pg_get_wal_record_info(pg_lsn) > > - function pg_get

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-03-14 Thread Masahiko Sawada
On Tue, Mar 14, 2023 at 8:27 PM John Naylor wrote: > > I wrote: > > > > > Since the block-level measurement is likely overestimating quite a bit, > > > > I propose to simply reverse the order of the actions here, effectively > > > > reporting progress for the *last page* and not the current one:

Re: psql \watch 2nd argument: iteration count

2023-03-14 Thread Andrey Borodin
On Tue, Mar 14, 2023 at 6:25 PM Michael Paquier wrote: > > On Wed, Mar 15, 2023 at 10:19:28AM +0900, Kyotaro Horiguchi wrote: > > I hesitated to propose such a level of simplification, but basically I > > was alsothinking the same thing. +1 > Okay, fine by me to use one single message. I'd rathe

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-14 Thread Amit Kapila
On Tue, Mar 14, 2023 at 3:18 PM Önder Kalacı wrote: >> Pushed this patch but forgot to add a new testfile. Will do that soon. -- With Regards, Amit Kapila.

Re: pg_dump/pg_restore: Fix stdin/stdout handling of custom format on Win32

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 01:47:09PM +0100, Juan José Santamaría Flecha wrote: > I have just posted a patch to enforce the detection of unseekable streams > in the fseek() calls [1], please feel free to review it. Thanks. I have been able to get around 0001 to fix _pgfstat64() and applied it down t

Re: psql \watch 2nd argument: iteration count

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 08:20:23PM -0700, Andrey Borodin wrote: > PFA v8. Thanks! Looks OK to me. I've looked as well at resetting query_buffer on failure, which I guess is better this way because this is an accumulation of the previous results, right? -- Michael signature.asc Description: PGP

Re: recovery modules

2023-03-14 Thread Nathan Bossart
I noticed that the new TAP test for basic_archive was failing intermittently for cfbot. It looks like the query for checking that the post-backup WAL is restored sometimes executes before archive recovery is complete (because hot_standby is on). To fix this, I adjusted the test to use poll_query_

Re: psql \watch 2nd argument: iteration count

2023-03-14 Thread Nathan Bossart
+ sleep = strtod(opt, &opt_end); + if (sleep < 0 || *opt_end || errno == ERANGE) Should we set errno to 0 before calling strtod()? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Fix fseek() detection of unseekable files on WIN32

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 01:26:27PM +0100, Juan José Santamaría Flecha wrote: > As highlighted in [1] fseek() might fail to error even when accessing > unseekable streams. > > PFA a patch that checks the file type before the actual fseek(), so only > supported calls are made. + * streams, so harde

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-14 Thread Thomas Munro
On Wed, Mar 15, 2023 at 7:54 AM Nathan Bossart wrote: > Here is roughly what I had in mind: > > NOTE: Although the delay is specified in microseconds, older Unixen > and > Windows use periodic kernel ticks to wake up, which might increase the > delay time significantly. W

Re: Should vacuum process config file reload more often

2023-03-14 Thread Masahiko Sawada
On Sat, Mar 11, 2023 at 8:11 AM Melanie Plageman wrote: > > Quotes below are combined from two of Sawada-san's emails. > > I've also attached a patch with my suggested current version. > > On Thu, Mar 9, 2023 at 10:27 PM Masahiko Sawada wrote: > > > > On Fri, Mar 10, 2023 at 11:23 AM Melanie Plag

Re: Allow logical replication to copy tables in binary format

2023-03-14 Thread Amit Kapila
On Tue, Mar 14, 2023 at 8:50 PM Takamichi Osumi (Fujitsu) wrote: > > On Tuesday, March 14, 2023 8:02 PM Melih Mutlu wrote: > (3) copy_table() > > + /* > +* If the publisher version is earlier than v14, it COPY command > doesn't > +* support the binary option. > +*/

Re: Add macros for ReorderBufferTXN toptxn

2023-03-14 Thread Masahiko Sawada
Hi, On Wed, Mar 15, 2023 at 8:55 AM Peter Smith wrote: > > On Tue, Mar 14, 2023 at 10:43 PM Amit Kapila wrote: > > > > On Tue, Mar 14, 2023 at 12:37 PM Peter Smith wrote: > > > > > > Thanks for the review! > > > > > > On Mon, Mar 13, 2023 at 6:19 PM vignesh C wrote: > > > ... > > > > > > > 4)

Re: Allow logical replication to copy tables in binary format

2023-03-14 Thread Peter Smith
Here are some review comments for v13-0001 == doc/src/sgml/logical-replication.sgml 1. @@ -241,10 +241,13 @@ types of the columns do not need to match, as long as the text representation of the data can be converted to the target type. For example, you can replicate from a column

Re: Allow logical replication to copy tables in binary format

2023-03-14 Thread Amit Kapila
On Wed, Mar 15, 2023 at 11:52 AM Peter Smith wrote: > > == > src/backend/replication/logical/tablesync.c > > 5. > + > + /* > + * If the publisher version is earlier than v14, it COPY command doesn't > + * support the binary option. > + */ > + if (walrcv_server_version(LogRepWorkerWalRcvConn) >

Re: Add pg_walinspect function with block info columns

2023-03-14 Thread Bharath Rupireddy
On Wed, Mar 15, 2023 at 7:20 AM Peter Geoghegan wrote: > > > But, perhaps you are suggesting a parameter to pg_get_wal_records_info() > > like "with_block_info" or something, which produces the full > > denormalized block + record output? > > I was thinking of something like that, yes -- though it

Re: ICU locale validation / canonicalization

2023-03-14 Thread Jeff Davis
On Tue, 2023-03-14 at 10:10 -0700, Jeff Davis wrote: > One loose end is that we really should support language tags like > "und" > in those older versions (54 and earlier). Your commit d72900bded > avoided the problem, but perhaps we should fix it by looking for > "und" > and replacing it with "roo

RE: Allow logical replication to copy tables in binary format

2023-03-14 Thread Takamichi Osumi (Fujitsu)
Hi, On Wednesday, March 15, 2023 2:34 PM Amit Kapila wrote: > On Tue, Mar 14, 2023 at 8:50 PM Takamichi Osumi (Fujitsu) > wrote: > > > > On Tuesday, March 14, 2023 8:02 PM Melih Mutlu > wrote: > > (3) copy_table() > > > > + /* > > +* If the publisher version is earlier than v14,

Re: Add pg_walinspect function with block info columns

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 06:50:15PM -0700, Peter Geoghegan wrote: > On Tue, Mar 14, 2023 at 5:34 PM Melanie Plageman > wrote: >> Well, I think if you only care about the WAL record-level information >> and not the block-level information, having the WAL record information >> denormalized like that

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-14 Thread Michael Paquier
On Tue, Mar 14, 2023 at 07:05:20PM -0700, Andres Freund wrote: > It's using ICU, but not a specific collation. The build I linked to is WIP > hackery to add ICU support to windows CI. Here's the initdb output: > https://api.cirrus-ci.com/v1/artifact/task/6288336663347200/testrun/build/testrun/pg_wa