Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-12 Thread Michael Paquier
On Tue, Jan 12, 2021 at 12:10:24PM -0300, Alvaro Herrera wrote: > Apparently b2a5545bd63f changed; before that commit, that code > (including the quoted comment) was all in RemoveOldXlogFiles, and > endlogSegNo was calculated only once. But ISTM that even with that > formulation it had the problem

RE: ResourceOwner refactoring

2021-01-12 Thread kuroda.hay...@fujitsu.com
Dear Heikki, Thank you for rebasing it, I confirmed it can be applied. I will check the source. Now I put the very elementary comment. ResourceOwnerEnlarge(), ResourceOwnerRemember(), and ResourceOwnerForget() are exported routines. They should put below L418. Best regards, Hayato Kuroda FUJITSU

Re: Track replica origin progress for Rollback Prepared

2021-01-12 Thread Ajin Cherian
On Wed, Jan 13, 2021 at 12:11 AM Amit Kapila wrote: > Thanks for doing these tests. I think you can put an elog in the below > code change as well to show that the recovery code path is also hit: > > +xact_redo_abort(xl_xact_parsed_abort *parsed, TransactionId xid, > + XLogRecPtr lsn, RepOriginId

Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-12 Thread Michael Paquier
On Tue, Jan 12, 2021 at 11:30:13PM +1300, Thomas Munro wrote: > I haven't heard any user complaints, and I'd personally be happy with > a fix on master only. I have been looking again at that, and the rebased version that Andres has provided would take care of that. Any thoughts? -- Michael From

Re: ResourceOwner refactoring

2021-01-12 Thread Michael Paquier
On Wed, Jan 13, 2021 at 09:18:57AM +0200, Heikki Linnakangas wrote: > --- a/src/common/cryptohash_openssl.c > +++ b/src/common/cryptohash_openssl.c > +static ResourceOwnerFuncs cryptohash_funcs = > +{ > + /* relcache references */ > + .name = "LLVM JIT context", > + .phase = RESOURCE_RE

Re: adding partitioned tables to publications

2021-01-12 Thread Amit Langote
On Tue, Jan 12, 2021 at 5:09 PM Amit Kapila wrote: > > On Mon, Jan 11, 2021 at 5:44 PM Mark Zhao <875941...@qq.com> wrote: > > > > Thanks for your reply. The patch is exactly what I want. > > My English name is Mark Zhao, which should be the current email name. > > > > Pushed the fix. Thanks Amit

Re: ResourceOwner refactoring

2021-01-12 Thread Heikki Linnakangas
On 13/01/2021 03:55, kuroda.hay...@fujitsu.com wrote: Dear Heikki, I'm also interested in this patch, but it cannot be applied to the current HEAD... $ git apply ~/v2-0001-Make-resowners-more-easily-extensible.patch error: patch failed: src/common/cryptohash_openssl.c:57 error: src/common/cryp

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread japin
On Wed, 13 Jan 2021 at 13:26, Amit Kapila wrote: > On Tue, Jan 12, 2021 at 4:59 PM Bharath Rupireddy > wrote: >> >> On Tue, Jan 12, 2021 at 12:06 PM Amit Kapila wrote: >> > > Here's my analysis: >> > > 1) in the publisher, alter publication drop table successfully >> > > removes(PublicationDrop

Wrong usage of RelationNeedsWAL

2021-01-12 Thread Kyotaro Horiguchi
Hello. Commit c6b92041d3 changed the definition of RelationNeedsWAL(). -#define RelationNeedsWAL(relation) \ - ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT) +#define RelationNeedsWAL(relation) \

Re: Single transaction in the tablesync worker?

2021-01-12 Thread Peter Smith
On Wed, Jan 13, 2021 at 1:07 PM Hou, Zhijie wrote: > > > Also PSA some detailed logging evidence of some test scenarios involving > > Drop/AlterSubscription: > > + Test-20210112-AlterSubscriptionRefresh-ok.txt = > > AlterSubscription_refresh which successfully drops

Re: remove unneeded pstrdup in fetch_table_list

2021-01-12 Thread Amit Kapila
On Wed, Jan 13, 2021 at 8:11 AM Hou, Zhijie wrote: > > Hi > > In function fetch_table_list, it get the table names from publicer and return > a list of tablenames. > When append the name to the list, it use the following code: > > ** > nspname = TextDatumGetCString(slot_getattr(slot, 1, &

RE: Disable WAL logging to speed up data loading

2021-01-12 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > XLogBeginInsert(); > XLogSetRecrodFlags(XLOG_MARK_ESSENTIAL); # new flag value > XLOGInsert(); Oh, sounds like a nice idea. That's more flexible by allowing WAL-emitting modules to specify which WAL records are mandatory even when wal_level is none. For example,

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-12 Thread Michael Paquier
On Tue, Jan 12, 2021 at 04:51:39PM -0300, Alvaro Herrera wrote: > I looked into this again, and I didn't like what I had added to > maintenance.sgml at all. It seems out of place where I put it; and I > couldn't find any great spots. Going back to your original proposal, > what about something li

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Amit Kapila
On Wed, Jan 13, 2021 at 11:08 AM Bharath Rupireddy wrote: > > On Wed, Jan 13, 2021 at 10:33 AM Amit Kapila wrote: > > > > On Tue, Jan 12, 2021 at 5:23 PM japin wrote: > > > > > > On Tue, 12 Jan 2021 at 19:32, Bharath Rupireddy wrote: > > > > On Tue, Jan 12, 2021 at 4:47 PM Li Japin wrote: > > >

Re: Single transaction in the tablesync worker?

2021-01-12 Thread Peter Smith
On Mon, Jan 4, 2021 at 10:48 PM Amit Kapila wrote: > 7. > @@ -905,7 +905,7 @@ replorigin_advance(RepOriginId node, > LWLockAcquire(&replication_state->lock, LW_EXCLUSIVE); > > /* Make sure it's not used by somebody else */ > - if (replication_state->acquired_by != 0) > + if (replication_state-

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Bharath Rupireddy
On Wed, Jan 13, 2021 at 10:33 AM Amit Kapila wrote: > > On Tue, Jan 12, 2021 at 5:23 PM japin wrote: > > > > On Tue, 12 Jan 2021 at 19:32, Bharath Rupireddy wrote: > > > On Tue, Jan 12, 2021 at 4:47 PM Li Japin wrote: > > >> IIUC the logical replication only replicate the tables in publication,

Re: Movement of restart_lsn position movement of logical replication slots is very slow

2021-01-12 Thread Amit Kapila
On Tue, Jan 12, 2021 at 9:15 AM Jammie wrote: > > Hi Amit, > Thanks for the response . > Can you please let me know what pg_current_wal_lsn returns ? > > is this position the LSN of the next log record to be created, or is it the > LSN of the last log record already created and inserted in the lo

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-01-12 Thread k.jami...@fujitsu.com
On Wed, January 13, 2021 2:15 PM (JST), Amit Kapila wrote: > On Wed, Jan 13, 2021 at 7:39 AM Kyotaro Horiguchi > wrote: > > > > At Tue, 12 Jan 2021 08:49:53 +0530, Amit Kapila > > wrote in > > > On Fri, Jan 8, 2021 at 7:03 AM Kyotaro Horiguchi > > > wrote: > > > > > > > > At Thu, 7 Jan 2021 09:2

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Amit Kapila
On Tue, Jan 12, 2021 at 4:59 PM Bharath Rupireddy wrote: > > On Tue, Jan 12, 2021 at 12:06 PM Amit Kapila wrote: > > > Here's my analysis: > > > 1) in the publisher, alter publication drop table successfully > > > removes(PublicationDropTables) the table from the catalogue > > > pg_publication_re

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-01-12 Thread Amit Kapila
On Wed, Jan 13, 2021 at 7:39 AM Kyotaro Horiguchi wrote: > > At Tue, 12 Jan 2021 08:49:53 +0530, Amit Kapila > wrote in > > On Fri, Jan 8, 2021 at 7:03 AM Kyotaro Horiguchi > > wrote: > > > > > > At Thu, 7 Jan 2021 09:25:22 +, "k.jami...@fujitsu.com" > > > wrote in: > > > > > Thanks for t

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Amit Kapila
On Tue, Jan 12, 2021 at 5:23 PM japin wrote: > > On Tue, 12 Jan 2021 at 19:32, Bharath Rupireddy wrote: > > On Tue, Jan 12, 2021 at 4:47 PM Li Japin wrote: > >> IIUC the logical replication only replicate the tables in publication, I > >> think > >> when the tables that aren't in publication sho

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-12 Thread Fujii Masao
On Tue, Jan 12, 2021 at 11:09 AM Fujii Masao wrote: > > On Tue, Jan 12, 2021 at 10:00 AM Masahiko Sawada > wrote: > > > > On Mon, Jan 11, 2021 at 11:00 PM Peter Eisentraut > > wrote: > > > > > > On 2021-01-05 10:56, Masahiko Sawada wrote: > > > > BTW according to the documentation, the options

RE: Terminate the idle sessions

2021-01-12 Thread kuroda.hay...@fujitsu.com
Dear Tom, > So I propose to change the new ERRCODE_IDLE_SESSION_TIMEOUT to be in > class 57 and call it good. I agreed your suggestion and I confirmed your commit. Thanks! Hayato Kuroda FUJITSU LIMITED

Re: Disable WAL logging to speed up data loading

2021-01-12 Thread movead li
I read the patch and have two points: 1. I do basebackup for database then switch wal level from logical to none to logical and of cause I archive the wal segments. Next I do PITR base on the basebackup, as a result it success startup with a waring said maybe data missed. Because the 'none' lev

Re: Logical decoding without slots: decoding in lockstep with recovery

2021-01-12 Thread Craig Ringer
On Sat, 26 Dec 2020 at 06:51, Andres Freund wrote: > Hi, > > On 2020-12-23 14:56:07 +0800, Craig Ringer wrote: > > I want to share an idea I've looked at a few times where I've run into > > situations where logical slots were inadvertently dropped, or where it > > became necessary to decode chang

Re: A failure of standby to follow timeline switch

2021-01-12 Thread Fujii Masao
On Wed, Jan 13, 2021 at 10:48 AM Kyotaro Horiguchi wrote: > > At Tue, 12 Jan 2021 10:47:21 +0900, Fujii Masao wrote > in > > On Sat, Jan 9, 2021 at 5:08 AM Alvaro Herrera > > wrote: > > > > > > Masao-san: Are you intending to act as committer for these? Since the > > > bug is mine I can look

remove unneeded pstrdup in fetch_table_list

2021-01-12 Thread Hou, Zhijie
Hi In function fetch_table_list, it get the table names from publicer and return a list of tablenames. When append the name to the list, it use the following code: ** nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull)); Assert(!isnull); relname = TextDatumGetCSt

Re: Tid scan improvements

2021-01-12 Thread Edmund Horner
On Fri, 1 Jan 2021 at 14:30, David Fetter wrote: > On Sun, Dec 01, 2019 at 11:34:16AM +0900, Michael Paquier wrote: > > Okay, still nothing has happened after two months. Once this is > > solved a new patch submission could be done. For now I have marked > > the entry as returned with feedback.

Re: Key management with tests

2021-01-12 Thread Neil Chen
Thank you for your reply, On Wed, Jan 13, 2021 at 12:08 AM Stephen Frost wrote: > > No, we can't 'modify the page format as we wish'- if we change away from > using a C structure then we're going to be modifying quite a bit of > code which otherwise doesn't need to be changed. The proposed flag

Re: Moving other hex functions to /common

2021-01-12 Thread Michael Paquier
On Tue, Jan 12, 2021 at 01:13:00PM -0500, Bruce Momjian wrote: > Thanks for you work on this. Looks good. I have been looking again at this patch again for a couple of hours this morning to double-check if I have not missed anything, and I think that we should be in good shape. This still needs

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-01-12 Thread Kyotaro Horiguchi
At Tue, 12 Jan 2021 08:49:53 +0530, Amit Kapila wrote in > On Fri, Jan 8, 2021 at 7:03 AM Kyotaro Horiguchi > wrote: > > > > At Thu, 7 Jan 2021 09:25:22 +, "k.jami...@fujitsu.com" > > wrote in: > > > > Thanks for the detailed tests. NBuffers/32 seems like an appropriate > > > > value for

Re: Disable WAL logging to speed up data loading

2021-01-12 Thread Kyotaro Horiguchi
At Tue, 12 Jan 2021 07:09:28 +, "osumi.takami...@fujitsu.com" wrote in > On Tuesday, January 12, 2021 12:52 PM Takayuki/綱川 貴之 > wrote: > > From: Osumi, Takamichi/大墨 昂道 > > > I updated the patch following this discussion, and fixed the > > > documentation as well. > > > > > > +

RE: Single transaction in the tablesync worker?

2021-01-12 Thread Hou, Zhijie
> Also PSA some detailed logging evidence of some test scenarios involving > Drop/AlterSubscription: > + Test-20210112-AlterSubscriptionRefresh-ok.txt = > AlterSubscription_refresh which successfully drops a tablesync slot > + Test-20210112-AlterSubscriptionRefr

RE: ResourceOwner refactoring

2021-01-12 Thread kuroda.hay...@fujitsu.com
Dear Heikki, I'm also interested in this patch, but it cannot be applied to the current HEAD... $ git apply ~/v2-0001-Make-resowners-more-easily-extensible.patch error: patch failed: src/common/cryptohash_openssl.c:57 error: src/common/cryptohash_openssl.c: patch does not apply error: patch fai

Re: A failure of standby to follow timeline switch

2021-01-12 Thread Kyotaro Horiguchi
At Tue, 12 Jan 2021 10:47:21 +0900, Fujii Masao wrote in > On Sat, Jan 9, 2021 at 5:08 AM Alvaro Herrera wrote: > > > > Masao-san: Are you intending to act as committer for these? Since the > > bug is mine I can look into it, but since you already did all the > > reviewing work, I'm good with

Re: outdated references to replication timeout

2021-01-12 Thread Fujii Masao
On Wed, Jan 13, 2021 at 5:39 AM John Naylor wrote: > > Hi, > > The parameter replication_timeout was retired in commit 6f60fdd701 back in > 2012, but some comments and error messages seem to refer to that old setting > instead of wal_sender_timeout or wal_receiver_timeout. The attached patch >

Re: Fix a typo in SearchCatCache function comment

2021-01-12 Thread Michael Paquier
On Tue, Jan 12, 2021 at 04:21:18PM +0900, Michael Paquier wrote: > Good catch. I'll go fix it tomorrow if nobody objects. And applied. Thanks! -- Michael signature.asc Description: PGP signature

Re: pg_preadv() and pg_pwritev()

2021-01-12 Thread Thomas Munro
On Mon, Jan 11, 2021 at 3:59 PM Thomas Munro wrote: > On Mon, Jan 11, 2021 at 3:34 PM Thomas Munro wrote: > > I pushed it with that name, and a couple more cosmetic changes. I'll > > keep an eye on the build farm. > > Since only sifaka has managed to return a result so far (nice CPU), I > had pl

Re: pg_dump INDEX ATTACH versus --clean option

2021-01-12 Thread Alvaro Herrera
On 2021-Jan-12, Tom Lane wrote: > I think actually the cleanest fix would be to invent ALTER INDEX DETACH > PARTITION and use that as the dropStmt for the INDEX ATTACH object. > No idea how painful that would be to do, though. I suppose it'd involve > reverting the parent index back to an invalid

Re: WIP: System Versioned Temporal Table

2021-01-12 Thread Ryan Lambert
On Mon, Jan 11, 2021 at 7:02 AM Simon Riggs wrote: > On Sat, Jan 9, 2021 at 10:39 AM Simon Riggs > wrote: > > > > On Fri, Jan 8, 2021 at 9:19 PM Ryan Lambert > wrote: > > > > >> Updated v11 with additional docs and some rewording of messages/tests > > >> to use "system versioning" correctly. >

Re: list of extended statistics on psql

2021-01-12 Thread Tatsuro Yamada
Hi Tomas, On 2021/01/12 20:08, Tomas Vondra wrote: On 1/12/21 2:57 AM, Tatsuro Yamada wrote: Hi Tomas, On 2021/01/09 9:01, Tomas Vondra wrote: ...> While working on that, I realized that 'defined' might be a bit ambiguous, I initially thought it means 'NOT NULL' (which it does not). I propo

Re: pgbench and timestamps (bounced)

2021-01-12 Thread Tom Lane
Fabien COELHO writes: >> Hi, this entry is "Waiting on Author" and the thread was inactive for a >> while. I see this discussion still has some open questions. Are you >> going to continue working on it, or should I mark it as "returned with >> feedback" until a better time? > IMHO the propose

Re: pg_dump INDEX ATTACH versus --clean option

2021-01-12 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Jan-12, Tom Lane wrote: >> Since there's no ALTER INDEX DETACH PARTITION, it's not entirely >> clear what to do about this. We could possibly not emit any >> dropStmt for partition child indexes, but that seems very likely >> to cause problems for partial-restore

Re: pg_upgrade test for binary compatibility of core data types

2021-01-12 Thread Andrew Dunstan
On 1/12/21 12:53 PM, Bruce Momjian wrote: > On Tue, Jan 12, 2021 at 11:27:53AM -0600, Justin Pryzby wrote: >> On Tue, Jan 12, 2021 at 12:15:59PM -0500, Bruce Momjian wrote: >>> Uh, what exactly is missing from the beta checklist? I read the patch >>> and commit message but don't understand it. >

Re: pg_dump INDEX ATTACH versus --clean option

2021-01-12 Thread Alvaro Herrera
On 2021-Jan-12, Tom Lane wrote: > then "pg_dump -n s1 -c mydb >mydb.dump" will emit > > ALTER TABLE ONLY s1.at12 DROP CONSTRAINT at12_pkey; > ALTER TABLE ONLY s1.at11 DROP CONSTRAINT at11_pkey; > ALTER TABLE ONLY s1.at1 DROP CONSTRAINT at1_pkey; > DROP TABLE s1.at12; > DROP TABLE s1.at11; > DROP

outdated references to replication timeout

2021-01-12 Thread John Naylor
Hi, The parameter replication_timeout was retired in commit 6f60fdd701 back in 2012, but some comments and error messages seem to refer to that old setting instead of wal_sender_timeout or wal_receiver_timeout. The attached patch replaces the old language with more specific references. -- John N

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-12 Thread Álvaro Herrera
On 2021-Jan-12, Álvaro Herrera wrote: > > For the 0001 patch, since ReindexIndexInfo is used only within > > ReindexRelationConcurrently() I think it’s a function-local structure > > type. So we can declare it within the function. What do you think? > > That's a good idea. Pushed with that chang

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-12 Thread Álvaro Herrera
On 2021-Jan-04, Masahiko Sawada wrote: > On Tue, Dec 8, 2020 at 5:18 PM Hamid Akhtar wrote: > > > > The following review has been posted through the commitfest application: > > make installcheck-world: tested, passed > > Implements feature: not tested > > Spec compliant: not test

Re: [DOC] Document concurrent index builds waiting on each other

2021-01-12 Thread Alvaro Herrera
On 2020-Dec-01, James Coleman wrote: > On Tue, Dec 1, 2020 at 6:51 PM Alvaro Herrera wrote: > > Makes sense. ISTM that if we want to have a cautionary blurb CIC docs, > > it should go in REINDEX CONCURRENTLY as well. > > Agreed. Or, alternatively, a blurb something like "Please note how CIC >

pg_dump INDEX ATTACH versus --clean option

2021-01-12 Thread Tom Lane
I noticed that pg_dump --clean does not work with partitioned indexes. Given for instance create schema s1; create table s1.at1 (f1 int, f2 int, primary key(f1,f2)) partition by list(f1); create table s1.at11 partition of s1.at1 for values in(11); create table s1.at12 partition of s1.at1 for valu

Re: Key management with tests

2021-01-12 Thread Andres Freund
Hi, On 2021-01-11 20:12:00 +0900, Masahiko Sawada wrote: > diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c > index 32b5d62e1f..d474af753c 100644 > --- a/contrib/bloom/blinsert.c > +++ b/contrib/bloom/blinsert.c > @@ -177,6 +177,7 @@ blbuildempty(Relation index) >* XLOG_DB

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:57:11PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > Well, we have eight unused bits in the IV, so we coul

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2021-01-12 Thread Tomas Vondra
Thanks. These patches seem to resolve the TOAST table issue, freezing it as expected. I think the code duplication is not an issue, but I wonder why heap_insert uses this condition: /* * ... * * No need to update the visibilitymap if it had all_frozen bit set * before th

Re: [HACKERS] [PATCH] Generic type subscripting

2021-01-12 Thread Pavel Stehule
ne 10. 1. 2021 v 19:52 odesílatel Pavel Stehule napsal: > Hi > > >> I'm thinking of the update path as a kind of implicit schema. JSON is >> intentionally not bound to any schema on creation, so I don't see a >> failure to enforce another schema at runtime (and outside the WHERE >> clause, at tha

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Well, we have eight unused bits in the IV, so we could just increment > > > that for every hint bit change that uses the same LS

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Well, we have eight unused bits in the IV, so we could just increment > > that for every hint bit change that uses the same LSN, and then force a > > dummy WAL record when that 8-bit count

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > I don't think there's any doubt that we need to make sure that the IV is > > distinct and advancing the LSN to get a new one when needed for this > > case seems like i

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > > I think one big question is that, since we are using a streaming cipher, > > > do we care about hint bit changes showing to users? I actually don't > > > know. If we do, so

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > I think one big question is that, since we are using a streaming cipher, > > do we care about hint bit changes showing to users? I actually don't > > know. If we do, some kind of dummy LSN record might be required, as you > > sugg

Re: Moving other hex functions to /common

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 11:26:51AM +0900, Michael Paquier wrote: > The two only things that were not present are the set of checks for > overflows, and the adjustments for varlena.c. The first point makes > the code of encode.c safer, as previously the code would issue a FATAL > *after* writing ou

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > > This says: > > > > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > > > wal_log_hints will be enabled auto

Re: Key management with tests

2021-01-12 Thread Andres Freund
On 2021-01-12 13:03:14 -0500, Bruce Momjian wrote: > I think one big question is that, since we are using a streaming cipher, > do we care about hint bit changes showing to users? I actually don't > know. If we do, some kind of dummy LSN record might be required, as you > suggested. That'd lead

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > This says: > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > wal_log_hints will be enabled automatically in encryption mode. > > > > Does that help? > > IIUC it h

Re: pg_upgrade test for binary compatibility of core data types

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 11:27:53AM -0600, Justin Pryzby wrote: > On Tue, Jan 12, 2021 at 12:15:59PM -0500, Bruce Momjian wrote: > > Uh, what exactly is missing from the beta checklist? I read the patch > > and commit message but don't understand it. > > Did you try to use test.sh to upgrade from

Re: WIP: BRIN multi-range indexes

2021-01-12 Thread Tomas Vondra
On 1/12/21 6:28 PM, John Naylor wrote: On Sat, Dec 19, 2020 at 8:15 PM Tomas Vondra mailto:tomas.von...@enterprisedb.com>> wrote: > [12-20 version] Hi Tomas, The measurements look good. In case it fell through the cracks, my earlier review comments for Bloom BRIN indexes regarding minor

pg_dump PublicationRelInfo objects need to be marked with owner

2021-01-12 Thread Tom Lane
Although pg_publication_rel entries don't have an owner per se, it's still important for the ArchiveEntry created for one to be marked with an appropriate owner, because that is what determines which role will be used to run the ALTER PUBLICATION command when doing a restore with --use-set-session-

Re: WIP: BRIN multi-range indexes

2021-01-12 Thread John Naylor
On Sat, Dec 19, 2020 at 8:15 PM Tomas Vondra wrote: > [12-20 version] Hi Tomas, The measurements look good. In case it fell through the cracks, my earlier review comments for Bloom BRIN indexes regarding minor details don't seem to have been addressed in this version. I'll point to earlier discu

Re: pg_upgrade test for binary compatibility of core data types

2021-01-12 Thread Justin Pryzby
On Tue, Jan 12, 2021 at 12:15:59PM -0500, Bruce Momjian wrote: > On Mon, Jan 11, 2021 at 10:13:52PM -0600, Justin Pryzby wrote: > > On Mon, Jan 11, 2021 at 03:28:08PM +0100, Peter Eisentraut wrote: > > > I think these patches could use some in-place documentation of what they > > > are > > > tryin

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2021-01-12 Thread Anastasia Lubennikova
On 12.01.2021 00:51, Tomas Vondra wrote: On 1/11/21 10:00 PM, Anastasia Lubennikova wrote: On 11.01.2021 01:35, Tomas Vondra wrote: Hi, I started looking at this patch again, hoping to get it committed in this CF, but I think there's a regression in handling TOAST tables (compared to the v

Re: pg_upgrade test for binary compatibility of core data types

2021-01-12 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 10:13:52PM -0600, Justin Pryzby wrote: > On Mon, Jan 11, 2021 at 03:28:08PM +0100, Peter Eisentraut wrote: > > I think these patches could use some in-place documentation of what they are > > trying to achieve and how they do it. The required information is spread > > over

Re: libpq compression

2021-01-12 Thread Andrey Borodin
> 12 янв. 2021 г., в 20:47, Konstantin Knizhnik > написал(а): > >> I think we should come up with an minimal, prelimininary 0001 patch which is >> common between the 3 compression patches (or at least the two using zstd). >> The >> ./configure changes and a compressionlibs struct would also

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2021-01-12 Thread Andrey Borodin
> 12 янв. 2021 г., в 13:49, Noah Misch написал(а): > > What do you think of abandoning slru-truncate-t-insurance entirely? As of > https://postgr.es/m/20200330052809.gb2324...@rfd.leadboat.com I liked the idea > behind it, despite its complicating the system for hackers and DBAs. The > Trunc

Re: Yet another fast GiST build

2021-01-12 Thread Andrey Borodin
> 12 янв. 2021 г., в 18:49, Heikki Linnakangas написал(а): > >> PFA patch with implementation. > > I did a bit of cleanup on the function signature. The .sql script claimed > that gist_page_items() took bytea as argument, but in reality it was a > relation name, as text. I changed it so tha

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Neil Chen (carpenter.nail...@gmail.com) wrote: > On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost wrote: > > This is an interesting question but ultimately I don't think we should > > be looking at this from the perspective of allowing arbitrary changes to > > the page format. The cha

Re: libpq compression

2021-01-12 Thread Konstantin Knizhnik
On 12.01.2021 18:38, Justin Pryzby wrote: On Tue, Jan 12, 2021 at 08:44:43AM +0300, Konstantin Knizhnik wrote: On 11.01.2021 20:38, Tomas Vondra wrote: 1) Fixes the MSVC makefile. The list of files is sorted alphabetically, so I've added the file at the end. Thank you This is still failing

Re: libpq compression

2021-01-12 Thread Justin Pryzby
On Tue, Jan 12, 2021 at 08:44:43AM +0300, Konstantin Knizhnik wrote: > On 11.01.2021 20:38, Tomas Vondra wrote: > > 1) Fixes the MSVC makefile. The list of files is sorted alphabetically, > > so I've added the file at the end. > Thank you This is still failing the windows build. I think you need

Re: Executing inet_server_addr/port() in parallel workers

2021-01-12 Thread Tom Lane
BTW, just for the archives' sake: inet_client_addr() and inet_client_port() also access MyProcPort, so they'd also have this issue, except that they're already marked parallel-restricted. So somebody just missed the server equivalents when marking the parallel safety of built-in functions. contrib

Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-12 Thread Alvaro Herrera
On 2021-Jan-11, Thomas Munro wrote: > I didn't check the migration history of this code but it seems that > endlogSegNo doesn't currently have the right scoping to achieve the > goal of that last comment, so checkpoints finish up repeatedly search > for the next free slot, starting at the low end

Re: Fix a typo in xlogreader.c

2021-01-12 Thread Alvaro Herrera
On 2021-Jan-12, Masahiko Sawada wrote: > Hi, > > The function comment of RestoreBlockImage() seems not correct since it > returns a boolean, not the buffer number. You're right -- this has always been wrong. Pushed your patch. Thank you! -- Álvaro Herrera39°49'3

Re: Executing inet_server_addr/port() in parallel workers

2021-01-12 Thread Tom Lane
Masahiko Sawada writes: > While investigating a customer issue it's turned out that if a > parallel worker executes inet_server_addr() and inet_server_port() the > results are always null because MyProcPort is not set in parallel > workers. Check. > To fix this issue, I think there are two optio

Re: Added schema level support for publication.

2021-01-12 Thread vignesh C
On Mon, Jan 11, 2021 at 11:45 AM Bharath Rupireddy wrote: > > On Sun, Jan 10, 2021 at 11:21 PM vignesh C wrote: > > On Sat, Jan 9, 2021 at 8:08 PM Bharath Rupireddy > > wrote: > > > I think this feature can be useful, in case a user has a lot of tables > > > to publish inside a schema. Having sa

Re: libpq compression

2021-01-12 Thread Konstantin Knizhnik
On 12.01.2021 4:20, Justin Pryzby wrote: On Mon, Jan 11, 2021 at 04:53:51PM +0300, Konstantin Knizhnik wrote: On 09.01.2021 23:31, Justin Pryzby wrote: I suggest that there should be an enum of algorithms, which is constant across all servers. They would be unconditionally included and not #

Re: Cirrus CI (Windows help wanted)

2021-01-12 Thread Andrew Dunstan
On 1/5/21 11:19 PM, Thomas Munro wrote: > > It seems we can make our own, either on-the-fly with caching, or > hosted somewhere, like this: > > https://cirrus-ci.org/guide/docker-builder-vm/#dockerfile-as-a-ci-environment > > OK, I got this working. There is some weirdness that I had to work

Re: Yet another fast GiST build

2021-01-12 Thread Heikki Linnakangas
On 10/12/2020 12:16, Andrey Borodin wrote: 9 дек. 2020 г., в 14:47, Andrey Borodin написал(а): 7 дек. 2020 г., в 23:56, Peter Geoghegan написал(а): On Mon, Dec 7, 2020 at 2:05 AM Andrey Borodin wrote: Here's version with tests and docs. I still have no idea how to print some useful informa

Re: Track replica origin progress for Rollback Prepared

2021-01-12 Thread Amit Kapila
On Tue, Jan 12, 2021 at 3:18 PM Ajin Cherian wrote: > > On Wed, Jan 6, 2021 at 11:56 PM Amit Kapila wrote: > > > > Now, let us see how the tests mentioned by me cover this code. In the > > first test (check that 2PC gets replicated to subscriber then ROLLBACK > > PREPARED), we do below on publish

Re: Single transaction in the tablesync worker?

2021-01-12 Thread Peter Smith
On Mon, Jan 11, 2021 at 3:32 PM Amit Kapila wrote: > > On Fri, Jan 8, 2021 at 8:20 AM Amit Kapila wrote: > > > > On Fri, Jan 8, 2021 at 7:14 AM Peter Smith wrote: > > > > > > FYI, I was able to reproduce this case in debugger. PSA logs showing > > > details. > > > > > > > Thanks for reproducing

Re: Key management with tests

2021-01-12 Thread Masahiko Sawada
On Tue, Jan 12, 2021 at 11:09 AM Bruce Momjian wrote: > > On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote: > > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > > Right, or ensure that the actual IV used is distinct (such as by using > > > another bit in the IV to distingui

Executing inet_server_addr/port() in parallel workers

2021-01-12 Thread Masahiko Sawada
Hi, While investigating a customer issue it's turned out that if a parallel worker executes inet_server_addr() and inet_server_port() the results are always null because MyProcPort is not set in parallel workers. We can reproduce it in all supported versions higher than 9.6. Here is an example: p

Re: Get memory contexts of an arbitrary backend process

2021-01-12 Thread torikoshia
v7 that fixes recent conflicts. It also changed the behavior of requestor when another requestor is already working for simplicity. In this case, v6 patch makes the requestor wait. v7 patch makes the requestor quit. Regards, -- Atsushi TorikoshiFrom f20e48d99f2770bfec275805185aa5ce08661fce Mon

Re: Single transaction in the tablesync worker?

2021-01-12 Thread Peter Smith
On Sat, Jan 9, 2021 at 5:44 PM Amit Kapila wrote: > > On Fri, Jan 8, 2021 at 2:55 PM Peter Smith wrote: > > > > On Fri, Jan 8, 2021 at 1:02 PM Hou, Zhijie > > wrote: > > > > > > > > 3. > > > + /* > > > +* To build a slot name for the sync work, we are limited to > > > NAMEDATALEN

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread japin
On Tue, 12 Jan 2021 at 19:32, Bharath Rupireddy wrote: > On Tue, Jan 12, 2021 at 4:47 PM Li Japin wrote: >> IIUC the logical replication only replicate the tables in publication, I >> think >> when the tables that aren't in publication should not be replicated. >> >> Attached the patch that fix

Re: Added schema level support for publication.

2021-01-12 Thread Rahila Syed
Hi Vignesh, I had a look at the patch, please consider following comments. On Thu, Jan 7, 2021 at 10:03 PM vignesh C wrote: > Hi, > > This feature adds schema option while creating publication. Users will > be able to specify one or more schemas while creating publication, > when the user speci

Re: Is it useful to record whether plans are generic or custom?

2021-01-12 Thread torikoshia
wrote in ISTM now that creating pg_stat_statements_xxx views both for generic andcustom plans is better than my PoC patch. On my second thought, it also makes pg_stat_statements too complicated compared to what it makes possible.. I'm also worrying that whether taking generic and custom pla

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Bharath Rupireddy
On Tue, Jan 12, 2021 at 4:47 PM Li Japin wrote: > IIUC the logical replication only replicate the tables in publication, I think > when the tables that aren't in publication should not be replicated. > > Attached the patch that fixes it. Thought? With that change, we don't get the behaviour that

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Bharath Rupireddy
On Tue, Jan 12, 2021 at 12:06 PM Amit Kapila wrote: > > Here's my analysis: > > 1) in the publisher, alter publication drop table successfully > > removes(PublicationDropTables) the table from the catalogue > > pg_publication_rel > > 2) in the subscriber, alter subscription refresh publication > >

Re: Logical Replication - behavior of ALTER PUBLICATION .. DROP TABLE and ALTER SUBSCRIPTION .. REFRESH PUBLICATION

2021-01-12 Thread Li Japin
On Jan 12, 2021, at 5:47 PM, japin mailto:japi...@hotmail.com>> wrote: On Tue, 12 Jan 2021 at 14:38, Amit Kapila wrote: On Tue, Jan 12, 2021 at 11:39 AM Bharath Rupireddy mailto:bharath.rupireddyforpostg...@gmail.com>> wrote: On Tue, Jan 12, 2021 at 9:05 AM Amit Kapila mailto:amit.kapil...@g

Re: list of extended statistics on psql

2021-01-12 Thread Tomas Vondra
On 1/12/21 2:57 AM, Tatsuro Yamada wrote: Hi Tomas, On 2021/01/09 9:01, Tomas Vondra wrote: ...> While working on that, I realized that 'defined' might be a bit ambiguous, I initially thought it means 'NOT NULL' (which it does not). I propose to change it to 'requested' instead. Tatsuro, do

Fix a typo in xlogreader.c

2021-01-12 Thread Masahiko Sawada
Hi, The function comment of RestoreBlockImage() seems not correct since it returns a boolean, not the buffer number. /* * Restore a full-page image from a backup block attached to an XLOG record. * * Returns the buffer number containing the page. */ bool RestoreBlockImage(XLogReaderSta

Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-12 Thread Thomas Munro
On Tue, Jan 12, 2021 at 8:27 PM Michael Paquier wrote: > Yeah, this rings a bell. I never went back to it even if the thing > looks rather clean at quick glance (not tested), but I may be able > to spend some cycles on that. I don't think that's critical enough > for a backpatch, so doing someth

Re: Implementing Incremental View Maintenance

2021-01-12 Thread Yugo NAGATA
Hi, Attached is the revised patch (v21) to add support for Incremental Materialized View Maintenance (IVM). In addition to some typos in the previous enhancement, I fixed a check to prevent a view from containing an expression including aggregates like sum(x)/sum(y) in this revision. Regards, Yu

  1   2   >