Re: shared-memory based stats collector - v67

2022-03-21 Thread Kyotaro Horiguchi
hink can be split out: > - Encapsulate "if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)" > style tests in a helper function. Then just the body needs to be changed, > rather than a lot of places needing such checks. > > Yep, that's it. I don't really see anything else that wouldn't be too > awkward. Would welcome suggestions!. I'm overwhelmed by the amout, but I'm going to look into them. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [PATCH] Accept IP addresses in server certificate SANs

2022-03-21 Thread Kyotaro Horiguchi
At Fri, 18 Mar 2022 16:38:57 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 17 Mar 2022 21:55:07 +, Jacob Champion > wrot> Thanks! .. and some nitpicks..(Sorry) > > fe-secure-common.c doesn't need netinet/in.h. > > > +++ b/src/include/utils/inet.h &

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-22 Thread Kyotaro Horiguchi
LSN is in futnre, wait for the record to come. 4-1. If end-LSN is in future, waits for new records. 5-1. If end-LSN is too past, error out? B. as a simple WAL dumper 1. If the whole region is filled with records, return them all. 2. If start-LSN is too past, starts from the first available record. 3-2. If start-LSN is in futnre, returns nothig. 4-2. If end-LSN is in future, ends with the last available record. 5-2. If end-LSN is too past, returns northing. 1 and 2 are uncontroversial. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-22 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 11:51:25 +0900 (JST), Kyotaro Horiguchi wrote in > The two places emit different outputs but the only difference is the > delimiter between two blockrefs. (By the way, the current code forgets > to insert a delimiter there). So even if the function took "boo

Re: [PATCH] Accept IP addresses in server certificate SANs

2022-03-22 Thread Kyotaro Horiguchi
h of the behavior (and edge cases) do you think we should detail > here? All of it? I tried to write out the doc part. What do you think about it? regards. -- Kyotaro Horiguchi NTT Open Source Software Center diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml index 3998b1781b..13

Re: shared-memory based stats collector - v67

2022-03-23 Thread Kyotaro Horiguchi
At Tue, 22 Mar 2022 11:56:40 +0900 (JST), Kyotaro Horiguchi wrote in > Docs.. Yeah I'll try it. This is the first cut, based on the earlier patchset. monitoring.sgml: >When using the statistics to monitor collected data, it is important I failed to read this clearly. I m

Re: shared-memory based stats collector - v67

2022-03-23 Thread Kyotaro Horiguchi
rst place, I'm not sure it is valid that a standby from a cold backup takes over the stats from the primary. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Support isEmptyStringInfo

2022-03-23 Thread Kyotaro Horiguchi
nly ".len == 0" doesn't help. Already in many places we pull out buf.data to use it separately from buf, we have a dozen of instances of "buf.len (<|>|<=|>=) " and even "buf.data[buf.len - 1] == '\n'" About read-easiness, isEmptyStringInfo(str) slightly spins my eyes than str->len == 0. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Huge memory consumption on partitioned table with FKs

2020-11-30 Thread Kyotaro Horiguchi
At Mon, 30 Nov 2020 21:03:45 -0300, Alvaro Herrera wrote in > On 2020-Nov-26, Kyotaro Horiguchi wrote: > > > This shares RI_ConstraintInfo cache by constraints that shares the > > same parent constraints. But you forgot that the cache contains some > > members that can

Re: Huge memory consumption on partitioned table with FKs

2020-12-02 Thread Kyotaro Horiguchi
At Wed, 2 Dec 2020 22:02:50 +0900, Amit Langote wrote in > Hello, > > On Tue, Dec 1, 2020 at 9:40 AM Kyotaro Horiguchi > wrote: > > > > At Mon, 30 Nov 2020 21:03:45 -0300, Alvaro Herrera > > wrote in > > > On 2020-Nov-26, Kyotaro Horiguchi wrote: >

Re: Huge memory consumption on partitioned table with FKs

2020-12-02 Thread Kyotaro Horiguchi
At Thu, 3 Dec 2020 12:27:53 +0900, Amit Langote wrote in > Hello, > > On Thu, Dec 3, 2020 at 10:15 AM Kyotaro Horiguchi > wrote: > > At Wed, 2 Dec 2020 22:02:50 +0900, Amit Langote > > wrote in > > > Hmm, I don't see what the problem is here, because it

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Kyotaro Horiguchi
At Thu, 3 Dec 2020 16:41:45 +0900, Amit Langote wrote in > On Thu, Dec 3, 2020 at 2:29 PM Kyotaro Horiguchi > wrote: > > At Thu, 3 Dec 2020 12:27:53 +0900, Amit Langote > > wrote in > > > On Thu, Dec 3, 2020 at 10:15 AM Kyotaro Horiguchi > > > wrote: >

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Kyotaro Horiguchi
At Thu, 03 Dec 2020 17:13:16 +0900 (JST), Kyotaro Horiguchi wrote in me> I agree that plans are rather large but the sharable part of the me> RI_ConstraintInfos is 536 bytes, I'm not sure it is small enough me> comparing to the plans. But that has somewhat large footprint.

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Kyotaro Horiguchi
At Thu, 3 Dec 2020 21:40:29 +0900, Amit Langote wrote in > On Thu, Dec 3, 2020 at 5:13 PM Kyotaro Horiguchi > wrote: > > At Thu, 3 Dec 2020 16:41:45 +0900, Amit Langote > > wrote in > > > Maybe I misread but I think you did in your email dated Dec 1 where you >

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Kyotaro Horiguchi
truct (RI_ConstraintInfo) is now > undocumented. What is the relationship between those two structs? I > see that they have pointers to each other, but I think the relationship > should be documented more clearly. I'm not sure the footprint of this patch worth doing but here is a bit more

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-03 Thread Kyotaro Horiguchi
> + * more details. Otherwise, if the conditions for optimization are > > not > > + * met, the buffer pool is sequentially scanned so that no > > buffers are > > + * left behind. > > > > I'm not confident on it, but it seems somewhat obscure. How about > > something like this? > > > > We mustn't leave a buffer for the relations to be dropped. We invalidate > > buffer blocks by locating using BufTableLookup() when we assure that we > > know up to what page of every fork we possiblly have a buffer for. We can > > know that by the "cached" flag returned by smgrblocks. It currently gets > > true > > only while recovery. See > > smgrnblocks() and smgrextend(). Otherwise we scan the whole buffer pool to > > find buffers for the relation, which is slower when a small part of buffers > > are > > to be dropped. > > Followed your advice and modified it a bit. > > I have changed the status to "Needs Review". > Feedbacks are always welcome. > > Regards, > Kirk Jamison regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-03 Thread Kyotaro Horiguchi
ESHOLD) !| do_fullscan = false; !| } !| !| if (!do_fullscan) !| { | for (i = 0; i < n; i++) | { | /* | * If block to drop is valid, drop the buffers of the fork. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Kyotaro Horiguchi
onstraint_root_id stores constraint_id if it is not a partition? That change makes the patch a bit simpler. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Wrong statistics for size of XLOG_SWITCH during pg_waldump.

2020-12-03 Thread Kyotaro Horiguchi
D, the spurious line is shown. The first one is for XLOG_HEAP2_VISIBLE and the latter is for XLOG_HEAP_HOT_UPDATE, that is, both of which are not for XLOG_SWITCH.. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Is it useful to record whether plans are generic or custom?

2020-12-03 Thread Kyotaro Horiguchi
_plans or pg_store_plans.. The former is faster but may lose plans, the latter doesn't lose plans but slower. I feel that we'd beter consider simpler feature if we are intendeng it to be a part of a contrib module, > Yamada-san, > > Do you think it's effective just distinguish between generic > and custom plans? > > Regards, regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
has already been called for. The most common way to make it happen was INSERTs *before* the truncating transaction starts. It may be a SELECT on a hot-standby. Sorry for the confusing expression. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
At Tue, 08 Dec 2020 09:45:53 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila > wrote in > > Hmm, how is it possible if Insert is done before Truncate? The insert > > should happen in old RelFileNode only. I have verified by add

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
I'm out of it more than usual.. At Tue, 08 Dec 2020 09:45:53 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 7 Dec 2020 17:18:31 +0530, Amit Kapila > wrote in > > On Mon, Dec 7, 2020 at 12:32 PM k.jami...@fujitsu.com > > wrote: > > > > > > On Frid

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Kyotaro Horiguchi
At Tue, 8 Dec 2020 01:16:00 +0900, Amit Langote wrote in > Hi Alvaro, > > On Mon, Dec 7, 2020 at 23:48 Alvaro Herrera wrote: > > > On 2020-Dec-07, Amit Langote wrote: > > > > > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi > > > wrot

Re: pg_rewind race condition just after promotion

2020-12-07 Thread Kyotaro Horiguchi
changes any successful behavior and it just saves the failure case so +1 for back-patching. > Thoughts? > > [1] > https://www.postgresql.org/message-id/20150428180253.GU30322%40tamriel.snowman.net regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread Kyotaro Horiguchi
At Tue, 8 Dec 2020 08:08:25 +0530, Amit Kapila wrote in > On Tue, Dec 8, 2020 at 7:24 AM Kyotaro Horiguchi > wrote: > > We drop > > buffers for the old relfilenode on truncation anyway. > > > > What I did is: > > > > a: Create a physical replication

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-08 Thread Kyotaro Horiguchi
closed midst of a session and smgr doesn't offer such persistence. In the first place smgr doesn't seem to be the place to store such persistent information. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

A failure of standby to follow timeline switch

2020-12-09 Thread Kyotaro Horiguchi
ects the wrong behavior. The second small patch fixes it. In the first patch, the test added to 001_stream_rep.pl involves two copied functions related to server-log investigation from 019_repslot_limit.pl. regards. -- Kyotaro Horiguchi NTT Open Source Software Cent

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-09 Thread Kyotaro Horiguchi
At Wed, 9 Dec 2020 16:27:30 +0530, Amit Kapila wrote in > On Wed, Dec 9, 2020 at 6:32 AM Kyotaro Horiguchi > wrote: > > > > At Tue, 8 Dec 2020 16:28:41 +0530, Amit Kapila > > wrote in > > > On Tue, Dec 8, 2020 at 12:13 PM tsunakawa.ta...@fujitsu.com &

Re: pg_shmem_allocations & documentation

2020-12-10 Thread Kyotaro Horiguchi
I'd like to propose instead to add an explanation why it is not exposed for anonymous allocations, like the column allocated_size. > Benoit > > [1] https://www.postgresql.org/docs/13/view-pg-shmem-allocations.html > [2] > https://www.postgresql.org/message-id/flat/20140504114417

Re: pg_waldump error message fix

2020-12-10 Thread Kyotaro Horiguchi
(uint32) xlogreader_state->currRecPtr, > errormsg); > > XLogReaderFree(xlogreader_state); currRecPtr is a private member of the struct (see the definition of the struct XLogReaderState). We should instead use EndRecPtr outside xlog reader. regardes. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_waldump error message fix

2020-12-11 Thread Kyotaro Horiguchi
At Fri, 11 Dec 2020 14:21:57 +0900, Michael Paquier wrote in > On Fri, Dec 11, 2020 at 01:30:16PM +0900, Kyotaro Horiguchi wrote: > > currRecPtr is a private member of the struct (see the definition of > > the struct XLogReaderState). We should instead use EndRecPtr outside

Re: pg_shmem_allocations & documentation

2020-12-11 Thread Kyotaro Horiguchi
At Fri, 11 Dec 2020 14:42:45 +0900, Michael Paquier wrote in > On Fri, Dec 11, 2020 at 11:00:58AM +0900, Kyotaro Horiguchi wrote: > > Although we could just rip some words off, I'd like to propose instead > > to add an explanation why it is not exposed for anonymous alloc

Re: pg_waldump error message fix

2020-12-11 Thread Kyotaro Horiguchi
At Fri, 11 Dec 2020 17:19:33 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 11 Dec 2020 14:21:57 +0900, Michael Paquier > wrote in > > On Fri, Dec 11, 2020 at 01:30:16PM +0900, Kyotaro Horiguchi wrote: > > > currRecPtr is a private member of the struct (see the de

Re: pg_waldump error message fix

2020-12-13 Thread Kyotaro Horiguchi
y happens on > the first page of the segment. > > Perhaps we could simply omit the LSN in the pg_waldump message. Yeah, I had the same feeling. At least, the two LSNs in the message under discussion are simply redundant. So +1 to just remove the LSN at the caller site. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_waldump error message fix

2020-12-13 Thread Kyotaro Horiguchi
ord I confirmed 0668719801 still works under the intended context using the steps shown in [1]. [1]: https://www.postgresql.org/message-id/flat/CACJqAM3xVz0JY1XFDKPP%2BJoJAjoGx%3DGNuOAshEDWCext7BFvCQ%40mail.gmail.com regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From d54531aa2774bad7e426cc1

Re: Asynchronous Append on postgres_fdw nodes.

2020-12-13 Thread Kyotaro Horiguchi
At Sat, 12 Dec 2020 18:25:57 +0900, Etsuro Fujita wrote in > On Fri, Nov 20, 2020 at 3:51 PM Kyotaro Horiguchi > wrote: > > At Tue, 17 Nov 2020 18:56:02 +0900, Etsuro Fujita > > wrote in > > > * In Robert's patch [1] (and Horiguchi-san's, whic

Re: Asynchronous Append on postgres_fdw nodes.

2020-12-14 Thread Kyotaro Horiguchi
At Sat, 12 Dec 2020 19:06:51 +0900, Etsuro Fujita wrote in > On Fri, Nov 20, 2020 at 8:16 PM Kyotaro Horiguchi > wrote: > > I looked through the nodeAppend.c and postgres_fdw.c part and those > > are I think the core of this patch. > > Tha

Some error messages are omitted while recovery.

2020-12-14 Thread Kyotaro Horiguchi
At Mon, 14 Dec 2020 16:48:05 +0900, Michael Paquier wrote in > On Mon, Dec 14, 2020 at 11:34:51AM +0900, Kyotaro Horiguchi wrote: > > Apart from this issue, while checking that, I noticed that if server > > starts having WALs from a server of a different systemid, the server

Re: Add Information during standby recovery conflicts

2020-12-14 Thread Kyotaro Horiguchi
002-Log-the-standby-recovery-conflict-waits.patch is applied. Ah. I was just writing a comment about that. I haven't looked it closer but it looks good to me. By the way doesn't it contains a simple fix of a comment for the base patch? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: archive status ".ready" files may be created too early

2020-12-15 Thread Kyotaro Horiguchi
or my latest attempt. > > On 10/13/20, 5:07 PM, "Kyotaro Horiguchi" wrote: > > F0F1 > > A F B > > |-|-|-| > >seg Xseg X+1 seg X+2 > > > > Matsumura-san has a concern about t

Re: archive status ".ready" files may be created too early

2020-12-15 Thread Kyotaro Horiguchi
At Tue, 15 Dec 2020 19:32:57 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 14 Dec 2020 18:25:23 +, "Bossart, Nathan" > wrote in > > I wonder if these are safe assumptions to make. For your example, if > > we've written record B to the WAL buffers, bu

Re: Add Information during standby recovery conflicts

2020-12-15 Thread Kyotaro Horiguchi
At Tue, 15 Dec 2020 15:40:03 +0900, Fujii Masao wrote in > > > On 2020/12/15 12:04, Kyotaro Horiguchi wrote: > > [40509:startup] DETAIL: Conflicting processes: 41171, 41194. ... > > I'm not sure, but it seems to me at least the period is unnecessary > > here

Re: Add Information during standby recovery conflicts

2020-12-15 Thread Kyotaro Horiguchi
At Wed, 16 Dec 2020 12:08:31 +0900, Masahiko Sawada wrote in On Wed, Dec 16, 2020 at 11:22 AM Kyotaro Horiguchi wrote: > > At Tue, 15 Dec 2020 15:40:03 +0900, Fujii Masao wrote in > > > > > > On 2020/12/15 12:04, Kyotaro Horiguchi wrote: > > > [40509:startu

Re: Add Information during standby recovery conflicts

2020-12-15 Thread Kyotaro Horiguchi
At Wed, 16 Dec 2020 14:49:25 +0900 (JST), Kyotaro Horiguchi wrote in > > > > Conflicting processes are 41171, 41194. > > > > Conflicting processes are: 41171, 41194. > > Or I came up with the following after scanning throught the tree. > > | Some processe

Re: Use standard SIGHUP and SIGTERM handlers in autoprewarm module

2020-12-15 Thread Kyotaro Horiguchi
d > 113d3591b8. > > [1] > https://www.postgresql.org/message-id/a802f1c0-58d9-bd3f-bc3e-bdad54726...@oss.nttdata.com As the result, I agree to revert them. But I think we need to add a comment for the reason we don't use MyLatch for recovery-wakeup after reverting them. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: archive status ".ready" files may be created too early

2020-12-17 Thread Kyotaro Horiguchi
At Thu, 17 Dec 2020 22:20:35 +, "Bossart, Nathan" wrote in > On 12/15/20, 2:33 AM, "Kyotaro Horiguchi" wrote: > > You're right in that regard. There's a window where partial record is > > written when write location passes F0 after insertion loc

Re: archive status ".ready" files may be created too early

2020-12-17 Thread Kyotaro Horiguchi
At Wed, 16 Dec 2020 11:01:20 +0900 (JST), Kyotaro Horiguchi wrote in > - Record the beginning LSN of the first cross-seg record and the end > LSN of the last cross-seg recrod in a consecutive segments bonded by > cross-seg recrods. Spcifically X and Y below. > >X

Re: Minor documentation error regarding streaming replication protocol

2020-12-18 Thread Kyotaro Horiguchi
client encoding is valid as server encoding. (since database encoding of walsender is always SQL_ASCII) I'm not sure it is the expected behavior. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Strange behavior with polygon and NaN

2020-12-21 Thread Kyotaro Horiguchi
At Tue, 01 Dec 2020 10:03:42 -0500, Tom Lane wrote in > I think it should be "needs review" now. Conflicted with some commit(s) uncertain to me. Rebased. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From 26d9edeccf3f3111a7e0ff049aa6e3ba3746dce9 Mon Sep 17 00

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-21 Thread Kyotaro Horiguchi
ly planning to > > write a theory why is this optimization safe and here is what I can > > come up with: > > Thank you, that's fluent and easier to understand. +1 regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-21 Thread Kyotaro Horiguchi
m I missing something? I think exactly like that. If we acquire the bufHdr lock before releasing the partition lock, that steal doesn't happen but it doesn't seem good as a locking protocol. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-21 Thread Kyotaro Horiguchi
LockBufHdr(); > LWLockRelease(); > InvalidateBuffer(); > > I think both are okay. If I must choose either, I kind of prefer 1), because > LWLockRelease() could take longer time to wake up other processes waiting on > the lwlock, which is not very good to do while holdin

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread Kyotaro Horiguchi
se to goto's but in this case the outer loop can break on the same condition with the inner loop, since cached is true whenever the inner loop runs to the end. It is needed to initialize the variable cache with true, instead of false, though. The same pattern is seen in the tree. Regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: In-placre persistance change of a relation

2020-12-24 Thread Kyotaro Horiguchi
Thanks for the comment! Sorry for the late reply. At Fri, 4 Dec 2020 07:49:22 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Kyotaro Horiguchi > > > No, not really. The issue is more around what happens if we crash > > > part way through. At crash rec

Re: In-placre persistance change of a relation

2020-12-24 Thread Kyotaro Horiguchi
Hello. At Thu, 24 Dec 2020 17:02:20 +0900 (JST), Kyotaro Horiguchi wrote in > The patch is attached to the next message. The reason for separating this message is that I modified this so that it could solve another issue. There's a complain about orphan files after crash. [1]

Re: A failure of standby to follow timeline switch

2020-12-24 Thread Kyotaro Horiguchi
l (Wstat: 65280 Tests: 37 Failed: 0) > Non-zero exit status: 255 > Parse errors: Bad plan. You planned 38 tests but ran 37. > Files=1, Tests=37, 10 wallclock secs ( 0.03 usr 0.00 sys + 3.64 cusr 2.05 > csy > s = 5.72 CPU) > Result: FAIL > make: *** [Makefile:19: check] Error 1 I can't guess what happenened on your environment.. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Deadlock between backend and recovery may not be detected

2020-12-24 Thread Kyotaro Horiguchi
imeout) + { ... + } + +cleanup: It is eqivalent to + if (!got_standby_lock_timeout && got_standby_deadlock_timeout) + { ... + } Is there any reason for the goto? regards. -- Kyotaro Horiguchi NTT Open Source Software Center

pg_waldump: Limit the number of lines shown at the start

2020-12-25 Thread Kyotaro Horiguchi
ed as attached. Any thoughts, opinions? regards. -- Kyotaro Horiguchi NTT Open Source Software Center diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index 31e99c2a6d..dacdb600bf 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@

Re: A failure of standby to follow timeline switch

2021-01-03 Thread Kyotaro Horiguchi
At Sat, 26 Dec 2020 02:15:06 +0900, Fujii Masao wrote in > > > On 2020/12/25 12:03, Kyotaro Horiguchi wrote: > > Thank you for looking this. > > At Thu, 24 Dec 2020 15:33:04 +0900, Fujii Masao > > wrote in > >> When I applied two patches in the master b

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-01-04 Thread Kyotaro Horiguchi
bove, we don't need to record about the database directory. By the way the name XLogLogMiss.. is somewhat confusing. How about XLogReportMissingDir (named after report_invalid_page). regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: A failure of standby to follow timeline switch

2021-01-05 Thread Kyotaro Horiguchi
At Mon, 4 Jan 2021 19:00:21 +0900, Fujii Masao wrote in > > > On 2021/01/04 12:06, Kyotaro Horiguchi wrote: > > At Sat, 26 Dec 2020 02:15:06 +0900, Fujii Masao > > wrote in > >> > >> On 2020/12/25 12:03, Kyotaro Horiguchi wrote: > > The attached i

Re: Cirrus CI (Windows help wanted)

2021-01-05 Thread Kyotaro Horiguchi
you want in a way that captures the resulting image using > Docker voodoo, so that you get fast startup times). Or something. > > [1] > https://www.postgresql.org/message-id/flat/CA%2BhUKG%2By_SHVQcU3CPokmJxuHp1niebCjq4XzZizf8SR9ZdQRQ%40mail.gmail.com regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Deadlock between backend and recovery may not be detected

2021-01-05 Thread Kyotaro Horiguchi
At Tue, 5 Jan 2021 15:26:50 +0900, Fujii Masao wrote in > > > On 2020/12/25 13:16, Kyotaro Horiguchi wrote: > > At Wed, 23 Dec 2020 21:42:47 +0900, Fujii Masao > > wrote in > >> you. Attached > >> is the updated of the patch. What about this versio

Re: A failure of standby to follow timeline switch

2021-01-05 Thread Kyotaro Horiguchi
At Tue, 05 Jan 2021 17:26:02 +0900 (JST), Kyotaro Horiguchi wrote in > Thanks. The attached is the revised patchset. It is not applicable to PG13 due to wording changes. This is an applicable all-in-one version to PG13. regards. -- Kyotaro Horiguchi NTT Open Source Software Center &g

Re: Corruption during WAL replay

2021-01-06 Thread Kyotaro Horiguchi
t(). We could > > have a similar mechanism to wait out the truncations before *completing* a > > checkpoint. > > What I outlined earlier *is* essentially a way to do so, by preventing > checkpointing from finishing the buffer scan while a dangerous state > exists. Seems

Re: A failure of standby to follow timeline switch

2021-01-06 Thread Kyotaro Horiguchi
At Thu, 7 Jan 2021 11:55:33 +0900, Fujii Masao wrote in > > > On 2021/01/05 17:26, Kyotaro Horiguchi wrote: > > At Mon, 4 Jan 2021 19:00:21 +0900, Fujii Masao > > wrote in > >> > >> > >> On 2021/01/04 12:06, Kyotaro Horiguchi wrote: >

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-01-07 Thread Kyotaro Horiguchi
he buffers by doing lookups in BufMapping table. > > This improves the performance by more than 100 times in many cases > > when several small tables (tested with 1000 relations) are truncated > > and where the server is configured with a large value of shared > > buffers (greater than 100GB)." > > Thank you for taking a look at the results of the tests. And it's also > consistent with the results from Tang too. > The commit message LGTM. +1. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Add Information during standby recovery conflicts

2021-01-07 Thread Kyotaro Horiguchi
+* Emit the log message if recovery conflict on buffer pin was resolved but +* the startup process waited longer than deadlock_timeout for it. The first line is beyond the 80th column. LGTM other than the above. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: In-placre persistance change of a relation

2021-01-07 Thread Kyotaro Horiguchi
At Fri, 25 Dec 2020 09:12:52 +0900 (JST), Kyotaro Horiguchi wrote in > Hello. > > At Thu, 24 Dec 2020 17:02:20 +0900 (JST), Kyotaro Horiguchi > wrote in > > The patch is attached to the next message. > > The reason for separating this message is that I modified

Re: Disable WAL logging to speed up data loading

2021-01-07 Thread Kyotaro Horiguchi
incompatible LSNs, LOGGED tables always uses real LSNs, maintained by that type of WAL record while wal_level = minimal. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: In-placre persistance change of a relation

2021-01-08 Thread Kyotaro Horiguchi
At Fri, 08 Jan 2021 14:47:05 +0900 (JST), Kyotaro Horiguchi wrote in > This version RelationChangePersistence() is changed not to choose > in-place method for indexes other than btree. It seems to be usable > with all kind of indexes other than Gist, but at the mement it applies

Re: parse mistake in ecpg connect string

2021-02-11 Thread Kyotaro Horiguchi
violated in the doc, but maybe no > > one take care. > > Pushed with a little more work on the documentation. Thanks for committing this (and further update of the document). regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_cryptohash_final possible out-of-bounds access (per Coverity)

2021-02-11 Thread Kyotaro Horiguchi
n.) Total 9/19 places. I think at least pg_checksum_final() should take the buffer length. I'm not sure about px_md_finish() and hmac_md_finish().. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Is Recovery actually paused?

2021-02-11 Thread Kyotaro Horiguchi
one seems on my side, I withdraw the proposal. > I have no futher comments on the v13 patch, too. Also, I agree with > Robert Haas's suggestions. Yeah, look reasonable. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Possible dereference after null check (src/backend/executor/ExecUtils.c)

2021-02-11 Thread Kyotaro Horiguchi
ble entry). The only cases where both are 0 and NULL are trigger-use, which is unrelated to the code path. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Possible dereference null return (src/backend/replication/logical/reorderbuffer.c)

2021-02-11 Thread Kyotaro Horiguchi
the top transaction. If the top transaction actualy did not exist, it's rather the correct behavior to cause SEGV, than creating a bogus rbtxn. THus it is wrong to set create=true and create_as_top=true. We could change the assertion like Assert (txn && txn->base_snapshot) to make things clearer. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Asynchronous Append on postgres_fdw nodes.

2021-02-12 Thread Kyotaro Horiguchi
ion? I understand that the currently intending restriction is one connection accepts at most one FDW-scan node. This looks somethig different... (Sorry, time's up for now.) regards. -- Kyotaro Horiguchi NTT Open Source Software Center

ERROR: "ft1" is of the wrong type.

2021-02-16 Thread Kyotaro Horiguchi
check the value itself is correct for the callers. regareds. -- Kyotaro Horiguchi NTT Open Source Software Center diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index a149ca044c..cf572a7c58 100644 --- a/src/backend/commands/tablecmds.c +++ b/s

Re: Is it worth accepting multiple CRLs?

2021-02-16 Thread Kyotaro Horiguchi
to be +server's certificate. If neither sslcrl sslcrldir is set, this +setting is assumed to be ~/.postgresql/root.crl. See And added a line for the new variable in postgresql.conf.sample. regards. -- Kyotaro Horiguchi NTT Open Source Software Ce

Re: Asynchronous Append on postgres_fdw nodes.

2021-02-17 Thread Kyotaro Horiguchi
could add it to > postgres_fdw later maybe as the server/table options. Sorry for the > back and forth. That was the natural extension from non-aggresive prefetching. However, maybe we can live without that since if some needs more speed, it is enought to give every remote tables a dedicate connection. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Is it worth accepting multiple CRLs?

2021-02-18 Thread Kyotaro Horiguchi
Thanks for committing this! At Thu, 18 Feb 2021 08:24:23 +0100, Peter Eisentraut wrote in > On 2021-02-17 05:05, Kyotaro Horiguchi wrote: > > The commit fe61df7f82 shot down this. > > This patch allows a new GUC ssl_crl_dir and a new libpq connection > > option ssl

Re: ERROR: "ft1" is of the wrong type.

2021-02-18 Thread Kyotaro Horiguchi
At Thu, 18 Feb 2021 16:27:23 +0900, Michael Paquier wrote in > On Tue, Feb 16, 2021 at 06:14:15PM +0900, Kyotaro Horiguchi wrote: > > The attached is just fixing that. I tried to make it generic but > > didn't find a clean and translatable way. > > > > Also

Re: Printing LSN made easy

2021-02-18 Thread Kyotaro Horiguchi
he parameter, unary operators are seldom (or never) applied to LSN. I think there's no need to fear about other (modifying) expressions, too. As a double-checking, I checked that the patch covers all output by '%X/%X' and " ?>> ?32)" that are handling LSNs, and there

Re: ERROR: "ft1" is of the wrong type.

2021-02-19 Thread Kyotaro Horiguchi
At Thu, 18 Feb 2021 17:17:37 +0900 (JST), Kyotaro Horiguchi wrote in > I can add some regression tests to cover all the live cases. That > could reveal no-longer-used combinations. The attached is that. ATT_VIEW is used for "CREATE OR REPLACE view" and checked ag

Re: Is Recovery actually paused?

2021-02-23 Thread Kyotaro Horiguchi
ase here. pg_get_replication_slots() is going that direction and apply_dispatch() is taking a slightly different way. Anyway I think that we can take away the default case. regard. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: pg_attribute.attname inconsistency when renaming primary key columns

2021-02-23 Thread Kyotaro Horiguchi
d) JOIN pg_attribute ar ON (ar.attrelid = cr.oid AND ar.attnum = ANY(i.indkey)) WHERE ci.relnamespace = 'public'::regnamespace; indexname | indcolname | relname | relattname | attnum ---++-++ bar_pk| foo_id | bar | bar_id |

Re: Is Recovery actually paused?

2021-02-24 Thread Kyotaro Horiguchi
At Wed, 24 Feb 2021 13:15:27 +0530, Dilip Kumar wrote in > On Wed, Feb 24, 2021 at 12:39 PM Kyotaro Horiguchi > wrote: > > > > At Tue, 23 Feb 2021 12:03:32 +0530, Dilip Kumar > > wrote in > > > On Fri, Feb 12, 2021 at 3:26 AM Robert Haas wrote:

Re: Is Recovery actually paused?

2021-02-24 Thread Kyotaro Horiguchi
At Wed, 24 Feb 2021 17:56:41 +0900 (JST), Kyotaro Horiguchi wrote in > At Wed, 24 Feb 2021 13:15:27 +0530, Dilip Kumar wrote > in > > > After the renaming of the function, the following structure looks > > > simpler and more natural. > > > &g

Re: Is Recovery actually paused?

2021-02-24 Thread Kyotaro Horiguchi
At Wed, 24 Feb 2021 15:25:57 +0530, Dilip Kumar wrote in > On Wed, Feb 24, 2021 at 2:26 PM Kyotaro Horiguchi > wrote: > > I should have took the meaning of "confirm" wrongly. I took that as > > "somehow determine if the recovery is to be paused". If t

Re: Is Recovery actually paused?

2021-02-24 Thread Kyotaro Horiguchi
At Thu, 25 Feb 2021 09:49:15 +0530, Dilip Kumar wrote in > On Thu, Feb 25, 2021 at 6:52 AM Kyotaro Horiguchi > wrote: > > > Recently we have mildly changed to the direction to utilize the > > compiler warning about enum coverage in switch struct. (Maybe we need > &

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
e assertion was inteded to be placed after subtracting size, and the operator > is inteded to be >=. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
At Fri, 26 Feb 2021 13:35:26 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 25 Feb 2021 21:04:00 -0300, Álvaro Herrera > wrote in > > This assert I added? > > > > > +static bool > > > +pqTraceMaybeBreakLine(int size, PGconn *conn) > > &g

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
nt_encoding\x00UTF8\x00\x00' < :::Invalid Protocol < :::Invalid Protocol 1. It seems to have desynced. 2. ":::Invalid Protocol" looks somewhat odd? 3. ":::Invalid Protocol" lines missing a timestamp. - type "select 1;" I haven't looked

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
(Is what I'm drinking now decafe?) At Fri, 26 Feb 2021 14:33:26 +0900 (JST), Kyotaro Horiguchi wrote in > - Tweaked psql to enable tracing. (attached) > > - Connect to a server using SSL. > > - Restart the server. -> This is a mistake. Just restaring the se

Re: libpq debug log

2021-02-25 Thread Kyotaro Horiguchi
conn->be_msg->logCursor = conn->inCursor - conn->inStart; This inhibits logCursor being updated. What is worse, I find that logCursor movement is quite dubious. pqTraceOutputBeByte1 donsn't move logCursor for LOG_FIRST_BYTE pqTraceOutputBeInt doesn't move logCursor if the messages *does* end. So it repeats the same message if the previous message consists of . regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Is Recovery actually paused?

2021-02-26 Thread Kyotaro Horiguchi
At Thu, 25 Feb 2021 13:22:53 +0530, Dilip Kumar wrote in > On Thu, Feb 25, 2021 at 12:42 PM Kyotaro Horiguchi > wrote: > > The latest version applies (almost) cleanly to the current master and > > works fine. > > I don't have further comment on this. > > >

Re: libpq debug log

2021-02-26 Thread Kyotaro Horiguchi
At Fri, 26 Feb 2021 16:12:39 +0900 (JST), Kyotaro Horiguchi wrote in > This inhibits logCursor being updated. What is worse, I find that > logCursor movement is quite dubious. Using (inCursor - inStart) as logCursor doesn't work correctly if tracing state desyncs. Once desync happ

Re: libpq debug log

2021-02-26 Thread Kyotaro Horiguchi
At Fri, 26 Feb 2021 17:30:32 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 26 Feb 2021 16:12:39 +0900 (JST), Kyotaro Horiguchi > wrote in > > This inhibits logCursor being updated. What is worse, I find that > > logCursor movement is quite dubious. > > Using

Re: PITR promote bug: Checkpointer writes to older timeline

2021-03-02 Thread Kyotaro Horiguchi
* enable XLogInsertAllowed. (This also ensures ThisTimeLineID is |* initialized, which we need here and in AdvanceXLInsertBuffer.) |*/ | if (flags & CHECKPOINT_END_OF_RECOVERY) | LocalSetXLogInsertAllowed(); It seems to e suficcient to recover ThisTimeL

<    1   2   3   4   5   6   7   8   9   10   >