Re: Failed transaction statistics to measure the logical replication progress

On Wed, Aug 4, 2021 at 12:17 PM Amit Kapila wrote: > > On Wed, Aug 4, 2021 at 6:19 AM Masahiko Sawada wrote: > > > > On Tue, Aug 3, 2021 at 6:11 PM Amit Kapila wrote: > > > > > > I was trying to think based on similar counters in pg_stat_database > > > but if you think there is a value in showin

Re: Division by zero error in to_char(num, '9.9EEEE')

On Fri, 30 Jul 2021 at 08:26, Dean Rasheed wrote: > > SELECT to_char(1.2e-1002, '9.9'); -- fails > ERROR: division by zero > > I think the simplest > solution is to just introduce a new local function, as in the attached > patch. This directly constructs 10^n, for integer n, which is pretty >

Possible dependency issue in makefile

Hi all, I am building libecpg 13.1 but 13.3 behaves in the same manner and my build fails with: path.c: In function 'get_html_path': path.c:787:38: error: 'HTMLDIR' undeclared (first use in this function) 787 | make_relative_path(ret_path, HTMLDIR, PGBINDIR, my_exec_path); |

RE: [BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada > On Mon, Aug 2, 2021 at 10:52 PM houzj.f...@fujitsu.com > wrote: > > > > > > Hi hackers, > > > > When testing some other logical replication related patches, I found > > two unexpected behaviours about ALTER SUBSCRIPTION ADD/DROP > PUBLICATION

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

ср, 4 авг. 2021 г. в 07:41, Greg Nancarrow : > On Wed, Aug 4, 2021 at 3:21 AM Robert Haas wrote: > > > >The idea I sort of had floating around in my mind is a little > >different than what Greg has implemented. I was thinking that we could > >just skip SerializeSnapshot and the corresponding shm_

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

On Wed, Aug 4, 2021 at 7:55 PM Pavel Borisov wrote: > >> > Greg, thanks for the fast response! I suppose that a check for > IsolationUsesXactSnapshot() is also useful in a GetTransactionSnapshot for > the correct processing of a case with NULL transaction snapshot. > This corrects mentioned chec

Re: Lowering the ever-growing heap->pd_lower

On Wed, 4 Aug 2021 at 03:51, Peter Geoghegan wrote: > > We generate an FPI the first time a page is modified after a > checkpoint. The FPI consists of the page *after* it has been modified. In that case, I misremembered when FPIs were written with relation to checkpoints. Thanks for reminding me.

Re: Added schema level support for publication.

On Tue, Aug 3, 2021 at 8:38 PM vignesh C wrote: > > Thanks for reporting this, this is fixed in the v18 patch attached. > I have started looking into this patch and below are some initial comments. 1. + /* Fetch publication name and schema oid from input list */ + schemaname = strVal(linitial(ob

Re: [HACKERS] logical decoding of two-phase transactions

On Wed, Aug 4, 2021 at 6:51 AM tanghy.f...@fujitsu.com wrote: > > On Tuesday, August 3, 2021 6:03 PM vignesh C wrote: > > > > On Tue, Aug 3, 2021 at 12:32 PM Amit Kapila wrote: > > > > > > On Tue, Aug 3, 2021 at 6:17 AM Peter Smith wrote: > > > > > > > > Please find attached the latest patch set

Re: Lowering the ever-growing heap->pd_lower

On Wed, 4 Aug 2021 at 02:43, Peter Geoghegan wrote: > > On Mon, Aug 2, 2021 at 11:57 PM Simon Riggs > wrote: > > 2. Reduce number of line pointers to 0 in some cases. > > Matthias - I don't think you've made a full case for doing this, nor > > looked at the implications. > > The comment clearly s

Re: [BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

On Wed, Aug 4, 2021 at 5:06 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada > > On Mon, Aug 2, 2021 at 10:52 PM houzj.f...@fujitsu.com > > wrote: > > > > > > > > > Hi hackers, > > > > > > When testing some other logical replication related patches, I fou

Re: speed up verifying UTF-8

I wrote: > If we have only 16 bytes in the input, it still seems to be faster to use SSE, even though it's called through a function pointer on x86. I didn't test the DFA path, but I don't think the conclusion would be different. I'll include the 16 threshold next time I need to update the patch.

Re: Skipping logical replication transactions on subscriber side

On Tue, Aug 3, 2021 at 7:54 PM vignesh C wrote: > > On Tue, Aug 3, 2021 at 12:20 PM Masahiko Sawada wrote: > > > > On Mon, Aug 2, 2021 at 12:21 PM Amit Kapila wrote: > > > > > > On Mon, Aug 2, 2021 at 7:45 AM Masahiko Sawada > > > wrote: > > > > > > > > On Fri, Jul 30, 2021 at 12:52 PM Amit Ka

Re: Skipping logical replication transactions on subscriber side

On Wed, Aug 4, 2021 at 1:02 PM houzj.f...@fujitsu.com wrote: > > On Tuesday, August 3, 2021 2:49 PM Masahiko Sawada > wrote: > > > > I've attached new patches that incorporate all comments I got so far. > > Please review them. > > Hi, > > I had a few comments for the 0003 patch. Thanks for revi

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

On Wed, Aug 4, 2021 at 8:17 PM Greg Nancarrow wrote: > > Ah, thanks for that (I didn't debug that failure). > But is the coredump issue reproducible now? (using v7 and your test script) > Er, with the v7 patch, the problem still occurs (that Assert still fires during a run of the SubTransGetTopmo

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

ср, 4 авг. 2021 г. в 14:18, Greg Nancarrow : > On Wed, Aug 4, 2021 at 7:55 PM Pavel Borisov > wrote: > > > >> > > Greg, thanks for the fast response! I suppose that a check for > IsolationUsesXactSnapshot() is also useful in a GetTransactionSnapshot for > the correct processing of a case with NUL

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

> > At SERIALIZABLE level with v2/v5 I get an error which I don't have before > the patch (but no crash): > pgbench: error: client 6 script 0 aborted in command 594 query 0: ERROR: > could not serialize access due to read/write dependencies among > transactions > DETAIL: Reason code: Canceled on

RE: [BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

On Wednesday, August 4, 2021 7:00 PM Masahiko Sawada wrote > On Wed, Aug 4, 2021 at 5:06 PM houzj.f...@fujitsu.com > wrote: > > > > On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada > > > > > > I've not looked at the patch deeply yet but I think that the > > > following one line change seem

Re: Failed transaction statistics to measure the logical replication progress

On Wed, Aug 4, 2021 at 12:35 PM Masahiko Sawada wrote: > > On Wed, Aug 4, 2021 at 12:17 PM Amit Kapila wrote: > > > > On Wed, Aug 4, 2021 at 6:19 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 3, 2021 at 6:11 PM Amit Kapila > > > wrote: > > > > > > > > I was trying to think based on si

Re: A varint implementation for PG?

On Tue, Aug 3, 2021 at 3:32 PM Andres Freund wrote: > I am now wondering if what we're talking about here would best be thought of > not as a variable width integer type, but a variable-width encoding for all > pass-by-value types. > > Leaving on-disk compatibility aside (:)), ISTM that we by defa

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

On Tue, Aug 3, 2021 at 11:41 PM Greg Nancarrow wrote: > I've tried to follow your description and have attached a patch to > hopefully match it, but it doesn't pass "make check-world". > Perhaps I messed something up (apologies if so), or additional changes > are needed to match what you had in mi

Re: Possible dependency issue in makefile

Filip Janus writes: > I am building libecpg 13.1 but 13.3 behaves in the same manner and my build > fails with: > ... > Complete build log: log It looks like you're just running configure and then trying to do this: /usr/bin/make -O -j40 V=1

Release 13.1 of the PostgreSQL BuildFarm client

I have just pushed Release 13.1 of the PostgreSQL BuildFarm client. This update to Release 13 fixes errors that occur from the new default branch name checking code when used with versions of git that are too old. This code is now disabled if the git version is not capable of running it, and in

RE: Added schema level support for publication.

On Tuesday, August 3, 2021 11:08 PM vignesh C wrote: > > Thanks for reporting this, this is fixed in the v18 patch attached. Thanks for fixing it. Few suggestions for V18: 1. +# Clean up the tables on both publisher and subscriber as we don't need them +$node_publisher->safe_psql('postgres',

Re: Lowering the ever-growing heap->pd_lower

On Tue, Aug 3, 2021 at 8:44 PM Peter Geoghegan wrote: > This time it's quite different: we're truncating the line pointer > array during pruning. Pruning often occurs opportunistically, during > regular query processing. In fact I'd say that it's far more common > than pruning by VACUUM. So the ch

RFC: Improve CPU cache locality of syscache searches

CPU caches have multiple levels, so I had an idea to use that concept in the syscaches. Imagine if catcache buckets were cacheline-sized. In that case, we would store the most recently accessed hash values and pointers to catctup, in addition to the dlist_head: typedef struct cc_bucket { uint32

Re: A varint implementation for PG?

Hi, On 2021-08-04 09:31:25 -0400, Robert Haas wrote: > This is pretty integer-centric, though. If your pass-by-value type is > storing timestamps, for example, they're not likely to be especially > close to zero. Since a 64-bit address is pretty big, perhaps they're > still close enough to zero th

Re: Commitfest overflow

> > I think there might be a higher number of work-in-progress patches > these days, which represent ongoing collaborative efforts, and are not > expected to be committed soon, but are registered to attract the > attention of humans and robots. Perhaps if there were a separate > status for that, i

Re: A varint implementation for PG?

Hi, On 2021-08-03 14:26:16 -0400, Robert Haas wrote: > [ resurrecting this 2-year-old thread ] > > On Fri, Dec 13, 2019 at 12:45 AM Andres Freund wrote: > > > If baking a new variant integer format now, I think limiting it to 64 bits > > > is probably a mistake given how long-lived PostgreSQL is,

Re: Use POPCNT on MSVC

On Tue, Aug 3, 2021 at 11:36 PM David Rowley wrote: > > On Tue, 3 Aug 2021 at 22:43, John Naylor wrote: > > 1. the __popcnt64() intrinsic is put inside pg_popcount64_asm(), which is a bit of a misnomer since it's not assembly. Renaming s/_asm/_fast/ would help it look better. But then looking aro

Re: Lowering the ever-growing heap->pd_lower

On Wed, 4 Aug 2021 at 01:43, Peter Geoghegan wrote: > > On Mon, Aug 2, 2021 at 11:57 PM Simon Riggs > wrote: > > 1. Allow same thing as PageTruncateLinePointerArray() during HOT cleanup > > That is going to have a clear benefit for HOT workloads, which by > > their nature will use a lot of line p

Re: A varint implementation for PG?

On Wed, Aug 4, 2021 at 3:01 PM Andres Freund wrote: > Extending that to arbitrary lengths obviously at some point makes the encoding > in unary wasteful, and the benefit of few branches vanishes. So what I was > thinking is that for variable length pieces of data that are not limited to 8 > bytes,

Re: RFC: Improve CPU cache locality of syscache searches

Hi, On 2021-08-04 12:39:29 -0400, John Naylor wrote: > CPU caches have multiple levels, so I had an idea to use that concept in > the syscaches. I do think we loose a good bit to syscache efficiency in real workloads, so I think it's worth investing time into it. However: > Imagine if catcache

Re: A varint implementation for PG?

On 2021-08-04 15:37:36 -0400, Robert Haas wrote: > On Wed, Aug 4, 2021 at 3:01 PM Andres Freund wrote: > > Extending that to arbitrary lengths obviously at some point makes the > > encoding > > in unary wasteful, and the benefit of few branches vanishes. So what I was > > thinking is that for var

Re: straightening out backend process startup

On Mon, Aug 2, 2021 at 12:41 PM Andres Freund wrote: > - AuxiliaryProcessMain() is used for two independent tasks: Start bootstrap / > checker mode and starting auxiliary processes. In HEAD there's maybe 5 lines > out 250 that are actually common to both uses. > > A related oddity is that we

Re: A varint implementation for PG?

On Wed, Aug 4, 2021 at 3:46 PM Andres Freund wrote: > > But what if I have a machine with more than 16 exabytes of RAM and I > > want to use all of its memory to store one really big integer? > > Then the embedded 8 byte length value would just have to do the same thing > recursively to store that

Re: A varint implementation for PG?

On 8/4/21 9:01 PM, Andres Freund wrote: Hi, On 2021-08-03 14:26:16 -0400, Robert Haas wrote: [ resurrecting this 2-year-old thread ] On Fri, Dec 13, 2019 at 12:45 AM Andres Freund wrote: If baking a new variant integer format now, I think limiting it to 64 bits is probably a mistake given

Re: [PATCH] Partial foreign key updates in referential integrity triggers

On Wed, Jul 14, 2021 at 6:51 AM Peter Eisentraut wrote: > I think this is an interesting feature with a legitimate use case. Great! I'm glad to hear that. > Consider what should happen when you update users.id. Per SQL standard, > for MATCH SIMPLE an ON UPDATE SET NULL should only set to null t

Bug fix for cache lookup failure for statistic_ext type

Hackers, You can easily get a cache lookup failure by changing the regression tests as included in this small patch. The failure looks thus: +COMMENT ON STATISTICS ab1_a_b_stats IS 'new comment'; +CREATE ROLE temp_role; +SET SESSION AUTHORIZATION temp_role; +COMMENT ON STATISTICS ab1_a_b_stats

Another regexp performance improvement: skip useless paren-captures

Here's a little finger exercise that improves a case that's bothered me for awhile. In a POSIX regexp, parentheses cause capturing by default; you have to write the very non-obvious "(?:...)" if you don't want the matching substring to be reported by the regexp engine. That'd be fine if capturing

Re: Lowering the ever-growing heap->pd_lower

On Wed, Aug 4, 2021 at 12:09 PM Simon Riggs wrote: > Truncating line pointers can make extra space on the page, so it could > be the difference between a HOT and a non-HOT update. My understanding > is that these just-in-time actions have a beneficial effect in other > circumstances, so we can do

Re: Lowering the ever-growing heap->pd_lower

On Wed, Aug 4, 2021 at 7:39 AM Robert Haas wrote: > How would it hurt? > > It's easy to see the harm caused by not shortening the line pointer > array when it is possible to do so: we're using up space in the page > that could have been made free. It's not so obvious to me what the > downside of s

Re: Bug fix for cache lookup failure for statistic_ext type

On 8/5/21 12:03 AM, Mark Dilger wrote: Hackers, You can easily get a cache lookup failure by changing the regression tests as included in this small patch. The failure looks thus: +COMMENT ON STATISTICS ab1_a_b_stats IS 'new comment'; +CREATE ROLE temp_role; +SET SESSION AUTHORIZATION temp_ro

Re: A varint implementation for PG?

Hi, On 2021-08-04 23:44:10 +0200, Tomas Vondra wrote: > How is that better than the two varint flavors that are already out there, > i.e. the bitcoin [1] and protocol buffers [2]? The protobuf one is *terrible* for CPU efficiency. You need to go through each byte, do masking and shifting for each

Re: A varint implementation for PG?

On 8/5/21 1:05 AM, Andres Freund wrote: Hi, On 2021-08-04 23:44:10 +0200, Tomas Vondra wrote: How is that better than the two varint flavors that are already out there, i.e. the bitcoin [1] and protocol buffers [2]? The protobuf one is *terrible* for CPU efficiency. You need to go through eac

Re: archive status ".ready" files may be created too early

At Tue, 3 Aug 2021 21:32:18 +, "Bossart, Nathan" wrote in > On 8/2/21, 7:37 PM, "Kyotaro Horiguchi" wrote: > > I'm afraid that using hash to store boundary info is too much. Isn't a > > ring buffer enough for this use? In that case it is enough to > > remember only the end LSN of the segme

Re: Parallel scan with SubTransGetTopmostTransaction assert coredump

On Wed, Aug 4, 2021 at 11:43 PM Robert Haas wrote: > > Why do you think it's right to install the serialized *active* > snapshot as the *transaction* snapshot? I've been operating on the > presumption that we wanted the worker to install the leader's > transaction snapshot as its transaction snaps

Re: Partition Check not updated when insert into a partition

Sorry that I missed this thread. On Wed, Jul 14, 2021 at 11:16 AM houzj.f...@fujitsu.com wrote: > On Tuesday, July 13, 2021 2:52 AM Alvaro Herrera > wrote: > > On 2021-Jun-23, houzj.f...@fujitsu.com wrote: > > > > > For a multi-level partition, for example: table 'A' is partition of > > > table

Re: [PATCH] postgres_fdw: suppress explicit casts in text:text comparisons (was: column option to override foreign types)

On Sun Mar 7, 2021 at 2:37 AM EST, Dian M Fay wrote: > > What'd be better, if we could do it, is to ship the clause in > > the form > > WHERE foreigncol = 'one' > > that is, instead of plastering a cast on the Var, try to not put > > any explicit cast on the constant. That fixes your original use >

Allow escape in application_name (was: [postgres_fdw] add local pid to fallback_application_name)

Dear Hackers, Tom, (I changed subject because this is no longer postgres_fdw's patch) > > What would be better to think about is how to let users specify this > > kind of behavior for themselves. I think it's possible to set > > application_name as part of a foreign server's connection options,

Re: Implementing Incremental View Maintenance

Hello Zhihong Yu, On Mon, 2 Aug 2021 14:33:46 -0700 Zhihong Yu wrote: > On Sun, Aug 1, 2021 at 11:30 PM Yugo NAGATA wrote: > > > Hi hackers, > > > > On Mon, 19 Jul 2021 09:24:30 +0900 > > Yugo NAGATA wrote: > > > > > On Wed, 14 Jul 2021 21:22:37 +0530 > > > vignesh C wrote: > > > > > > The p

Re: Implementing Incremental View Maintenance

Hello Takahashi-san, On Tue, 3 Aug 2021 10:15:42 + "r.takahash...@fujitsu.com" wrote: > Hi Nagata-san, > > > I am interested in this patch since it is good feature. > > I run some simple tests. > I found the following problems. Thank you for your interest for this patch! > (1) > Failed

Re: archive status ".ready" files may be created too early

By the way about the v3 patch, +#define InvalidXLogSegNo ((XLogSegNo) 0x) Like InvalidXLogRecPtr, the first valid segment number is 1 so we can use 0 as InvalidXLogSegNo. BootStrapXLOG(): /* Create first XLOG segment file */ openLogFile = XLogFileInit(1); K

Re: Sort keys are omitted incorrectly after grouping sets

On Tue, 3 Aug 2021 at 00:04, Richard Guo wrote: > > Is this a problem we should be worried about? It's easy to see this produce output in the wrong order: postgres=# select a, b from (values (1,1),(2,2)) as foo(a,b) where a = b group by cube(a, b) order by a, b nulls first; a | b ---+--- 1 |

Re: Commitfest overflow

On Tue, Aug 03, 2021 at 12:13:44PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Tue, Aug 3, 2021 at 04:53:40PM +0100, Simon Riggs wrote: > >> There are 273 patches in the queue for the Sept Commitfest already, so > >> it seems clear the queue is not being cleared down each CF as it was >

Re: [BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

On Wed, Aug 4, 2021 at 9:19 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, August 4, 2021 7:00 PM Masahiko Sawada > wrote > > On Wed, Aug 4, 2021 at 5:06 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada > > > > > > > > > > > I've not l

Re: archive status ".ready" files may be created too early

On 8/4/21, 9:05 PM, "Kyotaro Horiguchi" wrote: > By the way about the v3 patch, > > +#define InvalidXLogSegNo ((XLogSegNo) 0x) > > Like InvalidXLogRecPtr, the first valid segment number is 1 so we can > use 0 as InvalidXLogSegNo. It's been a while since I wrote this, but if

Re: [BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION

On Thu, Aug 5, 2021 at 2:08 PM Masahiko Sawada wrote: > > On Wed, Aug 4, 2021 at 9:19 PM houzj.f...@fujitsu.com > wrote: > > > > On Wednesday, August 4, 2021 7:00 PM Masahiko Sawada > > wrote > > > On Wed, Aug 4, 2021 at 5:06 PM houzj.f...@fujitsu.com > > > wrote: > > > > > > > > On Wednesday

Re: Commitfest overflow

> 5 авг. 2021 г., в 09:25, Noah Misch написал(а): > > On Tue, Aug 03, 2021 at 12:13:44PM -0400, Tom Lane wrote: >> Bruce Momjian writes: >>> On Tue, Aug 3, 2021 at 04:53:40PM +0100, Simon Riggs wrote: There are 273 patches in the queue for the Sept Commitfest already, so it seems c

Re: row filtering for logical replication

v21 --> v22 (This small change is only to keep the patch up-to-date with HEAD) Changes: * A recent commit [1] added a new TAP subscription test file 023, so now this patch's test file (previously "023_row_filter.pl") has been bumped to "024_row_filter.pl". -- [1] https://github.com/postgre