Re: wal stats questions

Thanks for many your suggestions! I made the patch to handle the issues. > 1) What is the motivation to have both prevWalUsage and pgWalUsage, >instead of just accumulating the stats since the last submission? >There doesn't seem to be any comment explaining it? Computing >differences

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

On 2021/03/26 9:54, Fujii Masao wrote: > On 2021/03/26 9:25, Masahiro Ikeda wrote: >> On 2021/03/25 21:23, Fujii Masao wrote: >>> On 2021/03/25 9:58, Andres Freund wrote: Also, won't this lead to postmaster now starting to complain about pgstat having crashed in an immediate shutdown?

Re: [PATCH] More docs on what to do and not do in extension code

On Fri, 26 Mar 2021 at 06:15, Bruce Momjian wrote: > On Thu, Mar 25, 2021 at 08:49:44AM -0400, David Steele wrote: > > On 1/22/21 1:36 AM, Craig Ringer wrote: > > > > > > Would you mind attaching a revised version of the patch with your > edits? > > > Otherwise I'll go and merge them in once you'

Re: Is it useful to record whether plans are generic or custom?

On 2021/03/26 0:33, torikoshia wrote: On 2021-03-25 22:14, Fujii Masao wrote: On 2021/03/23 16:32, torikoshia wrote: On 2021-03-05 17:47, Fujii Masao wrote: Thanks for your comments! Thanks for updating the patch! PostgreSQL Patch Tester reported that the patched version failed to be com

Re: [PATCH] add concurrent_abort callback for output plugin

On 26.03.21 04:28, Amit Kapila wrote: I think as you have noted that stream_abort or rollback_prepared will be sent (the remaining changes in-between will be skipped) as we decode them from WAL Yes, but as outlined, too late. Multiple other transactions may get decoded until the decoder reach

Walsender may fail to send wal to the end.

Hello, I happened to see a doubious behavior of walsender. On a replication set with wal_keep_size/(segments) = 0, running the following command on the primary causes walsender to fail to send up to the final shutdown checkpoint record to the standby. (create table t in advance) psql -c 'insert

Re: [PATCH] Allow multiple recursive self-references

Thanks for the feedback, Tom. > Tom Lane writes: > [...] > As far as I can see, the spec flat-out forbids this. In SQL:2021, > it's discussed in 7.17 syntax rule 3) j) ix), which > defines [linear recursion] (Aside: We don't have a copy of the SQL:2021 specification here (all we've got here is

Re: [PATCH] pg_permissions

On Fri, Mar 26, 2021, at 07:53, Joel Jacobson wrote: > On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: >> "Joel Jacobson" mailto:joel%40compiler.org>> writes: >> > On Thu, Mar 25, 2021, at 16:16, Alvaro Herrera wrote: >> >> Ah, of course -- the only way to obtain the acl columns is by going >> >> t

Re: [PATCH] add concurrent_abort callback for output plugin

On Fri, Mar 26, 2021 at 2:42 PM Markus Wanner wrote: > > On 26.03.21 04:28, Amit Kapila wrote: > > I think as you have noted that stream_abort or rollback_prepared will > > be sent (the remaining changes in-between will be skipped) as we > > decode them from WAL > > Yes, but as outlined, too late.

Re: [PATCH] pg_permissions

On 2021-Mar-26, Joel Jacobson wrote: > On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: > > I wonder what performance will be like with lots o' objects. > > I guess pg_get_acl() would need to be implemented using a switch(classid) > with 36 cases (one for each class)? No, we have a generalized

Re: PoC/WIP: Extended statistics on expressions

On Thu, 25 Mar 2021 at 19:59, Tomas Vondra wrote: > > Attached is an updated patch series, with all the changes discussed > here. I've cleaned up the ndistinct stuff a bit more (essentially > reverting back from GroupExprInfo to GroupVarInfo name), and got rid of > the UpdateStatisticsForTypeChang

Re: [PATCH] add concurrent_abort callback for output plugin

On 26.03.21 11:19, Amit Kapila wrote: No, I am not assuming that. I am just trying to describe you that it is not necessary that we will be able to detect concurrent abort in each and every case. Sure. Nor am I claiming that would be necessary or that the patch changed anything about it. As

Re: [PATCH] pg_permissions

On Fri, Mar 26, 2021, at 11:30, Alvaro Herrera wrote: > On 2021-Mar-26, Joel Jacobson wrote: > > > On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: > > > > I wonder what performance will be like with lots o' objects. > > > > I guess pg_get_acl() would need to be implemented using a switch(classid

Re: [PATCH] pg_permissions

On 2021-Mar-26, Joel Jacobson wrote: > On Fri, Mar 26, 2021, at 11:30, Alvaro Herrera wrote: > > On 2021-Mar-26, Joel Jacobson wrote: > > > > > On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: > > > > > > I wonder what performance will be like with lots o' objects. > > > > > > I guess pg_get_acl

Re: PoC/WIP: Extended statistics on expressions

On 3/26/21 12:37 PM, Dean Rasheed wrote: > On Thu, 25 Mar 2021 at 19:59, Tomas Vondra > wrote: >> >> Attached is an updated patch series, with all the changes discussed >> here. I've cleaned up the ndistinct stuff a bit more (essentially >> reverting back from GroupExprInfo to GroupVarInfo name

Re: WIP: BRIN multi-range indexes

On 3/23/21 7:28 PM, Alvaro Herrera wrote: > On 2021-Mar-23, Tomas Vondra wrote: > >> FWIW there's yet another difference between the current BRIN opclass >> definition, compared to what CREATE OPERATOR CLASS would do. Or more >> precisely, how we'd define opfamily for the cross-type cases (inte

Re: WIP: BRIN multi-range indexes

Hi, I've pushed both the bloom and minmax-multi indexes today. Based on the feedback and limitations described before I decided to keep them in core (i.e. not move them to contrib), but it was very useful experiment as it uncovered a couple issues - both in existing code, and in the definition of

Re: Stronger safeguard for archive recovery not to miss data

On 3/25/21 9:23 PM, Fujii Masao wrote: On 2021/03/25 23:21, David Steele wrote: On 1/25/21 3:55 AM, Laurenz Albe wrote: On Mon, 2021-01-25 at 08:19 +, osumi.takami...@fujitsu.com wrote: I think you should pst another patch where the second, now superfluous, error message is removed. Up

Re: [PATCH] pg_permissions

Alvaro Herrera writes: > On 2021-Mar-26, Joel Jacobson wrote: >> On Thu, Mar 25, 2021, at 17:51, Tom Lane wrote: >> I wonder what performance will be like with lots o' objects. > I guess he is concerned about the number of catalog accesses. My concern is basically that you're forcing the join be

Re: Allow matching whole DN from a client certificate

On 3/24/21 12:54 AM, Michael Paquier wrote: [numerous useful comments] OK, here's a new patch. I hope to commit this within a few days. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index b4204

Re: WIP: BRIN multi-range indexes

On 2021-Mar-26, Tomas Vondra wrote: > I've pushed both the bloom and minmax-multi indexes today. Congratulations! I think this reimplementation of the minmax opclass infrastructure makes BRIN much more useful (read: actually usable). -- Álvaro Herrera Valdivia, Chile

Re: WIP: BRIN multi-range indexes

On 3/26/21 2:08 PM, Tomas Vondra wrote: > Hi, > > I've pushed both the bloom and minmax-multi indexes today. > Hmmm, I see a couple buildfarm animals are upset about the minmax-multi patch. It does pass for me both on x86_64 and 32-bit ARM (rpi4), so I'm unable to reproduce this. But most of

Re: [UNVERIFIED SENDER] Re: Challenges preventing us moving to 64 bit transaction id (XID)?

Hi Jim, On 3/26/21 12:01 AM, Thomas Munro wrote: On Fri, Mar 26, 2021 at 2:57 AM David Steele wrote: On 1/22/21 6:46 PM, Finnerty, Jim wrote: First 3 patches derived from the original 64-bit xid patch set by Alexander Korotkov The patches no longer apply (http://cfbot.cputube.org/patch_32_

Re: WIP: BRIN multi-range indexes

Tomas Vondra writes: > I recall seeing "bus error" on sparc with other patches because of > alignment issues, so I wonder if this is what's happening here. Try compiling with the address sanitizer enabled. Per c.h, * Testing can be done with "-fsanitize=alignment -fsanitize-trap=alignment" *

Re: WIP: BRIN multi-range indexes

On 3/26/21 2:55 PM, Tom Lane wrote: > Tomas Vondra writes: >> I recall seeing "bus error" on sparc with other patches because of >> alignment issues, so I wonder if this is what's happening here. > > Try compiling with the address sanitizer enabled. Per c.h, > > * Testing can be done with "-fs

Re: invalid data in file backup_label problem on windows

On 3/21/21 10:40 AM, Magnus Hagander wrote: On Sat, Mar 20, 2021 at 3:10 AM wangsh.f...@fujitsu.com wrote: David Steele wrote: It's not clear to me what text editors have to do with this? Are you editing the file manually? When I execute SELECT * FROM pg_stop_backup(false, true) in psql.

Re: WIP: BRIN multi-range indexes

On 3/26/21 3:04 PM, Tomas Vondra wrote: > On 3/26/21 2:55 PM, Tom Lane wrote: >> Tomas Vondra writes: >>> I recall seeing "bus error" on sparc with other patches because of >>> alignment issues, so I wonder if this is what's happening here. >> >> Try compiling with the address sanitizer enabled.

Re: MultiXact\SLRU buffers configuration

> 26 марта 2021 г., в 11:00, Andrey Borodin написал(а): > >> I'm not saying the 0002 patch is bug-free yet though, it's a bit finickity. > I think the idea of speeding up linear search is really really good for > scaling SLRUs. It's not even about improving normal performance of the > cluste

Re: WIP: BRIN multi-range indexes

On 3/26/21 3:45 PM, Tomas Vondra wrote: > On 3/26/21 3:04 PM, Tomas Vondra wrote: >> On 3/26/21 2:55 PM, Tom Lane wrote: >>> Tomas Vondra writes: I recall seeing "bus error" on sparc with other patches because of alignment issues, so I wonder if this is what's happening here. >>> >>> Try

Bug on update timing of walrcv->flushedUpto variable

Hi, all Recently, I found a bug on update timing of walrcv->flushedUpto variable, consider the following scenario, there is one Primary node, one Standby node which streaming from Primary: There are a large number of SQL running in the Primary, and the length of the xlog record generated by the

Re: invalid data in file backup_label problem on windows

On 3/26/21 10:19 AM, David Steele wrote: > >> No, the problem is you are using copy/paste and in doing so you are >> *changing'* the value that is being returned. You'll either need to >> update your copy/paste procedure to not mess with the newlines, or to >> use a better way to get the data out

Re: wal stats questions

Hi, On 2021-03-25 16:37:10 +0900, Kyotaro Horiguchi wrote: > On the other hand, the counters are incremented in XLogInsertRecord() > and I think we don't want add instructions there. I don't really buy this. Setting a boolean to true, in a cacheline you're already touching, isn't that much compar

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

Hi, On 2021-03-25 21:23:17 +0900, Fujii Masao wrote: > > This strikes me as a quite a misleading function name. > > Yeah, better name is always welcome :) It might just be best to not introduce a generic function and just open code one just for the stats collector... > > Outside of very > > na

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

Hi, On 2021-03-26 09:27:19 +0900, Masahiro Ikeda wrote: > On 2021/03/25 19:48, Fujii Masao wrote: > > Yes. So we should wait for the shared memory stats patch to be committed > > before working on walreceiver stats patch more? > > Yes, I agree. Agreed. One thing that I didn't quite see discusse

Re: Walsender may fail to send wal to the end.

Hi, On 2021-03-26 18:20:14 +0900, Kyotaro Horiguchi wrote: > This is because XLogSendPhysical detects removal of the wal segment > currently reading by shutdown checkpoint. However, there' no fear of > overwriting of WAL segments at the time. > > So I think we can omit the call to CheckXLogRemove

Re: invalid data in file backup_label problem on windows

On Fri, Mar 26, 2021 at 5:52 PM Andrew Dunstan wrote: > > > On 3/26/21 10:19 AM, David Steele wrote: > > > >> No, the problem is you are using copy/paste and in doing so you are > >> *changing'* the value that is being returned. You'll either need to > >> update your copy/paste procedure to not me

Re: Calendar support in localization

Thomas Munro wrote: > Right, so if this is done by trying to extend Daniel Verite's icu_ext > extension (link given earlier) and Robert's idea of a fast-castable > type, I suppose you might want now()::icu_date + '1 month'::internal > to advance you by one Ethiopic month if you have done S

Re: Query about pg asynchronous processing support

Hi, You should search informations about postgres hooks like thoses used in extension pg_stat_statements https://github.com/postgres/postgres/blob/master/contrib/pg_stat_statements/pg_stat_statements.c And about background capabilities as thoses used in extension pg_background https://github.com/

Re: invalid data in file backup_label problem on windows

On 3/26/21 1:20 PM, Magnus Hagander wrote: On Fri, Mar 26, 2021 at 5:52 PM Andrew Dunstan wrote: On 3/26/21 10:19 AM, David Steele wrote: No, the problem is you are using copy/paste and in doing so you are *changing'* the value that is being returned. You'll either need to update your copy/p

[PATCH] Allow CustomScan nodes to signal projection support

Hi, The attached patch allows CustomScan nodes to signal whether they support projection. Currently all CustomScan nodes are treated as supporting projection. But it would be nice for custom nodes to opt out of this to prevent postgres from modifying the targetlist of the custom node. For feature

Re: UniqueKey on Partitioned table.

> On Sat, Feb 20, 2021 at 10:25:59AM +0800, Andy Fan wrote: > > The attached is a UnqiueKey with EquivalenceClass patch, I just complete the > single relation part and may have bugs. I just attached it here for design > review only. and the not-null-attrs is just v1 which we can continue > discussi

Re: Support for NSS as a libpq TLS backend

Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > On Wed, 2021-03-24 at 14:10 -0400, Stephen Frost wrote: > > * Jacob Champion (pchamp...@vmware.com) wrote: > > > I could see this being a problem if two client certificate nicknames > > > collide across multiple in-use databases, maybe? >

Re: libpq debug log

Hello I added an option to the new libpq_pipeline program that it activates libpq trace. It works nicely and I think we can add that to the regression tests. However I have two observations. 1. The trace output for the error message won't be very nice, because it prints line numbers. So if I w

Re: MultiXact\SLRU buffers configuration

On Sat, Mar 27, 2021 at 4:52 AM Andrey Borodin wrote: > Some thoughts on HashTable patch: > 1. Can we allocate bigger hashtable to reduce probability of collisions? Yeah, good idea, might require some study. > 2. Can we use specialised hashtable for this case? I'm afraid hash_search() > does co

Re: SQL/JSON: JSON_TABLE

On 3/25/21 8:10 AM, David Steele wrote: > On 1/20/21 8:42 PM, Nikita Glukhov wrote: >> Thank you for review. >> >> Attached 45th version of the patches. "SQL/JSON functions" patch >> corresponds to >> v52 patch set posted in the separate thread. > > Another rebase needed (http://cfbot.cputube.org/

Re: SQL/JSON: JSON_TABLE

> On 2021.03.26. 21:28 Andrew Dunstan wrote: > On 3/25/21 8:10 AM, David Steele wrote: > > On 1/20/21 8:42 PM, Nikita Glukhov wrote: > >> Thank you for review. > >> > >> Attached 45th version of the patches. "SQL/JSON functions" patch > >> corresponds to > >> v52 patch set posted in the separate t

Re: SQL/JSON: functions

On 3/26/21 4:22 PM, Andrew Dunstan wrote: > On 3/8/21 1:55 PM, Ibrar Ahmed wrote: >> >> On Sat, Jan 23, 2021 at 3:37 PM Erik Rijkers > > wrote: >> >> On 2021-01-20 03:49, Nikita Glukhov wrote: >> >> > [0001-Add-common-SQL-JSON-clauses-v52.patch.gz] >> > [0002-SQ

Re: SQL/JSON: JSON_TABLE

On 3/26/21 4:48 PM, Erik Rijkers wrote: >> On 2021.03.26. 21:28 Andrew Dunstan wrote: >> On 3/25/21 8:10 AM, David Steele wrote: >>> On 1/20/21 8:42 PM, Nikita Glukhov wrote: Thank you for review. Attached 45th version of the patches. "SQL/JSON functions" patch corresponds to

Re: Support for NSS as a libpq TLS backend

On Fri, 2021-03-26 at 15:33 -0400, Stephen Frost wrote: > * Jacob Champion (pchamp...@vmware.com) wrote: > > Databases that are opened *after* the first one are given their own > > separate slots. [...] > > This is more-or-less what we would want though, right..? If a user asks > for a connection

Re: Support for NSS as a libpq TLS backend

Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > On Fri, 2021-03-26 at 15:33 -0400, Stephen Frost wrote: > > * Jacob Champion (pchamp...@vmware.com) wrote: > > > Databases that are opened *after* the first one are given their own > > > separate slots. [...] > > > > This is more-or-less

Re: SQL/JSON: functions

On 3/26/21 4:49 PM, Andrew Dunstan wrote: > On 3/26/21 4:22 PM, Andrew Dunstan wrote: >> On 3/8/21 1:55 PM, Ibrar Ahmed wrote: >>> On Sat, Jan 23, 2021 at 3:37 PM Erik Rijkers >> > wrote: >>> >>> On 2021-01-20 03:49, Nikita Glukhov wrote: >>> >>> > [0001-Add-common-

Re: Proposal: Save user's original authenticated identity for logging

On Fri, 2021-03-26 at 09:12 +0900, Michael Paquier wrote: > Does it really matter to have the full contents of the file from the > previous tests though? For a few of the bugs I was tracking down, it was imperative. The tests aren't isolated enough (or at all) to keep one from affecting the others

Re: PoC/WIP: Extended statistics on expressions

On 3/26/21 1:54 PM, Tomas Vondra wrote: > > > On 3/26/21 12:37 PM, Dean Rasheed wrote: >> On Thu, 25 Mar 2021 at 19:59, Tomas Vondra >> wrote: >>> >>> Attached is an updated patch series, with all the changes discussed >>> here. I've cleaned up the ndistinct stuff a bit more (essentially >>> rev

Re: libpq debug log

Proposed changes on top of v29. -- Álvaro Herrera Valdivia, Chile >From b32ae3805bb28553c0a1cf308c6ed27f58576f3c Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Fri, 26 Mar 2021 19:12:12 -0300 Subject: [PATCH 1/5] libpq_pipeline: add -t support for PQtrace --- .../modules/libpq_pipel

Showing applied extended statistics in explain

Hi, With extended statistics it may not be immediately obvious if they were applied and to which clauses. If you have multiple extended statistics, we may also apply them in different order, etc. And with expressions, there's also the question of matching expressions to the statistics. So it seem

Re: PoC/WIP: Extended statistics on expressions

On 3/27/21 1:17 AM, Tomas Vondra wrote: > On 3/26/21 1:54 PM, Tomas Vondra wrote: >> >> >> On 3/26/21 12:37 PM, Dean Rasheed wrote: >>> On Thu, 25 Mar 2021 at 19:59, Tomas Vondra >>> wrote: Attached is an updated patch series, with all the changes discussed here. I've cleaned up

Re: MultiXact\SLRU buffers configuration

> 27 марта 2021 г., в 01:26, Thomas Munro написал(а): > > On Sat, Mar 27, 2021 at 4:52 AM Andrey Borodin wrote: >> Some thoughts on HashTable patch: >> 1. Can we allocate bigger hashtable to reduce probability of collisions? > > Yeah, good idea, might require some study. In a long run we alw

Re: UniqueKey on Partitioned table.

On Sat, Mar 27, 2021 at 3:07 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Sat, Feb 20, 2021 at 10:25:59AM +0800, Andy Fan wrote: > > > > The attached is a UnqiueKey with EquivalenceClass patch, I just complete > the > > single relation part and may have bugs. I just attached it here for

Re: Extend more usecase for planning time partition pruning and init partition pruning.

Hi David: On Mon, Mar 8, 2021 at 9:34 AM David Rowley wrote: > On Thu, 4 Mar 2021 at 22:07, Amit Langote wrote: > > * Or maybe have you considered generalizing what > > build_implied_pruning_quals() does so that other places like > > indxpath.c can use the facility? > > I agree with doing it an

Re: [PATCH] add concurrent_abort callback for output plugin

On Fri, Mar 26, 2021 at 5:50 PM Markus Wanner wrote: > > On 26.03.21 11:19, Amit Kapila wrote: > > No, I am not assuming that. I am just trying to describe you that it > > is not necessary that we will be able to detect concurrent abort in > > each and every case. > > Sure. Nor am I claiming that