Re: Double partition lock in bufmgr

2020-12-18 Thread Zhihong Yu
Hi, w.r.t. the code in BufferAlloc(), the pointers are compared. Should we instead compare the tranche Id of the two LWLock ? Cheers

Re: proposal: schema variables

2020-12-18 Thread Pavel Stehule
Hi only rebase Regards Pavel schema-variables-20201219.patch.gz Description: application/gzip

Re: Misleading comment in prologue of ReorderBufferQueueMessage

2020-12-18 Thread Amit Kapila
On Fri, Dec 18, 2020 at 3:37 PM Ashutosh Bapat wrote: > > On Wed, Dec 16, 2020 at 8:00 AM Amit Kapila wrote: >> >> How about something like below: >> A transactional message is queued to be processed upon commit and a >> non-transactional message gets processed immediately. > > > Used this one. P

Re: Single transaction in the tablesync worker?

2020-12-18 Thread Amit Kapila
On Fri, Dec 18, 2020 at 6:41 PM Peter Smith wrote: > > TODO / Known Issues: > > * the current implementation of tablesync drop slot (e.g. from > DropSubscription or finish_sync_worker) regenerates the tablesync slot > name so it knows what slot to drop. > If you always drop the slot at finish_syn

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Michael Paquier
On Fri, Dec 18, 2020 at 11:51:55AM +0200, Heikki Linnakangas wrote: > On 18/12/2020 11:35, Heikki Linnakangas wrote: > > BTW, looking at pg_cryptohash_ctx and pg_cryptohash_state, why do we > > need two structs? They're both allocated and controlled by the > > cryptohash implementation. It would se

[PATCH] Identify LWLocks in tracepoints

2020-12-18 Thread Craig Ringer
Hi all The attached patch set follows on from the discussion in [1] "Add LWLock blocker(s) information" by adding the actual LWLock* and the numeric tranche ID to each LWLock related TRACE_POSTGRESQL_foo tracepoint. This does not provide complete information on blockers, because it's not necessar

Re: Commit fest manager for 2021-01

2020-12-18 Thread Michael Paquier
On Sat, Dec 19, 2020 at 10:03:47AM +0530, Amit Kapila wrote: > Glad to hear. I am confident that you can do justice to this role. I also think you will do just fine. Thanks for taking care of this. -- Michael signature.asc Description: PGP signature

Re: Commit fest manager for 2021-01

2020-12-18 Thread Amit Kapila
On Sat, Dec 19, 2020 at 9:10 AM Masahiko Sawada wrote: > > Hi all, > > The next commit fest is going to begin in two weeks. > > I would like to volunteer as commit fest manager for 2021-01 > Glad to hear. I am confident that you can do justice to this role. if the > role is not filled and there

Commit fest manager for 2021-01

2020-12-18 Thread Masahiko Sawada
Hi all, The next commit fest is going to begin in two weeks. I would like to volunteer as commit fest manager for 2021-01 if the role is not filled and there are no objections. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Proposed patch for key managment

2020-12-18 Thread Bruce Momjian
On Fri, Dec 18, 2020 at 04:36:24PM -0500, Stephen Frost wrote: > > Starting with the passphrase key wrapper, since it's what's in place now. > > > > - cluster_key_challenge_command = 'password_key_wrapper %q' > > I do tend to like this idea of having what > cluster_key_challenge_command, or what

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Michael Paquier
On Fri, Dec 18, 2020 at 01:04:27PM +0200, Heikki Linnakangas wrote: > BTW, it's a bit weird that the pg_cryptohash_init/update/final() functions > return success, if the ctx argument is NULL. It would seem more sensible for > them to return an error. Okay. > That way, if a caller forgets to check

Re: Refactoring HMAC in the core code

2020-12-18 Thread Bruce Momjian
On Fri, Dec 18, 2020 at 07:42:02PM -0500, Bruce Momjian wrote: > > Please note that on a related thread that I have begun yesterday, > > Heikki has suggested some changes in the way we handle the opaque data > > used by each cryptohash implementation. > > https://www.postgresql.org/message-id/6ebe7

Re: Refactoring HMAC in the core code

2020-12-18 Thread Bruce Momjian
On Sat, Dec 19, 2020 at 09:35:57AM +0900, Michael Paquier wrote: > On Fri, Dec 18, 2020 at 10:48:00AM -0500, Bruce Momjian wrote: > > Great. See my questions in the key manager thread about whether I > > should use the init/update/final API or just keep the one-line version. > > As far as when to

Re: Refactoring HMAC in the core code

2020-12-18 Thread Michael Paquier
On Fri, Dec 18, 2020 at 10:48:00AM -0500, Bruce Momjian wrote: > Great. See my questions in the key manager thread about whether I > should use the init/update/final API or just keep the one-line version. > As far as when to commit this, I think the quiet time is actually better > because if you b

Re: terminate called after throwing an instance of 'std::bad_alloc'

2020-12-18 Thread Justin Pryzby
I'm re-raising this issue (since I needed to track down why I'd reflexively saved a 512g backup of the filesystem where this first happened). I'm 99% sure the "bad_alloc" is from LLVM. It happened multiple times on different servers (running a similar report) after setting jit=on during pg13 upgr

Re: [PATCH] Implements SPIN_LOCK on ARM

2020-12-18 Thread Tom Lane
David CARLIER writes: > Hi here a little update proposal for ARM architecture. This sort of thing is not a "little proposal" where you can just send in an unsupported patch and expect it to be accepted. You need to provide some evidence that (a) it actually does anything useful and (b) it isn't a

[PATCH] Implements SPIN_LOCK on ARM

2020-12-18 Thread David CARLIER
Hi here a little update proposal for ARM architecture. Kind regards. 0001-Implements-SPIN_LOCK-macro-for-ARM.patch Description: Binary data

Re: Proposed patch for key managment

2020-12-18 Thread Stephen Frost
Greetings Alastair, * Alastair Turner (min...@decodable.me) wrote: > On Wed, 16 Dec 2020 at 22:43, Stephen Frost wrote: > > If I'm following, you're suggesting something like: > > > > cluster_passphrase_command = 'aws get %q' > > > > and then '%q' gets replaced with "Please provide the WAL DEK: "

libpq @windows : leaked singlethread_lock makes AppVerifier unhappy

2020-12-18 Thread Першин Юрий Петрович
HI address d2be8fe8 found in _DPH_HEAP_ROOT @ fe321000 in busy allocation ( DPH_HEAP_BLOCK: UserAddr UserSize - VirtAddr VirtSize) d2dd3444: d2be8fe8 18 - d2be8000 2000 576b

Re: [HACKERS] [PATCH] Generic type subscripting

2020-12-18 Thread Dmitry Dolgov
> On Thu, Dec 17, 2020 at 03:29:35PM -0500, Tom Lane wrote: > Dmitry Dolgov <9erthali...@gmail.com> writes: > > On Thu, Dec 17, 2020 at 01:49:17PM -0500, Tom Lane wrote: > >> We can certainly reconsider the API for the parsing hook if there's > >> really a good reason for these to be different type

Re: WIP: System Versioned Temporal Table

2020-12-18 Thread Ryan Lambert
On Thu, Nov 19, 2020 at 11:04 AM Surafel Temesgen wrote: > > Attached is a rebased one. > regards > Surafel > Thank you for your work on this! The v7 patch fails on the current master branch. Error from make: gram.y:16695:1: error: static declaration of ‘makeAndExpr’ follows non-static declar

Re: [PATCH] Automatic HASH and LIST partition creation

2020-12-18 Thread Pavel Borisov
I've realized one strange effect in current grammar parsing: if I do CREATE TABLE foo (a int) PARTITION BY LIST (a) CONFIGURATION (a 1); ERROR: unrecognized auto partition bound specification "a" I consulted the patch code and realized that in fact, the patch considers it the (invalid) HASH boun

Re: New Table Access Methods for Multi and Single Inserts

2020-12-18 Thread Justin Pryzby
On Fri, Dec 18, 2020 at 07:39:14AM +0530, Bharath Rupireddy wrote: > On Fri, Dec 18, 2020 at 2:14 AM Justin Pryzby wrote: > > Are you thinking that TableInsertState would eventually have additional > > attributes which would apply to other tableams, but not to heap ? Is > > heap_insert_begin() re

Re: allow to \dtS+ pg_toast.*

2020-12-18 Thread Justin Pryzby
On Fri, Dec 18, 2020 at 12:43:07PM +0100, Laurenz Albe wrote: > On Fri, 2020-12-18 at 00:58 -0600, Justin Pryzby wrote: > > On Thu, Dec 17, 2020 at 04:16:52PM +0100, Laurenz Albe wrote: > > > On Mon, 2020-11-30 at 10:54 -0600, Justin Pryzby wrote: > > > > This makes toast tables a bit less special

Weird special case in jsonb_concat()

2020-12-18 Thread Tom Lane
The discussion in [1] pointed out that the existing documentation for the "jsonb || jsonb" concatenation operator is far short of reality: it fails to acknowledge that the operator will accept any cases other than two jsonb array inputs or two jsonb object inputs. I'd about concluded that other ca

Re: Proposed patch for key managment

2020-12-18 Thread Bruce Momjian
On Fri, Dec 18, 2020 at 11:13:44AM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Fri, Dec 18, 2020 at 08:18:53AM -0500, Stephen Frost wrote: > > > - C API in backend - we should try to at least set up the structure to > > > allow multiple encryptio

initdb fails under bleeding-edge valgrind

2020-12-18 Thread Tom Lane
With valgrind 3.16.1, we fail to get through initdb: ==00:00:00:41.608 11346== Source and destination overlap in memcpy(0xc190a8, 0xc190a8, 512) ==00:00:00:41.609 11346==at 0x486C674: __GI_memcpy (vg_replace_strmem.c:1035) ==00:00:00:41.609 11346==by 0x9017DB: write_relmap_file (relmappe

Re: Refactor routine to check for ASCII-only case

2020-12-18 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 18/12/2020 05:57, Michael Paquier wrote: > >As mentioned in [1], there are three places where there is the same > >routine to check if a string is made only of ASCII characters. > > > >This makes for a small-ish but nice cleanup, as per

Re: Proposed patch for key managment

2020-12-18 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Dec 18, 2020 at 08:18:53AM -0500, Stephen Frost wrote: > > - C API in backend - we should try to at least set up the structure to > > allow multiple encryption implementations, either via different > > libraries or if someone feels

Re: [DOC] Document concurrent index builds waiting on each other

2020-12-18 Thread James Coleman
On Tue, Dec 1, 2020 at 8:05 PM James Coleman wrote: > > On Tue, Dec 1, 2020 at 6:51 PM Alvaro Herrera wrote: > > > > On 2020-Nov-30, James Coleman wrote: > > > > > On Mon, Nov 30, 2020 at 4:53 PM Alvaro Herrera > > > wrote: > > > > > > > > On 2020-Sep-30, Michael Paquier wrote: > > > > > > Yeah

Re: how to use valgrind for TAP tests

2020-12-18 Thread Tom Lane
"osumi.takami...@fujitsu.com" writes: > I have a question about how to execute valgrind with TAP tests > in order to check some patches in the community. > My main interest is testing src/test/subscription now but > is there any general way to do it ? The standard solution is (1) Build normally

Re: Refactoring HMAC in the core code

2020-12-18 Thread Bruce Momjian
On Fri, Dec 18, 2020 at 03:46:42PM +0900, Michael Paquier wrote: > On Fri, Dec 18, 2020 at 08:41:01AM +0900, Michael Paquier wrote: > > Knowing that we are in a period of vacations for a lot of people, and > > that this is a sensitive area of the code that involves > > authentication, I think that

Re: Proposed patch for key managment

2020-12-18 Thread Bruce Momjian
On Fri, Dec 18, 2020 at 08:18:53AM -0500, Stephen Frost wrote: > What I would be thinking about with this are really three pieces- > > - C API for libpq (not relevant for this, but we have had issues in the > past with exposing OpenSSL-specific things there) Right. > - C API in backend - we sh

Re: pgbench failed when -f option contains a char '@'

2020-12-18 Thread Tom Lane
Heikki Linnakangas writes: > I think we should just leave this as it is. The user can simply rename > the file. Yeah. The assumption when we defined the script-weight syntax was that there's no particular reason to use "@" in a script file name, and I don't see why that's a bad assumption. > O

Re: Proposed patch for key managment

2020-12-18 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Dec 18, 2020 at 10:01:22AM +0900, Michael Paquier wrote: > > On Thu, Dec 17, 2020 at 12:10:22PM -0500, Bruce Momjian wrote: > > > On Thu, Dec 17, 2020 at 11:39:55AM -0500, Stephen Frost wrote: > > >> I don't think there's any need for

Re: postgres_fdw - cached connection leaks if the associated user mapping/foreign server is dropped

2020-12-18 Thread Bharath Rupireddy
On Fri, Dec 18, 2020 at 6:39 PM Bharath Rupireddy wrote: > > On Fri, Dec 18, 2020 at 5:06 PM Hou, Zhijie wrote: > > I have an issue about the existing testcase. > > > > """ > > -- Test that alteration of server options causes reconnection > > SELECT c3, c4 FROM ft1 ORDER BY c3, c1 LIMIT 1; -- sh

Re: Proposed patch for key managment

2020-12-18 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Thu, Dec 17, 2020 at 12:10:22PM -0500, Bruce Momjian wrote: > > Agreed. I think there is serious risk we would do AES in a different > > way than OpenSSL, especially if I did it. ;-) We can add a native AES > > one day if we want, b

Re: Single transaction in the tablesync worker?

2020-12-18 Thread Peter Smith
Hi Amit. PSA my v4 WIP patch for the Solution1. This patch applies onto the v30 patch set [1] from other 2PC thread: [1] https://www.postgresql.org/message-id/CAFPTHDYA8yE6tEmQ2USYS68kNt%2BkM%3DSwKgj%3Djy4AvFD5e9-UTQ%40mail.gmail.com Although incomplete it does still pass all the make check, an

Double partition lock in bufmgr

2020-12-18 Thread Konstantin Knizhnik
Hi hackers, I am investigating incident with one of out customers: performance of the system isdropped dramatically. Stack traces of all backends can be found here: http://www.garret.ru/diag_20201217_102056.stacks_59644 (this file is 6Mb so I have not attached it to this mail). What I have se

Re: allow to \dtS+ pg_toast.*

2020-12-18 Thread Laurenz Albe
On Fri, 2020-12-18 at 00:58 -0600, Justin Pryzby wrote: > On Thu, Dec 17, 2020 at 04:16:52PM +0100, Laurenz Albe wrote: > > On Mon, 2020-11-30 at 10:54 -0600, Justin Pryzby wrote: > > > This makes toast tables a bit less special and easier to inspect. > > > > > > postgres=# \dtS+ pg_toast.pg_toast

RE: postgres_fdw - cached connection leaks if the associated user mapping/foreign server is dropped

2020-12-18 Thread Hou, Zhijie
Hi I have an issue about the existing testcase. """ -- Test that alteration of server options causes reconnection SELECT c3, c4 FROM ft1 ORDER BY c3, c1 LIMIT 1; -- should work ALTER SERVER loopback OPTIONS (SET dbname 'no such database'); SELECT c3, c4 FROM ft1 ORDER BY c3, c1 LIMIT 1; -- s

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 12:55, Heikki Linnakangas wrote: On 18/12/2020 12:10, Michael Paquier wrote: On Fri, Dec 18, 2020 at 11:35:14AM +0200, Heikki Linnakangas wrote: pg_cryptohash_create() is now susceptible to leaking memory in TopMemoryContext, if the allocations fail. I think the attached should fi

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 12:10, Michael Paquier wrote: On Fri, Dec 18, 2020 at 11:35:14AM +0200, Heikki Linnakangas wrote: pg_cryptohash_create() is now susceptible to leaking memory in TopMemoryContext, if the allocations fail. I think the attached should fix it (but I haven't tested it at all). Yeah, y

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Michael Paquier
On Fri, Dec 18, 2020 at 11:51:55AM +0200, Heikki Linnakangas wrote: > Something like this. Passes regression tests, but otherwise untested. ... And I wanted to keep track of the type of cryptohash directly in the shared structure. ;) -- Michael signature.asc Description: PGP signature

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Michael Paquier
On Fri, Dec 18, 2020 at 11:35:14AM +0200, Heikki Linnakangas wrote: > pg_cryptohash_create() is now susceptible to leaking memory in > TopMemoryContext, if the allocations fail. I think the attached should fix > it (but I haven't tested it at all). Yeah, you are right here. If the second allocati

Re: Misleading comment in prologue of ReorderBufferQueueMessage

2020-12-18 Thread Ashutosh Bapat
On Wed, Dec 16, 2020 at 8:00 AM Amit Kapila wrote: > On Tue, Dec 15, 2020 at 11:25 AM Ashutosh Bapat > wrote: > > > > On Mon, Dec 14, 2020 at 3:14 PM Amit Kapila > wrote: > >> > >> On Mon, Dec 14, 2020 at 2:45 PM Ashutosh Bapat > >> wrote: > >> > > >> > The name of the function suggests that t

Re: Refactor routine to check for ASCII-only case

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 05:57, Michael Paquier wrote: As mentioned in [1], there are three places where there is the same routine to check if a string is made only of ASCII characters. This makes for a small-ish but nice cleanup, as per the attached. +1 - Heikki

Re: [BUG] orphaned function

2020-12-18 Thread Gilles Darold
Le 18/12/2020 à 00:26, Tom Lane a écrit : Gilles Darold writes: The same problem applies if the returned type or the procedural language is dropped. I have tried to fix that in ProcedureCreate() by using an AccessSharedLock for the data types and languages involved in the function declaration.

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 11:35, Heikki Linnakangas wrote: BTW, looking at pg_cryptohash_ctx and pg_cryptohash_state, why do we need two structs? They're both allocated and controlled by the cryptohash implementation. It would seem simpler to have just one. Something like this. Passes regression tests, but

Re: Deadlock between backend and recovery may not be detected

2020-12-18 Thread Fujii Masao
On 2020/12/17 2:15, Fujii Masao wrote: On 2020/12/16 23:28, Drouvot, Bertrand wrote: Hi, On 12/16/20 2:36 PM, Victor Yegorov wrote: *CAUTION*: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the co

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 09:35, Michael Paquier wrote: Hi all, As of the work done in 87ae9691, I have played with error injections in the code paths using this code, but forgot to count for cases where cascading resowner cleanups are involved. Like other resources (JIT, DSM, etc.), this requires an alloc

Re: pgbench failed when -f option contains a char '@'

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 08:22, Wang, Shenhao wrote: Hi, hackers pgbench use -f filename[@weight] to receive a sql script file with a weight, but if I create a file contains char'@', like a...@2.sql, specify this file without weigth, pgbench will failed with error: pgbench: fatal: invalid weight s

how to use valgrind for TAP tests

2020-12-18 Thread osumi.takami...@fujitsu.com
Hello, hackers I have a question about how to execute valgrind with TAP tests in order to check some patches in the community. My main interest is testing src/test/subscription now but is there any general way to do it ? The documentation [1] says "It's important to realize that the TAP tests wi

Re: Minor documentation error regarding streaming replication protocol

2020-12-18 Thread Kyotaro Horiguchi
At Fri, 18 Dec 2020 10:18:45 +0900, Michael Paquier wrote in > On Thu, Dec 17, 2020 at 03:13:25PM -0800, Jeff Davis wrote: > > On second thought, I'm going to retract this patch. The docs[1] say: > > > > "You can, if you like, add comments to a history file to record your > > own notes about ho