Re: [PoC] run SQL over ciphertext

2023-10-11 Thread Mingyu Li
Hi, > the idea is to split users into 'client users' who can see data unencrypted, and 'server users', who are administrators unable to decrypt data. Exactly! > how are secrets managed? Do you use a sort of vault to keep encryption keys? Good question. The client holds the key and uses a proxy

Re: RFC: Logging plan of the running query

2023-10-11 Thread Ashutosh Bapat
Hi, On Tue, Oct 10, 2023 at 7:00 PM torikoshia wrote: > Thanks for your comments! > > I tested v30 patch with > v28-0002-Testing-attempt-logging-plan-on-ever-CFI-call.patch which makes > CFI() call ProcessLogQueryPlanInterrupt() internally, and confirmed that > very few logging queries failed wi

Re: [PoC] run SQL over ciphertext

2023-10-11 Thread Mingyu Li
Hello Peter, > https://www.postgresql.org/message-id/flat/89157929-c2b6-817b-6025-8e4b2d89d...@enterprisedb.com Thanks for referring me to your TCE project, nice work! It takes time to go through the long thread of discussion and the patch. A quick question: what operations do pg_encrypted_* sup

Re: Pre-proposal: unicode normalized text

2023-10-11 Thread Jeff Davis
On Wed, 2023-10-11 at 08:56 +0200, Peter Eisentraut wrote: > On 11.10.23 03:08, Jeff Davis wrote: > >    * unicode_is_valid(text): returns true if all codepoints are > > assigned, false otherwise > > We need to be careful about precise terminology.  "Valid" has a > defined > meaning for Unicode. 

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-10-11 Thread Amit Kapila
On Tue, Oct 10, 2023 at 6:17 PM Amit Kapila wrote: > > DecodeTXNNeedSkip(LogicalDecodingContext *ctx, XLogRecordBuffer *buf, > Oid txn_dbid, RepOriginId origin_id) > { > - return (SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr) || > - (txn_dbid != InvalidOid && txn_dbid != ctx->s

Re: Problem, partition pruning for prepared statement with IS NULL clause.

2023-10-11 Thread David Rowley
On Wed, 11 Oct 2023 at 15:49, David Rowley wrote: > It might have been better if PartClauseInfo could also describe IS > NULL quals, but I feel if we do that now then it would require lots of > careful surgery in partprune.c to account for that. Probably the fix > should be localised to get_steps

Re: Pre-proposal: unicode normalized text

2023-10-11 Thread Jeff Davis
On Wed, 2023-10-11 at 08:51 +0200, Peter Eisentraut wrote: > I don't see how this would really work in practice.  Whether your > data > has unassigned code points or not, when the collations are updated to > the next Unicode version, the collations will have a new version > number, > and so you n

Re: False "pg_serial": apparent wraparound” in logs

2023-10-11 Thread Michael Paquier
On Thu, Oct 05, 2023 at 11:28:02PM +, Imseih (AWS), Sami wrote: > I spent sometime studying this and it appears to be a good approach. > > Passing the cutoff page as headPage (SLRU not needed code path ) instead of > the tailPage to > SimpleLruTruncate is already being done when the tailXid

Re: Fix typo in psql zh_CN.po

2023-10-11 Thread jinser
Sorry, I missed the pgsql-translators. Well, I searched for typo patches in pgsql-hackers before, and now that you reminded me, those were supposed to be source code typos... I'll send the patch to pgsql-translators :) Tom Lane 于 2023年10月11日周三 10:51写道: > Richard Guo writes: > > On Wed, Oct 11,

Re: LLVM 16 (opaque pointers)

2023-10-11 Thread Thomas Munro
On Thu, Sep 21, 2023 at 12:47 PM Thomas Munro wrote: > On Thu, Sep 21, 2023 at 12:24 PM Devrim Gündüz wrote: > > RHEL releases new LLVM version along with their new minor releases every > > 6 month, and we have to build older versions with new LLVM each time. > > From RHEL point of view, it would

Re: Problem, partition pruning for prepared statement with IS NULL clause.

2023-10-11 Thread tender wang
David Rowley 于2023年10月11日周三 15:52写道: > On Wed, 11 Oct 2023 at 15:49, David Rowley wrote: > > It might have been better if PartClauseInfo could also describe IS > > NULL quals, but I feel if we do that now then it would require lots of > > careful surgery in partprune.c to account for that. Prob

Re: Problem, partition pruning for prepared statement with IS NULL clause.

2023-10-11 Thread Sergei Glukhov
Hi, Thanks for fixing this! I verified that issues are fixed. On 10/11/23 11:50, David Rowley wrote: I'm aiming to commit these as two separate fixes, so I'm going to go look again at the first one and wait to see if anyone wants to comment on this patch in the meantime. Regarding test case

Re: Add null termination to string received in parallel apply worker

2023-10-11 Thread Peter Smith
Hi Hou-san. + /* + * Note that the data received via the shared memory queue is not + * null-terminated. So we use the StringInfo API to store the + * string so as to maintain the convention that StringInfos has a + * trailing null. + */ "... that StringInfos has a trailing null." Probably shoul

Re: Problem, partition pruning for prepared statement with IS NULL clause.

2023-10-11 Thread David Rowley
On Wed, 11 Oct 2023 at 22:09, Sergei Glukhov wrote: > Thanks for fixing this! > > I verified that issues are fixed. Thanks for having a look. Unfortunately, I'd not long sent the last email and realised that the step_lastkeyno parameter is now unused and can just be removed from both get_steps_u

Re: LLVM 16 (opaque pointers)

2023-10-11 Thread Ronan Dunklau
Le mercredi 11 octobre 2023, 10:59:50 CEST Thomas Munro a écrit : > The back-patch to 12 was a little trickier than anticipated, but after > taking a break and trying again I now have PG 12...17 patches that > I've tested against LLVM 10...18 (that's 54 combinations), in every > case only with the

Re: Problem, partition pruning for prepared statement with IS NULL clause.

2023-10-11 Thread Sergei Glukhov
On 10/11/23 13:19, David Rowley wrote: Thanks for having a look. Unfortunately, I'd not long sent the last email and realised that the step_lastkeyno parameter is now unused and can just be removed from both get_steps_using_prefix() and get_steps_using_prefix_recurse(). This requires some co

Re: Logging parallel worker draught

2023-10-11 Thread Alvaro Herrera
On 2023-Oct-09, Imseih (AWS), Sami wrote: > > I think we should definitely be afraid of that. I am in favor of a > > separate GUC. I agree. > Currently explain ( analyze ) will give you the "Workers Planned" > and "Workers launched". Logging this via auto_explain is possible, so I am > not sure

RE: [PoC] pg_upgrade: allow to upgrade publisher node

2023-10-11 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Thank you for reviewing! PSA new version. > > I think you need to set the new flag only when we are not skipping the > > transaction or in other words when we decide to process the > > transaction. Otherwise, how will you distinguish the case where the > > xact is already decoded and s

SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-11 Thread Dilip Kumar
The small size of the SLRU buffer pools can sometimes become a performance problem because it’s not difficult to have a workload where the number of buffers actively in use is larger than the fixed-size buffer pool. However, just increasing the size of the buffer pool doesn’t necessarily help, beca

Re: Is this a problem in GenericXLogFinish()?

2023-10-11 Thread Heikki Linnakangas
On 10/10/2023 22:57, Jeff Davis wrote: On Thu, 2023-09-28 at 12:05 -0700, Jeff Davis wrote: Also, I ran into some problems with GIN that might require a bit more refactoring in ginPlaceToPage(). Perhaps we could consider REGBUF_NO_CHANGE a general bypass of the Assert(BufferIsDirty()), and use i

Re: Is this a problem in GenericXLogFinish()?

2023-10-11 Thread Robert Haas
On Wed, Oct 11, 2023 at 7:53 AM Heikki Linnakangas wrote: > > + * Buffer must be pinned and exclusive-locked. (If caller does not hold > > + * exclusive lock, then the result may be stale before it's returned.) > The comment suggests that you don't need to hold an exclusive lock when > you call t

Re: Add null termination to string received in parallel apply worker

2023-10-11 Thread Amit Kapila
On Wed, Oct 11, 2023 at 12:18 PM Zhijie Hou (Fujitsu) wrote: > > The parallel apply worker didn't add null termination to the string received > from the leader apply worker via the shared memory queue. This action doesn't > bring bugs as it's binary data but violates the rule established in > Str

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-11 Thread Dilip Kumar
On Wed, Oct 11, 2023 at 4:34 PM Dilip Kumar wrote: > > The small size of the SLRU buffer pools can sometimes become a > performance problem because it’s not difficult to have a workload > where the number of buffers actively in use is larger than the > fixed-size buffer pool. However, just increas

Re: remaining sql/json patches

2023-10-11 Thread Amit Langote
On Wed, Oct 11, 2023 at 2:08 PM Amit Langote wrote: > On Sat, Oct 7, 2023 at 6:49 AM Andres Freund wrote: > > On 2023-09-29 13:57:46 +0900, Amit Langote wrote: > > > Thanks. I will push the attached 0001 shortly. > > > > Sorry for not looking at this earlier. > > Thanks for the review. Replying

Re: CREATE DATABASE with filesystem cloning

2023-10-11 Thread Robert Haas
On Mon, Oct 9, 2023 at 7:49 PM Andres Freund wrote: > If we do this, I think we should consider creating template0, template1 with > the new strategy, so that a new initdb cluster ends up with deduplicated data. Seems a little questionable given the reports earlier in the thread about some filesy

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Matthias van de Meent
On Wed, 11 Oct 2023 at 01:29, Andres Freund wrote: > > Hi, > > On 2023-10-10 21:30:44 +0200, Matthias van de Meent wrote: > > On Tue, 10 Oct 2023 at 06:14, Andres Freund wrote: > > > On 2023-10-09 23:16:30 -0400, Tom Lane wrote: > > >> Andres Freund writes: > > >>> There's an alternative approac

Re: [PATCH] Extend ALTER OPERATOR to support adding commutator, negator, hashes, and merges

2023-10-11 Thread Tommy Pavlicek
On Tue, Oct 10, 2023 at 9:32 PM Tom Lane wrote: > > Tommy Pavlicek writes: > > I did notice one further potential bug. When creating an operator and > > adding a commutator, PostgreSQL only links the commutator back to the > > operator if the commutator has no commutator of its own, but the > > c

Re: Add null termination to string received in parallel apply worker

2023-10-11 Thread Alvaro Herrera
On 2023-Oct-11, Amit Kapila wrote: > Yeah, it may not be a good idea to modify the buffer pointing to > shared memory without any lock as we haven't reserved that part of > memory. So, we can't follow the trick used in exec_bind_message() to > maintain the convention that StringInfos have a traili

Re: Logging parallel worker draught

2023-10-11 Thread Imseih (AWS), Sami
>> Currently explain ( analyze ) will give you the "Workers Planned" >> and "Workers launched". Logging this via auto_explain is possible, so I am >> not sure we need additional GUCs or debug levels for this info. >> >> -> Gather (cost=10430.00..10430.01 rows=2 width=8) (actual tim >> e=131.826..13

Re: Add null termination to string received in parallel apply worker

2023-10-11 Thread Tom Lane
Alvaro Herrera writes: > I was thinking about this when skimming the other StringInfo thread a > couple of days ago. I wondered if it wouldn't be more convenient to > change the convention that all StringInfos are null-terminated: what is > really the reason to have them all be like that? It mak

Re: Check each of base restriction clauses for constant-FALSE-or-NULL

2023-10-11 Thread Tom Lane
I wrote: > So this logic in relation_excluded_by_constraints is just wrong --- > premature optimization on my part, looks like. Pushed after a bit of fiddling with the comment. regards, tom lane

Re: interval_ops shall stop using btequalimage (deduplication)

2023-10-11 Thread Noah Misch
On Tue, Oct 10, 2023 at 09:35:45PM -0700, Peter Geoghegan wrote: > On Tue, Oct 10, 2023 at 8:51 PM Peter Geoghegan wrote: > > I don't see any reason to delay committing your fix. The issue that > > you've highlighted is exactly the kind of issue that I anticipated > > might happen at some point. T

Re: stopgap fix for signal handling during restore_command

2023-10-11 Thread Nathan Bossart
On Wed, Oct 11, 2023 at 01:02:14PM +0900, Michael Paquier wrote: > On Tue, Oct 10, 2023 at 08:39:29PM -0700, Andres Freund wrote: >> We shouldn't call proc_exit() in a signal handler. We perhaps have a few >> remaining calls left, but we should (and I think in some cases are) working >> on >> remo

Re: interval_ops shall stop using btequalimage (deduplication)

2023-10-11 Thread Peter Geoghegan
On Wed, Oct 11, 2023 at 11:38 AM Noah Misch wrote: > Interesting. So, >99% of interval-type indexes, even ones WITH > (deduplicate_items=off), will get amcheck failures. The <1% of exceptions > might include indexes having allequalimage=off due to an additional column, > e.g. a two-column (inter

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Robert Haas
On Tue, Oct 10, 2023 at 7:29 PM Andres Freund wrote: > > Hmm. I don't think we should remove those checks, as I can see people > > that would want to change their XLog block size with e.g. > > pg_reset_wal. > > I don't think that's something we need to address in every physical > segment. For one,

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Thomas Munro
On Thu, Oct 12, 2023 at 9:05 AM Robert Haas wrote: > But if we do want to keep those cross-checks, why not take what Thomas > proposed a little further and move all of xlp_sysid, xlp_seg_size, and > xlp_xlog_blcksz into XLOG_CHECKPOINT_REDO? Then long and short page > headers would become identica

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Thomas Munro
On Thu, Oct 12, 2023 at 9:27 AM Thomas Munro wrote: > On Thu, Oct 12, 2023 at 9:05 AM Robert Haas wrote: > > But if we do want to keep those cross-checks, why not take what Thomas > > proposed a little further and move all of xlp_sysid, xlp_seg_size, and > > xlp_xlog_blcksz into XLOG_CHECKPOINT_R

Re: A new strategy for pull-up correlated ANY_SUBLINK

2023-10-11 Thread Alena Rybakina
Hi! I reviewed your patch and it was interesting for me! Thank you for the explanation. It was really informative for me! I think we need the restriction and that should be enough for this feature . Given the query Richard provided before: explain select * from tenk1 A where exists (select 1

Regarding Postgresql Transaction isolation

2023-10-11 Thread Ajay P S
Hi, I understand that in READ COMMITTED isolation level, SELECT queries reads a snapshot of the database as of the instant the query begins. And also a concurrent transaction(uncommitted) writing to the same table won't block the readers. However, I see that in the heap_update(heapam.c) function t

Re: Add null termination to string received in parallel apply worker

2023-10-11 Thread David Rowley
On Thu, 12 Oct 2023 at 05:04, Tom Lane wrote: > > Alvaro Herrera writes: > > I was thinking about this when skimming the other StringInfo thread a > > couple of days ago. I wondered if it wouldn't be more convenient to > > change the convention that all StringInfos are null-terminated: what is >

Re: New WAL record to detect the checkpoint redo location

2023-10-11 Thread Thomas Munro
On Tue, Oct 10, 2023 at 11:33 AM Robert Haas wrote: > On Mon, Oct 9, 2023 at 4:47 PM Andres Freund wrote: > > I think we might be able to speed some of this up by pre-compute values so > > we > > can implement things like bytesleft / UsableBytesInPage with shifts. IIRC we > > already insist on p

Re: Regarding Postgresql Transaction isolation

2023-10-11 Thread Tom Lane
Ajay P S writes: > However, I see that in the heap_update(heapam.c) function there is a > brief interval(Lock and unlock the buffer) where a writer may block > readers if the writer is updating the same row which readers are > reading. > Could anyone please help me with the below query? > 1) Is m

Re: The danger of deleting backup_label

2023-10-11 Thread Thomas Munro
Hi David, Even though I spent a whole bunch of time trying to figure out how to make concurrent reads of the control file sufficiently atomic for backups (pg_basebackup and low level filesystem tools), and we explored multiple avenues with varying results, and finally came up with something that b

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Andres Freund
Hi, On 2023-10-11 16:05:02 -0400, Robert Haas wrote: > On Tue, Oct 10, 2023 at 7:29 PM Andres Freund wrote: > > > Hmm. I don't think we should remove those checks, as I can see people > > > that would want to change their XLog block size with e.g. > > > pg_reset_wal. > > > > I don't think that's

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Andres Freund
Hi, On 2023-10-11 16:09:21 +0200, Matthias van de Meent wrote: > On Wed, 11 Oct 2023 at 01:29, Andres Freund wrote: > > > After that we'll only have the system ID left from the extended > > > header, which we could store across 2 pages in the (current) alignment > > > losses of xlp_rem_len - even

Re: The danger of deleting backup_label

2023-10-11 Thread Michael Paquier
On Tue, Oct 10, 2023 at 05:06:45PM -0400, David Steele wrote: > That fails because there is a check to make sure the checkpoint is valid > when pg_control is loaded. Another possibility is to use a special LSN like > we use for unlogged tables. Anything >= 24 and < WAL segment size will work > fine

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-10-11 Thread Andres Freund
Hi, On 2023-10-03 16:05:32 -0700, Jeff Davis wrote: > On Sat, 2023-01-14 at 12:34 -0800, Andres Freund wrote: > > One benefit would be that it'd make it more realistic to use direct > > IO for WAL > > - for which I have seen significant performance benefits. But when we > > afterwards have to re-r

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-10-11 Thread Thomas Munro
I'm planning to push 0002 (retries in frontend programs, which is where this thread began) and 0004 (add missing locks to SQL functions), including back-patches as far as 12, in a day or so. I'll abandon the others for now, since we're now thinking bigger[1] for backups, side stepping the problem.

Re: Add a new BGWORKER_BYPASS_ROLELOGINCHECK flag

2023-10-11 Thread Michael Paquier
On Wed, Oct 11, 2023 at 08:48:04AM +0200, Drouvot, Bertrand wrote: > Yeah, agree. Changed in v12 attached. I have tweaked a few comments, and applied that. Thanks. -- Michael signature.asc Description: PGP signature

Re: Eager page freeze criteria clarification

2023-10-11 Thread Andres Freund
Hi, Robert, Melanie and I spent an evening discussing this topic around pgconf.nyc. Here are, mildly revised, notes from that: First a few random points that didn't fit with the sketch of an approach below: - Are unlogged tables a problem for using LSN based heuristics for freezing? We conc

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-10-11 Thread Michael Paquier
On Thu, Oct 12, 2023 at 12:25:34PM +1300, Thomas Munro wrote: > I'm planning to push 0002 (retries in frontend programs, which is > where this thread began) and 0004 (add missing locks to SQL > functions), including back-patches as far as 12, in a day or so. > > I'll abandon the others for now, si

Wait events for delayed checkpoints

2023-10-11 Thread Thomas Munro
Hi, You can't tell if your checkpointer is spending a lot of time waiting around for flags in delayChkptFlags to clear. Trivial patch to add that. I've managed to see it a few times when checkpointing repeatedly with a heavy pgbench workload. I had to stop and think for a moment about whether t

Memory knob testing (was Re: Let's make PostgreSQL multi-threaded)

2023-10-11 Thread Merlin Moncure
On Fri, Aug 25, 2023 at 8:35 AM Stephen Frost wrote: > Greetings, > > This is getting a bit far afield in terms of this specific thread, but > there's an ongoing effort to give PG administrators knobs to be able to > control how much actual memory is used rather than depending on the > kernel to

RE: Add null termination to string received in parallel apply worker

2023-10-11 Thread Zhijie Hou (Fujitsu)
On Thursday, October 12, 2023 12:04 AM Tom Lane wrote: Hi, > > Alvaro Herrera writes: > > I was thinking about this when skimming the other StringInfo thread a > > couple of days ago. I wondered if it wouldn't be more convenient to > > change the convention that all StringInfos are null-termi

Re: Synchronizing slots from primary to standby

2023-10-11 Thread shveta malik
On Tue, Oct 10, 2023 at 12:52 PM Peter Smith wrote: > > On Mon, Oct 9, 2023 at 9:34 PM shveta malik wrote: > > > > On Wed, Oct 4, 2023 at 8:53 AM Peter Smith wrote: > > > > > > Here are some review comments for v20-0002. > > > > > > > Thanks Peter for the feedback. Comments from 31 till end are

Re: [PATCH] Extend ALTER OPERATOR to support adding commutator, negator, hashes, and merges

2023-10-11 Thread jian he
errmsg("operator attribute \"negator\" cannot be changed if it has already been set"))); I feel like the above message is not very helpful. Something like the following may be more helpful for diagnosis. errmsg("operator %s's attribute \"negator\" cannot be changed if it has already been set", ope

Re: Synchronizing slots from primary to standby

2023-10-11 Thread shveta malik
On Tue, Oct 10, 2023 at 12:52 PM Peter Smith wrote: > > On Mon, Oct 9, 2023 at 9:34 PM shveta malik wrote: > > > > On Wed, Oct 4, 2023 at 8:53 AM Peter Smith wrote: > > > > > > Here are some review comments for v20-0002. > > > > > > > Thanks Peter for the feedback. Comments from 31 till end are

Re: PGDOCS - add more links in the pub/sub reference pages

2023-10-11 Thread Amit Kapila
On Mon, Oct 9, 2023 at 12:15 PM Peter Smith wrote: > > On Mon, Oct 9, 2023 at 3:32 PM Amit Kapila wrote: > > > > In v1, I used the same pattern as on the CREATE SUBSCRIPTION page, > which doesn't look like those... > Yeah, I think it would have been better if we used params in the CREATE SUBSCRI

Re: Invalidate the subscription worker in cases where a user loses their superuser status

2023-10-11 Thread Amit Kapila
On Sun, Oct 8, 2023 at 8:22 AM vignesh C wrote: > --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -127,6 +127,7 @@ typedef struct Subscription * skipped */ char*name; /* Name of the subscription */ Oid owner; /* Oid of the subscription owner */

Re: Add a new BGWORKER_BYPASS_ROLELOGINCHECK flag

2023-10-11 Thread Drouvot, Bertrand
Hi, On 10/12/23 2:26 AM, Michael Paquier wrote: On Wed, Oct 11, 2023 at 08:48:04AM +0200, Drouvot, Bertrand wrote: Yeah, agree. Changed in v12 attached. I have tweaked a few comments, and applied that. Thanks. Oh and you also closed the CF entry, thanks! Regards, -- Bertrand Drouvot Post

Re: Tab completion for AT TIME ZONE

2023-10-11 Thread Michael Paquier
On Fri, Apr 14, 2023 at 12:05:25PM +0200, Jim Jones wrote: > The patch applies cleanly and it does what it is proposing. - and it's IMHO > a very nice addition. > > I've marked the CF entry as "Ready for Committer". +/* ... AT TIME ZONE ... */ + else if (TailMatches("AT")) + C