RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-27 Thread Takamichi Osumi (Fujitsu)
On Wednesday, January 25, 2023 11:24 PM I wrote: > Attached the updated v22. Hi, During self-review, I noticed some changes are required for some variable types related to 'min_apply_delay' value, so have conducted the adjustment changes for the same. Additionally, I made some comments for trans

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Alvaro Herrera
On 2023-Jan-27, Bharath Rupireddy wrote: > Looking at the patch, the feature, in its current shape, focuses on > improving replication lag (by throttling WAL on the primary) only when > synchronous replication is enabled. Why is that? Why can't we design > it for replication in general (async, syn

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-26 23:11:41 -0800, Peter Geoghegan wrote: > > Essentially the "any fpi" logic is a very coarse grained way of using the > > page > > LSN as a measurement. As I said, I don't think "has a checkpoint occurred > > since the last write" is a good metric to avoid unnecessary freezing -

Re: Lazy JIT IR code generation to increase JIT speed with partitions

2023-01-27 Thread David Geier
Hi, Thanks for taking a look! On 12/1/22 22:12, Dmitry Dolgov wrote: First to summarize things a bit: from what I understand there are two suggestions on the table, one is about caching modules when doing inlining, the second is about actual lazy jitting. Are those to tightly coupled together,

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 00:51:59 -0800, Andres Freund wrote: > One use-case where the eager strategy is particularly useful is > [nearly-]append-only tables - and it's also the one workload that's reasonably > easy to detect using stats. Maybe something like > (dead_tuples_since_last_vacuum / inserts_si

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Thomas Munro
After 1000 make check loops, and 1000 make -C src/test/modules/test_shm_mq check loops, on the same FBSD 13.1 machine as elver which has failed like this once before, I haven't been able to reproduce this on REL_12_STABLE. Not really sure how to chase this, but if you see this situation again, I'd

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-27 Thread Bharath Rupireddy
On Fri, Jan 27, 2023 at 12:16 PM Masahiko Sawada wrote: > > I'd like to confirm whether there is any performance regression caused > by this patch in some cases, especially when not using DIO. Thanks. I ran some insert tests with primary and 1 async standby. Please see the numbers below and attac

Re: improving user.c error messages

2023-01-27 Thread Peter Eisentraut
On 26.01.23 01:22, Nathan Bossart wrote: Here is an early draft of some modest improvements to the user.c error messages. I basically just tried to standardize the style of and add context to the existing error messages. I used errhint() for this extra context, but errdetail() would work, too.

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-27 Thread shveta malik
On Thu, Jan 26, 2023 at 7:53 PM Melih Mutlu wrote: > > If a relation is currently being synced by a tablesync worker and uses a > replication slot/origin for that operation, then srrelslotname and > srreloriginname fields will have values. > When a relation is done with its replication slot/orig

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread vignesh C
On Mon, 23 Jan 2023 at 10:52, Amit Kapila wrote: > > IIRC, this is done to prevent concurrent drops of origin drop say by > exposed API pg_replication_origin_drop(). See the discussion in [1] > related to it. If we want we can optimize it so that we can acquire > the lock on the specific origin as

Re: meson oddities

2023-01-27 Thread Peter Eisentraut
On 26.01.23 19:05, Andres Freund wrote: Oh, that is interesting. In that case, this is not the right patch. We should proceed with my previous patch in [0] then. WFM. I still think it'd be slightly more legible if we tested the prefix for postgres|pgsql once, rather than do the per-variable .

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 1:39 PM Takamichi Osumi (Fujitsu) wrote: > > On Wednesday, January 25, 2023 11:24 PM I wrote: > > Attached the updated v22. > Hi, > > During self-review, I noticed some changes are > required for some variable types related to 'min_apply_delay' value, > so have conducted th

RE: Assertion failure in SnapBuildInitialSnapshot()

2023-01-27 Thread Hayato Kuroda (Fujitsu)
Dear Sawada-san, Thank you for making the patch! I'm still considering whether this approach is correct, but I can put a comment to your patch anyway. ``` - Assert(!already_locked || LWLockHeldByMe(ProcArrayLock)); - - if (!already_locked) - LWLockAcquire(ProcArrayLock,

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Bharath Rupireddy
On Fri, Jan 27, 2023 at 2:03 PM Alvaro Herrera wrote: > > On 2023-Jan-27, Bharath Rupireddy wrote: > > > Looking at the patch, the feature, in its current shape, focuses on > > improving replication lag (by throttling WAL on the primary) only when > > synchronous replication is enabled. Why is tha

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Jakub Wartak
Hi, v2 is attached. On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: > Huh? Why did you remove the GUC? After reading previous threads, my optimism level of getting it ever in shape of being widely accepted degraded significantly (mainly due to the discussion of wider category of 'WAL I/O

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Jakub Wartak
Hi Bharath, On Fri, Jan 27, 2023 at 12:04 PM Bharath Rupireddy wrote: > > On Fri, Jan 27, 2023 at 2:03 PM Alvaro Herrera > wrote: > > > > On 2023-Jan-27, Bharath Rupireddy wrote: > > > > > Looking at the patch, the feature, in its current shape, focuses on > > > improving replication lag (by th

Re: [DOCS] Stats views and functions not in order?

2023-01-27 Thread Peter Eisentraut
On 19.01.23 00:45, Peter Smith wrote: The original $SUBJECT requirements evolved to also try to make each view appear on a separate page after that was suggested by DavidJ [2]. I was unable to achieve per-page views "without radically changing the document structure." [3], but DavidJ found a way

RE: Logical replication timeout problem

2023-01-27 Thread houzj.f...@fujitsu.com
On Wednesday, January 25, 2023 7:26 PM Amit Kapila > > On Tue, Jan 24, 2023 at 8:15 AM wangw.f...@fujitsu.com > wrote: > > > > Attach the new patch. > > > > I think the patch missed to handle the case of non-transactional messages > which > was previously getting handled. I have tried to addre

Re: Logical replication timeout problem

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 5:18 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, January 25, 2023 7:26 PM Amit Kapila > > > > On Tue, Jan 24, 2023 at 8:15 AM wangw.f...@fujitsu.com > > wrote: > > > > > > Attach the new patch. > > > > > > > I think the patch missed to handle the case of non-transa

Re: PG11 to PG14 Migration Slowness

2023-01-27 Thread Vigneshk Kvignesh
Hi, Sorry for the delayed response. We have an fdw extension, we started code changes in the extension for PGv14 on 14.3, we just completed code changes, testing and benchmarking. We'll retarget to 14.6 Also we'll take a look at the changes for pg_dump in v15 . Thanks for the advice. Thanks and R

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila wrote: > > > > IIRC, this is done to prevent concurrent drops of origin drop say by > > exposed API pg_replication_origin_drop(). See the discussion in [1] > > related to it. If we want we can optimize

Add a test case related to the error "cannot fetch toast data without an active snapshot"

2023-01-27 Thread Nitin Jadhav
Hi, I was going through the comments [1] mentioned in init_toast_snapshot() and based on the comments understood that the error "cannot fetch toast data without an active snapshot" will occur if a procedure fetches a toasted value into a local variable, commits, and then tries to detoast the value

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-27 Thread Önder Kalacı
Hi Marco, Tom, > But still it doesn't seem to me to be appropriate to use the planner to find a suitable index. As Marco noted, here we are trying to pick an index that is non-unique. We could pick the index based on information extracted from pg_index (or such), but then, it'd be a premature sel

Re: Improve GetConfigOptionValues function

2023-01-27 Thread Nitin Jadhav
> Both of you are arguing as though GUC_NO_SHOW_ALL is a security > property. It is not, or at least it's so trivially bypassable > that it's useless to consider it one. All it is is a de-clutter > mechanism. Understood. If that is the case, then I am ok with the patch. Thanks & Regards, Nitin

Re: improving user.c error messages

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 5:00 AM Peter Eisentraut wrote: > This is good. If I may assign some more work ;-), we have a bunch of > error messages like > > errmsg("must be superuser or a role with privileges of the > pg_write_server_files role to create backup stored on server") > > errmsg("must be

Timeline ID hexadecimal format

2023-01-27 Thread Sébastien Lardière
Hi, I've been puzzled by this message: ~~~ LOG:  fetching timeline history file for timeline 17 from primary server FATAL:  could not receive timeline history file from the primary server: ERROR:  could not open file "pg_xlog/0011.history": No such file or directory ~~~ It took me a whil

Re: GUC for temporarily disabling event triggers

2023-01-27 Thread Mikhail Gribkov
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Hi Daniel, I have reviewed the patch and I liked it (well I

Re: GUCs to control abbreviated sort keys

2023-01-27 Thread Robert Haas
On Wed, Jan 25, 2023 at 4:16 PM Jeff Davis wrote: > $ perl text_generator.pl 1000 10 > /tmp/strings.txt > > CREATE TABLE s (t TEXT); > COPY s FROM '/tmp/strings.txt'; > VACUUM FREEZE s; > CHECKPOINT; > SET work_mem='10GB'; > SET max_parallel_workers = 0; > SET max_parallel_workers_per_gather =

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-27 Thread vignesh C
On Thu, 26 Jan 2023 at 19:53, Melih Mutlu wrote: > > Hi Shveta, > > Thanks for reviewing. > > shveta malik , 25 Oca 2023 Çar, 16:02 tarihinde şunu > yazdı: >> >> On Mon, Jan 23, 2023 at 6:30 PM Melih Mutlu wrote: >> --I see initial data copied, but new catalog columns srrelslotname >> and srrelo

Re: Inconsistency in reporting checkpointer stats

2023-01-27 Thread Nitin Jadhav
> IMO, there's no need for 2 separate patches for these changes. I will make it a single patch while sharing the next patch. > +(errmsg("restartpoint complete: wrote %d buffers (%.1f%%), " > +"wrote %d slru buffers (%.1f%%); %d WAL > file(s) added, " > +

Re: SQL/JSON revisited

2023-01-27 Thread vignesh C
On Tue, 17 Jan 2023 at 19:01, Amit Langote wrote: > > On Wed, Dec 28, 2022 at 4:28 PM Amit Langote wrote: > > > > Hi, > > > > Rebased the SQL/JSON patches over the latest HEAD. I've decided to > > keep the same division of code into individual commits as that > > mentioned in the revert commit 2

Re: old_snapshot_threshold bottleneck on replica

2023-01-27 Thread Maxim Orlov
On Wed, 25 Jan 2023 at 16:52, Robert Haas wrote: > On Wed, Jan 25, 2023 at 3:52 AM Maxim Orlov wrote: > > Well, that's something we - and ideally you, as the patch author - > need to analyze and figure out. We can't just take a shot and hope for > the best. > I thank you for your advices. I've

Re: Latches vs lwlock contention

2023-01-27 Thread vignesh C
On Tue, 1 Nov 2022 at 16:40, Thomas Munro wrote: > > On Fri, Oct 28, 2022 at 4:56 PM Thomas Munro wrote: > > See attached sketch patches. I guess the main thing that may not be > > good enough is the use of a fixed sized latch buffer. Memory > > allocation in don't-throw-here environments like

Re: logical decoding and replication of sequences, take 2

2023-01-27 Thread vignesh C
On Mon, 16 Jan 2023 at 04:49, Tomas Vondra wrote: > > cfbot didn't like the rebased / split patch, and after looking at it I > believe it's a bug in parallel apply of large transactions (216a784829), > which seems to have changed interpretation of in_remote_transaction and > in_streamed_transactio

Re: pg_stat_statements and "IN" conditions

2023-01-27 Thread vignesh C
On Sun, 25 Sept 2022 at 05:29, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Sat, Sep 24, 2022 at 04:07:14PM +0200, Dmitry Dolgov wrote: > > > On Fri, Sep 16, 2022 at 09:25:13PM +0300, Sergei Kornilov wrote: > > > Hello! > > > > > > Unfortunately the patch needs another rebase due to the re

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Robert Haas
On Thu, Jan 26, 2023 at 4:51 PM Peter Geoghegan wrote: > This is the kind of remark that makes me think that you don't get it. > > The most influential OLTP benchmark of all time is TPC-C, which has > exactly this problem. In spades -- it's enormously disruptive. Which > is one reason why I used i

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Peter Eisentraut
On 25.01.23 22:25, Tom Lane wrote: So this invents an initdb switch "-c NAME=VALUE" just like the one that the server itself has long had. This seems useful. The specified settings are applied on the command line of the initial probe calls (which happen before we've made any config files), an

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Robert Haas
On Thu, Jan 26, 2023 at 6:37 PM Peter Geoghegan wrote: > > I don't see what your reference to checkpoint timeout is about here? > > > > Also, as I mentioned before, the problem isn't specific to > > checkpoint_timeout > > = 1min. It just makes it cheaper to reproduce. > > That's flagrantly intell

Re: Timeline ID hexadecimal format

2023-01-27 Thread Peter Eisentraut
On 27.01.23 14:52, Sébastien Lardière wrote: The attached patch proposes to change the format of timelineid from %u to %X. I think your complaint has merit. But note that if we did a change like this, then log files or reports from different versions would have different meaning without a vi

Re: run pgindent on a regular basis / scripted manner

2023-01-27 Thread Andrew Dunstan
On 2023-01-26 Th 17:54, Jelte Fennema wrote: > >> I'm still mildly inclined to say this material would be better placed >> in the developer wiki. After all, this isn't the only thing a postgres >> developer might use a git hook for > I think it should definitely be somewhere. I have a preference

Re: old_snapshot_threshold bottleneck on replica

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 9:30 AM Maxim Orlov wrote: > I thank you for your advices. I've dived deeper into the problem and I think > v2 patch is wrong. Cool! > Accessing threshold_timestamp and threshold_xid in > TransactionIdLimitedForOldSnapshots > without lock would lead to an improper xlimi

Re: Lazy JIT IR code generation to increase JIT speed with partitions

2023-01-27 Thread Dmitry Dolgov
> On Fri, Jan 27, 2023 at 10:02:32AM +0100, David Geier wrote: > It's very curious as to why we didn't really see that when dumping the > bitcode. It seems like the bitcode is always different enough to not spot > that. As I've noted off-list, there was noticeable difference in the dumped bitcode,

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Isaac Morland
On Fri, 27 Jan 2023 at 09:49, Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 25.01.23 22:25, Tom Lane wrote: > > So this invents an initdb switch "-c NAME=VALUE" just like the > > one that the server itself has long had. > > This seems useful. > > > The specified settings > > ar

Re: Named Operators

2023-01-27 Thread Peter Eisentraut
On 12.01.23 14:55, Matthias van de Meent wrote: Matter of taste, I guess. But more importantly, defining an operator gives you many additional features that the planner can use to optimize your query differently, which it can't do with functions. See the COMMUTATOR, HASHES, etc. clause in the CRE

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Robert Haas
On Wed, Jan 25, 2023 at 4:26 PM Tom Lane wrote: > So this invents an initdb switch "-c NAME=VALUE" just like the > one that the server itself has long had. HUGE +1 from me. This will, I think, be extremely convenient in many situations. > The specified settings > are applied on the command line

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
Peter Eisentraut writes: > On 25.01.23 22:25, Tom Lane wrote: >> The specified settings >> are applied on the command line of the initial probe calls >> (which happen before we've made any config files), and then they >> are added to postgresql.auto.conf, which causes them to take >> effect for th

Re: Named Operators

2023-01-27 Thread Matthias van de Meent
On Fri, 27 Jan 2023 at 16:26, Peter Eisentraut wrote: > > On 12.01.23 14:55, Matthias van de Meent wrote: > >> Matter of taste, I guess. But more importantly, defining an operator > >> gives you many additional features that the planner can use to > >> optimize your query differently, which it can

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 10:34 AM Tom Lane wrote: > One idea if we want to make it work like that could be to stop > trying to edit out the default value, and instead make the file > contents look like, say, > > #huge_pages = try # on, off, or try > huge_pages = off

Re: improving user.c error messages

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 08:31:32AM -0500, Robert Haas wrote: > I almost hate to bring this up since I'm not sure how far we want to > go down this rat hole, but what should be our policy about mentioning > superuser? I don't think we're entirely consistent right now, and I'm > not sure whether ever

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
Robert Haas writes: > The idea is that instead of: > replace_token(conflines, "#max_connections = 100", repltok); > You'd write something like: > replace_guc_value(conflines, "max_connections", repltok); > Which would look for a line matching /^#max_connections\s+=\s/, and > then identify ever

Re: improving user.c error messages

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 10:52 AM Nathan Bossart wrote: > IMHO superuser should typically only be mentioned when it is the only way > to do something. Since superusers have all privileges, I think logs like > "superuser or privileges of X" are kind of redundant. If Robert has > privileges of X, w

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread David G. Johnston
On Fri, Jan 27, 2023 at 8:53 AM Tom Lane wrote: > Robert Haas writes: > > The idea is that instead of: > > > replace_token(conflines, "#max_connections = 100", repltok); > > > You'd write something like: > > > replace_guc_value(conflines, "max_connections", repltok); > > > Which would look for a

Re: improving user.c error messages

2023-01-27 Thread Tom Lane
Robert Haas writes: > I almost hate to bring this up since I'm not sure how far we want to > go down this rat hole, but what should be our policy about mentioning > superuser? I don't think we're entirely consistent right now, and I'm > not sure whether every error message needs to mention that if

Re: Timeline ID hexadecimal format

2023-01-27 Thread Sébastien Lardière
On 27/01/2023 15:55, Peter Eisentraut wrote: On 27.01.23 14:52, Sébastien Lardière wrote: The attached patch proposes to change the format of timelineid from %u to %X. I think your complaint has merit.  But note that if we did a change like this, then log files or reports from different versi

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 6:48 AM Robert Haas wrote: > > One of the key strengths of systems like Postgres is the ability to > > inexpensively store a relatively large amount of data that has just > > about zero chance of being read, let alone modified. While at the same > > time having decent OLTP

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
"David G. Johnston" writes: > On Fri, Jan 27, 2023 at 8:53 AM Tom Lane wrote: >> Anyway, it seems like I gotta work harder. I'll produce a >> new patch. > How about just adding a "section" to the end of the file as needed: > # AdHoc Settings Specified During InitDB > max_connections=75 > ...

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-27 Thread Masahiko Sawada
On Wed, Jan 25, 2023 at 3:27 PM Amit Kapila wrote: > > On Wed, Jan 25, 2023 at 10:05 AM Amit Kapila wrote: > > > > On Wed, Jan 25, 2023 at 3:15 AM Peter Smith wrote: > > > > > > 1. > > > @@ -210,7 +210,7 @@ int logical_decoding_work_mem; > > > static const Size max_changes_in_memory = 4096; /*

Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread João Paulo Labegalini de Carvalho
Hi all, I am investigating the benefits of different profile-guided optimizations (PGO) and link-time optimizations (LTO) versus binary optimizers (e.g. BOLT) for applications such as PostgreSQL. I am facing issues when applying LTO to PostgreSQL as the produced binary seems broken (the server di

Re: Improve GetConfigOptionValues function

2023-01-27 Thread Tom Lane
Nitin Jadhav writes: >> Both of you are arguing as though GUC_NO_SHOW_ALL is a security >> property. It is not, or at least it's so trivially bypassable >> that it's useless to consider it one. All it is is a de-clutter >> mechanism. > Understood. If that is the case, then I am ok with the patc

Re: Add LZ4 compression in pg_dump

2023-01-27 Thread Justin Pryzby
On Thu, Jan 26, 2023 at 12:22:45PM -0600, Justin Pryzby wrote: > That commit also added this to pg-dump.c: > > + case PG_COMPRESSION_ZSTD: > + pg_fatal("compression with %s is not yet supported", > "ZSTD"); > + break; > + cas

Re: Show various offset arrays for heap WAL records

2023-01-27 Thread Melanie Plageman
Hi, I have taken a stab at doing some of the tasks listed in this email. I have made the new files rmgr_utils.c/h. I have come up with a standard format that I like for the output and used it in all the heap record types. Examples below: snapshotConflictHorizon: 2184, nplans: 2, plans [ { xmax

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 12:58 AM Andres Freund wrote: > Essentially the "any fpi" logic is a very coarse grained way of using the page > LSN as a measurement. As I said, I don't think "has a checkpoint occurred > since the last write" is a good metric to avoid unnecessary freezing - it's > too coa

Re: improving user.c error messages

2023-01-27 Thread Alvaro Herrera
On 2023-Jan-27, Tom Lane wrote: > Good point. My vote is for standardizing on *not* mentioning it. > Error messages should say "you need privilege X". That is not > the place to go into all the ways you could hold privilege X > (one of which is being superuser). +1 -- Álvaro Herrera

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:53:58 -0500, Robert Haas wrote: > On Fri, Jan 27, 2023 at 12:58 AM Andres Freund wrote: > > Essentially the "any fpi" logic is a very coarse grained way of using the > > page > > LSN as a measurement. As I said, I don't think "has a checkpoint occurred > > since the last wri

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 12:52 AM Andres Freund wrote: > I agree with bringing high-level context into the decision about whether to > freeze agressively - my problem with the eager freezing strategy patch isn't > that it did that too much, it's that it didn't do it enough. > > > But I also don't t

Re: improving user.c error messages

2023-01-27 Thread Alvaro Herrera
While we're here, On 2023-Jan-26, Nathan Bossart wrote: > @@ -838,7 +867,8 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt) > if (!should_be_super && roleid == BOOTSTRAP_SUPERUSERID) > ereport(ERROR, > > (errcode(ERRC

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Komяpa
Hi, We have implemented LTO in PostGIS's build system a couple releases ago. It definitely gives +10% on heavy maths. Unfortunately we did not manage to get it running under FreeBSD because of default system linker issues so we had to hide it under --with-lto switch which we recommend to everyone.

Re: GUCs to control abbreviated sort keys

2023-01-27 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 3:29 PM Jeff Davis wrote: > On Thu, 2023-01-26 at 22:39 +0100, Peter Eisentraut wrote: > > Maybe an easier way to enable or disable it in the source code with a > > #define would serve this. Making it a GUC right away seems a bit > > heavy-handed. Further exploration and

Re: Non-superuser subscription owners

2023-01-27 Thread Robert Haas
On Thu, Jan 19, 2023 at 8:46 PM Andres Freund wrote: > > If we already had (or have) that logic someplace else, it would > > probably make sense to reuse it > > We hve. See at least postgres_fdw's check_conn_params(), dblink's > dblink_connstr_check() and dblink_security_check(). In the patch I p

Re: Show various offset arrays for heap WAL records

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 12:24 PM Melanie Plageman wrote: > I believe I have addressed this in the attached patch. I'm not sure what's best in terms of formatting details but I definitely like the idea of making pg_waldump show more details. I'd even like to have a way to extract the tuple data, w

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
I wrote: >>> Anyway, it seems like I gotta work harder. I'll produce a >>> new patch. The string-hacking was fully as tedious as I expected. However, the output looks pretty nice, and this does have the advantage that the pre-programmed substitutions become a lot more robust: they are no longer

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 3:02 PM Tom Lane wrote: > The string-hacking was fully as tedious as I expected. However, the > output looks pretty nice, and this does have the advantage that the > pre-programmed substitutions become a lot more robust: they are no > longer dependent on the initdb code ex

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Tom Lane
=?UTF-8?Q?Jo=C3=A3o_Paulo_Labegalini_de_Carvalho?= writes: > I am facing issues when applying LTO to PostgreSQL as the produced binary > seems broken (the server dies quickly after it has started). This is > definitely a compiler bug, but I was wondering if anyone here have > experimented with L

Re: Authentication fails for md5 connections if ~/.postgresql/postgresql.{crt and key} exist

2023-01-27 Thread Cary Huang
> I think the sslcertmode=disable option that I introduced in [1] solves > this issue too; would it work for your case? That whole patchset is > meant to tackle the general case of the problem you've described. > > (Eventually I'd like to teach the server not to ask for a client >  certificate if

Re: GUCs to control abbreviated sort keys

2023-01-27 Thread Jeff Davis
On Fri, 2023-01-27 at 11:41 -0800, Peter Geoghegan wrote: > I cannot recreate the issue you describe. Interesting. For my test: glibc 2.35 ICU 70.1 gcc11.3.0LLVM 14.0.0 > It's not impossible that the perl program you wrote produces > non-deterministic output It is non-deterministi

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Tomas Vondra
On 1/27/23 08:18, Bharath Rupireddy wrote: > On Thu, Jan 26, 2023 at 9:21 PM Andres Freund wrote: >> >>> 7. I think we need to not let backends throttle too frequently even >>> though they have crossed wal_throttle_threshold bytes. The best way is >>> to rely on replication lag, after all the goal

Re: postgres_fdw, dblink, and CREATE SUBSCRIPTION security

2023-01-27 Thread Robert Haas
On Wed, Jan 25, 2023 at 6:22 PM Jacob Champion wrote: > Sure: Ambient authority [1] means that something is granted access based > on some aspect of its existence that it can't remove (or even > necessarily enumerate). Up above, when you said "I cannot choose not to > be myself," that's a clear ma

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 14:42:01 -0500, Robert Haas wrote: > At first, I found it a bit tempting to see this as a further > indication that the force-a-password approach is not the right idea, > because the test case clearly memorializes a desire *not* to require a > password in this situation. However,

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:06:49 +0100, Jakub Wartak wrote: > On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: > > > Huh? Why did you remove the GUC? > > After reading previous threads, my optimism level of getting it ever > in shape of being widely accepted degraded significantly (mainly due > t

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:48:43 +0530, Bharath Rupireddy wrote: > Looking at the patch, the feature, in its current shape, focuses on > improving replication lag (by throttling WAL on the primary) only when > synchronous replication is enabled. Why is that? Why can't we design > it for replication in g

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 21:45:16 +0100, Tomas Vondra wrote: > On 1/27/23 08:18, Bharath Rupireddy wrote: > >> I think my idea of only forcing to flush/wait an LSN some distance in the > >> past > >> would automatically achieve that? > > > > I'm sorry, I couldn't get your point, can you please explain

Re: Non-superuser subscription owners

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 4:09 PM Andres Freund wrote: > Hm, compared to postgres_fdw, the user has far less control over what's > happening using that connection. Is there a way a subscription owner can > trigger evaluation of near-arbitrary SQL on the publisher side? I'm not aware of one, but wha

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Melanie Plageman
Hi, I have written the patch to remove the unreachable code in heapgettup_pagemode](). On Wed, Jan 25, 2023 at 10:02 AM Tom Lane wrote: > > I wonder if we couldn't also get rid of this confusingly-inconsistent > alternative usage in the planner: > > * 'indexscandir' is one of: > *Forwa

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:35:11 -0500, Robert Haas wrote: > > Maybe a daft question: > > > > Have we considered using a "normal grant", e.g. on the database, instead of > > a > > role? Could it e.g. be useful to grant a user the permission to create a > > subscription in one database, but not in anot

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Tom Lane
Melanie Plageman writes: > I have written the patch to remove the unreachable code in > heapgettup_pagemode](). A few thoughts ... 1. Do we really need quite so many Asserts? I'd kind of lean to just having one, at some higher level of the executor. 2. I'm not sure if we want to do this: -

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Melanie Plageman
On Fri, Jan 27, 2023 at 05:05:16PM -0500, Tom Lane wrote: > Melanie Plageman writes: > > I have written the patch to remove the unreachable code in > > heapgettup_pagemode](). > > A few thoughts ... > > 1. Do we really need quite so many Asserts? I'd kind of lean > to just having one, at some h

Re: Non-superuser subscription owners

2023-01-27 Thread Mark Dilger
> On Jan 27, 2023, at 1:35 PM, Robert Haas wrote: > >> I started out asking what benefits it provides to own a subscription one >> cannot modify. But I think it is a good capability to have, to restrict the >> set of relations that replication could target. Although perhaps it'd be >> better

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 10:05:09 -0700, João Paulo Labegalini de Carvalho wrote: > I am investigating the benefits of different profile-guided optimizations > (PGO) and link-time optimizations (LTO) versus binary optimizers (e.g. > BOLT) for applications such as PostgreSQL. > > I am facing issues when

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 15:06:37 -0500, Tom Lane wrote: > There are a lot of places where we're implicitly relying on > cross-compilation-unit optimizations NOT happening, because the code isn't > adequately decorated with memory barriers and the like. We have a fallback compiler barrier implementation

Re: improving user.c error messages

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 07:31:19PM +0100, Alvaro Herrera wrote: > On 2023-Jan-26, Nathan Bossart wrote: >> ereport(ERROR, >> >> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), >> - errmsg("permission denied: bo

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Tom Lane
Melanie Plageman writes: > On Fri, Jan 27, 2023 at 05:05:16PM -0500, Tom Lane wrote: >> AFAIR, there is noplace today that depends on the exact encoding >> of ForwardScanDirection and BackwardScanDirection, and I'm not >> sure that we want to introduce such a dependency. > I think you mean the en

Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Corey Huinker
This patch adds the n_tup_newpage_upd to all the table stat views. Just as we currently track HOT updates, it should be beneficial to track updates where the new tuple cannot fit on the existing page and must go to a different one. Hopefully this can give users some insight as to whether their cu

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Tom Lane
Andres Freund writes: > On 2023-01-27 15:06:37 -0500, Tom Lane wrote: >> There are a lot of places where we're implicitly relying on >> cross-compilation-unit optimizations NOT happening, because the code isn't >> adequately decorated with memory barriers and the like. > We have a fallback compil

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 09:40:58PM -0800, Nathan Bossart wrote: > On Wed, Jan 25, 2023 at 04:34:21PM +0900, Michael Paquier wrote: >> The loop part is annoying.. I've never been a fan of adding this >> cross-value checks for the archiver modules in the first place, and it >> would make things much

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread David Rowley
On Sat, 28 Jan 2023 at 12:15, Tom Lane wrote: > /* > * Determine the net effect of two direction specifications. > * This relies on having ForwardScanDirection = +1, BackwardScanDirection = > -1, > * and will probably not do what you want if applied to any other values. > */ > #define Combine

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Tom Lane
David Rowley writes: > My personal preference would have been to call it > ScanDirectionCombine, so the naming is more aligned to the 4 other > macro names that start with ScanDirection in sdir.h, but I'm not going > to fuss over it. No objection to that. regards, tom lan

Re: recovery modules

2023-01-27 Thread Michael Paquier
On Thu, Jan 26, 2023 at 09:40:58PM -0800, Nathan Bossart wrote: > Yeah, there are some problems here. If we ERROR, we'll just bounce back to > the sigsetjmp() block once a second, and we'll never pick up configuration > reloads, shutdown signals, etc. If we FATAL, we'll just rapidly restart > ove

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 18:23:39 -0500, Corey Huinker wrote: > This patch adds the n_tup_newpage_upd to all the table stat views. > > Just as we currently track HOT updates, it should be beneficial to track > updates where the new tuple cannot fit on the existing page and must go to > a different one.

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 15:28:21 -0800, Nathan Bossart wrote: > The more I think about this, the more I wonder whether we really need to > include archive_cleanup_command and recovery_end_command in recovery > modules. I think it would be hard to write a good module that isn't just implementing the exi

Re: How to find the number of cached pages for a relation?

2023-01-27 Thread Amin
Thank you Andres. If I want to do "a" ( Do one probe of the buffer mapping table for each block of the relation. Thus O(#relation blocks)) what function calls can I use, assuming I only have access to the relation id? How can I access and scan the buffer mapping table? On Fri, Jan 13, 2023 at 6:2

  1   2   >