Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-01 Thread Amit Kapila
On Fri, Apr 28, 2023 at 2:24 PM Drouvot, Bertrand wrote: > > > Can you > > please explain the logic behind this test a bit more like how the WAL > > file switch helps you to achieve the purpose? > > > > The idea was to generate enough "wal switch" on the primary to ensure > the WAL file has been r

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-05-01 Thread Yurii Rashkovskii
On Mon, May 1, 2023 at 11:05 PM Eric Ridge wrote: > > > We expect the .so's own minor version number to suffice for that, > > but I realize that that might not be the most user-friendly answer. > > One of my desires is that the on-disk .so's filename be associated with > the pg_extension entry an

Re: ssl tests aren't concurrency safe due to get_free_port()

2023-05-01 Thread Peter Eisentraut
On 25.04.23 12:27, Peter Eisentraut wrote: These patches have affected pgxs-using extensions that have their own TAP tests. The portlock directory is created at     my $build_dir = $ENV{top_builddir}   || $PostgreSQL::Test::Utils::tmp_check ;     $portdir ||= "$build_dir/portlock"; but

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 8:04 PM John Naylor wrote: > Oh that's rich. I'll note that 5% of your review was actually helpful (actual > correction), the other 95% was needless distraction trying to enlist me in > your holy crusade against the term "wraparound". It had the opposite effect. I went ba

Re: Perform streaming logical transactions by background workers and parallel apply

2023-05-01 Thread Amit Kapila
On Tue, May 2, 2023 at 9:06 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, April 28, 2023 2:18 PM Masahiko Sawada > wrote: > > > > > > > > Alexander, does the proposed patch fix the problem you are facing? > > > Sawada-San, and others, do you see any better way to fix it than what > > > has been

RE: Perform streaming logical transactions by background workers and parallel apply

2023-05-01 Thread Zhijie Hou (Fujitsu)
On Friday, April 28, 2023 2:18 PM Masahiko Sawada wrote: > > On Fri, Apr 28, 2023 at 11:51 AM Amit Kapila wrote: > > > > On Wed, Apr 26, 2023 at 4:11 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Wednesday, April 26, 2023 5:00 PM Alexander Lakhin > wrote: > > > > > > > > IIUC, that assert

Re: Large files for relations

2023-05-01 Thread Pavel Stehule
Hi I like this patch - it can save some system sources - I am not sure how much, because bigger tables usually use partitioning usually. Important note - this feature breaks sharing files on the backup side - so before disabling 1GB sized files, this issue should be solved. Regards Pavel

Re: Fix typos and inconsistencies for v16

2023-05-01 Thread Michael Paquier
On Fri, Apr 21, 2023 at 12:00:00PM +0300, Alexander Lakhin wrote: > Please look at the following two bunches for v14+ and v13+ (split to ease > back-patching if needed). Having processed them, I've reached the state that > could be considered "clean" ([2], [3]); at least I don't see how to detect >

Re: Perform streaming logical transactions by background workers and parallel apply

2023-05-01 Thread Amit Kapila
On Fri, Apr 28, 2023 at 11:48 AM Masahiko Sawada wrote: > > While investigating this issue, I've reviewed the code around > callbacks and worker termination etc and I found a problem. > > A parallel apply worker calls the before_shmem_exit callbacks in the > following order: > > 1. ShutdownPostgre

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 8:04 PM John Naylor wrote: > Here's some advice on how to avoid pushback: > > 1. Insist that all terms can only be interpreted in the most pig-headedly > literal sense possible. > 2. Use that premise to pretend basic facts are a complete mystery. I can't imagine why you fe

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread John Naylor
On Tue, May 2, 2023 at 12:09 AM Peter Geoghegan wrote: > > On Mon, May 1, 2023 at 9:16 AM Peter Geoghegan wrote: > > On Mon, May 1, 2023 at 9:08 AM Robert Haas wrote: > > > I disagree. If you start the cluster in single-user mode, you can > > > actually wrap it around, unless something has chang

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 7:55 PM Peter Geoghegan wrote: > Obviously there are certain things that can hold back OldestMXact by a > wildly excessive amount. But I don't think that there is anything that > can hold back OldestMXact by a wildly excessive amount that won't more > or less do the same thi

Re: Support logical replication of DDLs

2023-05-01 Thread shveta malik
On Fri, Apr 28, 2023 at 5:11 PM Amit Kapila wrote: > > On Tue, Apr 25, 2023 at 9:28 AM Zhijie Hou (Fujitsu) > wrote: > > > > I have a few high-level comments on the deparsing approach used in the > patch. As per my understanding, we first build an ObjTree from the DDL > command, then convert the

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 5:34 AM John Naylor wrote: > Well, since you have a placeholder "xidStopLimit mode" in your other patch, > I'll confine my response to there. Inventing "modes" seems like an awkward > thing to backpatch, not to mention moving the goalposts. My modest goal here > is quite

Re: Autogenerate some wait events code and documentation

2023-05-01 Thread Thomas Munro
> [patch] This is not a review of the perl/make/meson glue/details, but I just wanted to say thanks for working on this Bertrand & Michael, at a quick glance that .txt file looks like it's going to be a lot more fun to maintain!

Re: Improve logging when using Huge Pages

2023-05-01 Thread Michael Paquier
On Thu, Apr 20, 2023 at 02:16:17PM -0700, Nathan Bossart wrote: > AFAICT this would involve adding a bool to BackendParameters and using it > in save_backend_variables() and restore_backend_variables(), which is an > additional 3 lines of code. That doesn't sound too bad to me, but perhaps > I am

Re: Tab completion for CREATE SCHEMAAUTHORIZATION

2023-05-01 Thread Michael Paquier
On Sat, Apr 15, 2023 at 11:06:25AM +0900, Michael Paquier wrote: > Thanks, I'll look at it. + else if (Matches("CREATE", "SCHEMA", "AUTHORIZATION", MatchAny) || +Matches("CREATE", "SCHEMA", MatchAny, "AUTHORIZATION", MatchAny)) + COMPLETE_WITH("CREATE", "GRANT"); + else if (M

Large files for relations

2023-05-01 Thread Thomas Munro
Big PostgreSQL databases use and regularly open/close huge numbers of file descriptors and directory entries for various anachronistic reasons, one of which is the 1GB RELSEG_SIZE thing. The segment management code is trickier that you might think and also still harbours known bugs. A nearby anal

Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options

2023-05-01 Thread Tatsuo Ishii
> The attached test patch is mostly the same as in the previous patch > set, but it doesn't fail on row_number anymore as the main patch > only rejects aggregate functions. The test patch also adds a test for > +SELECT sum(orbit) RESPECT NULLS OVER () FROM planets; -- succeeds I think the standar

Re: Add PQsendSyncMessage() to libpq

2023-05-01 Thread Michael Paquier
On Sat, Apr 29, 2023 at 05:06:03PM +0100, Anton Kirilov wrote: > In any case I am not particularly attached to any naming or the exact shape > of the new API, as long as it achieves the same goal (reducing the number of > system calls), but before I make any substantial changes to my patch, I > wou

Re: Add PQsendSyncMessage() to libpq

2023-05-01 Thread Michael Paquier
On Sun, Apr 30, 2023 at 01:59:17AM +0100, Anton Kirilov wrote: > I did a quick check using the TechEmpower Framework Benchmarks ( > https://www.techempower.com/benchmarks/ ) - they define 4 Web application > tests that are database-bound. Everything was running on a single machine, > and 3 of the t

RE: [PATCH] Support % wildcard in extension upgrade filenames

2023-05-01 Thread Regina Obe
> It isn't. ZDB, and I think (at least) PostGIS, have their own "version()" function. > Keeping everything the same version keeps me "sane" and eliminates a class > of round-trip questions with users. > Yes we have several version numbers and yes we too like to keep the extension version the same

Order problem in GiST index

2023-05-01 Thread Kotroczó Roland
Hi! We are developing a GiST index structure. We use M-tree for this work. We have a problem with it. We would like to get some closest neighbors. In my select we order the query by distance, and we use limit. Sometimes it works well, but in some cases I don’t get back the closest points. If we

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-05-01 Thread Eric Ridge
> On May 1, 2023, at 4:24 PM, Tom Lane wrote: > > Eric Ridge writes: >> FWIW, outside of major ZDB releases, most of those have little-to-zero >> schema changes. But that doesn't negate the fact each release needs its own >> upgrade.sql script. I'm working on a point release right this mome

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-05-01 Thread Tom Lane
Eric Ridge writes: > FWIW, outside of major ZDB releases, most of those have little-to-zero schema > changes. But that doesn't negate the fact each release needs its own > upgrade.sql script. I'm working on a point release right this moment and it > won't have any schema changes but it'll have

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 12:03 PM Maciek Sakrejda wrote: > I don't want to put words into Peter's mouth, but I think that he's arguing > that the term "wraparound" suggests that there is something special about the > transition between xid 2^32 and xid 0 (or, well, 3). There isn't. Yes, that's ex

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Maciek Sakrejda
On Mon, May 1, 2023, 18:08 Robert Haas wrote: > I am saying that, while wraparound is perhaps not a perfect term > for what's happening, it is not, in my opinion, a bad term either. I don't want to put words into Peter's mouth, but I think that he's arguing that the term "wraparound" suggests t

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-05-01 Thread Eric Ridge
> On May 1, 2023, at 12:12 PM, Robert Haas wrote: > > On Fri, Apr 28, 2023 at 10:03 AM Eric Ridge wrote: >> ZomboDB has 137 releases over the past 8 years. > > Dang. > > This may be one of those cases where the slow pace of change for > extensions shipped with PostgreSQL gives those of us wh

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 9:16 AM Peter Geoghegan wrote: > On Mon, May 1, 2023 at 9:08 AM Robert Haas wrote: > > I disagree. If you start the cluster in single-user mode, you can > > actually wrap it around, unless something has changed that I don't > > know about. > > This patch relies on John's ot

Re: base backup vs. concurrent truncation

2023-05-01 Thread Aleksander Alekseev
Hi, > I think that to reproduce the scenario, you want the truncate to happen in > its own checkpoint cycle. OK, let's try this again. In order to effectively disable the checkpointer I added the following lines to postgresql.conf: ``` checkpoint_timeout = 3600 max_wal_size = 100G ``` I'm also

Re: Logging parallel worker draught

2023-05-01 Thread Robert Haas
On Sat, Apr 22, 2023 at 7:06 AM Amit Kapila wrote: > I don't think introducing a GUC for this is a good idea. We can > directly output this message in the server log either at LOG or DEBUG1 > level. Why not? It seems like something some people might want to log and others not. Running the whole s

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 9:08 AM Robert Haas wrote: > I disagree. If you start the cluster in single-user mode, you can > actually wrap it around, unless something has changed that I don't > know about. This patch relies on John's other patch which strongly discourages the use of single-user mode.

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-05-01 Thread Robert Haas
On Fri, Apr 28, 2023 at 10:03 AM Eric Ridge wrote: > ZomboDB has 137 releases over the past 8 years. Dang. This may be one of those cases where the slow pace of change for extensions shipped with PostgreSQL gives those of us who are developers for PostgreSQL itself a sort of tunnel vision. The s

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Robert Haas
On Mon, May 1, 2023 at 12:01 PM Peter Geoghegan wrote: > > If the XID space were non-circular, it could > > never try to reuse the XID values that have previously been used, and > > this entire class of problems would go away. Because it is circular, > > it's possible for the XID counter to arrive

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Peter Geoghegan
On Mon, May 1, 2023 at 8:03 AM Robert Haas wrote: > To me, the fact that the XID space is circular is the whole point of > talking about wraparound. The word wraparound is ambiguous. It's not the same thing as xidStopLimit in my view. It's literal integer wraparound. If you think of XIDs as havi

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-01 Thread Robert Haas
On Wed, Apr 26, 2023 at 1:58 PM Peter Geoghegan wrote: > Why do we call wraparound wraparound, anyway? The 32-bit XID space is > circular! The whole point of the design is that unsigned integer > wraparound is meaningless -- there isn't really a point in "the > circle" that you should think of as

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-05-01 Thread John Naylor
On Mon, May 1, 2023 at 2:30 AM Peter Geoghegan wrote: > > On Sat, Apr 29, 2023 at 7:30 PM John Naylor > wrote: > > How about > > > > -HINT: To avoid a database shutdown, [...] > > +HINT: To prevent XID exhaustion, [...] > > > > ...and "MXID", respectively? We could explain in the docs that vacu

Re: Add RESPECT/IGNORE NULLS and FROM FIRST/LAST options

2023-05-01 Thread Oliver Ford
On Sun, Apr 23, 2023 at 4:29 AM Tatsuo Ishii wrote: > > Vik Fearing writes: > > > >> For me, this is perfectly okay. Keep them at the lowest level of > >> reservation as possible. > > > > Yeah, keep them unreserved if at all possible. Any higher reservation > > level risks breaking existing ap