Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-04 Thread Alvaro Herrera
On 2021-Mar-04, Alvaro Herrera wrote: > I think the problem is that the project is called pipeline and not > test_libpq, so there's no match in the name. I'm going to rename the > whole thing to src/test/modules/libpq_pipeline/ and see if the msvc > tooling likes that bet

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-04 Thread Alvaro Herrera
On 2021-Mar-04, Alvaro Herrera wrote: > v31. Got this: libpq_pipeline.obj : error LNK2019: unresolved external symbol __WSAFDIsSet referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj] 5019libpq_pipeline.obj : error LNK2019: unresolved external sym

Re: CI/windows docker vs "am a service" autodetection on windows

2021-03-04 Thread Alvaro Herrera
On 2021-Mar-04, Andres Freund wrote: > > > There does seem to be isatty(), so we could improve the case of > > > pg_ctl/postgres run interactively without breaking a sweat. And there is > > > fstat() too, so if stderr in a service is something distinguishable... > > > > We seem to have used that

Re: Add .log file ending to tap test log files

2021-03-04 Thread Alvaro Herrera
On 2021-Mar-04, Andres Freund wrote: > Right now it's harder than necessary to capture the log output from tap > tests because the the regression tests files don't end with a common > file ending with other types of logs. They're > # Open the test log file, whose name depends on the test na

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-04 Thread Alvaro Herrera
On 2021-Mar-04, Alvaro Herrera wrote: > I don't know where do __WSAFDIsSet and __imp_select come from or what to > do about them. Let's see if adding pgport and pgcommon fixes things. Indeed all those other problems were fixed and these remain. New failure is: "C

Re: pgbench - add pseudo-random permutation function

2021-03-05 Thread Alvaro Herrera
Hi David On 2021-Mar-05, David Steele wrote: > On 4/2/20 3:01 AM, Alvaro Herrera wrote: > > On 2020-Apr-02, Fabien COELHO wrote: > > > > > > I'm planning to mark this patch RwF on April 8 and I suggest you > > > > resubmit if you are able to get some

Re: We should stop telling users to "vacuum that database in single-user mode"

2021-03-05 Thread Alvaro Herrera
On 2021-Mar-02, David Rowley wrote: > However, I wonder if it's worth going a few steps further to try and > reduce the chances of that message being seen in the first place. > Maybe it's worth considering ditching any (auto)vacuum cost limits for > any table which is within X transaction from wra

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-05 Thread Alvaro Herrera
On 2021-Mar-05, James Coleman wrote: > Do you have any thoughts on what you'd like to see the message be? I > could change the PM_RECOVERY (without hot standby enabled) to return > CAC_RECOVERY which would give us the message "the database system is > in recovery mode", but that would be a change

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-05 Thread Alvaro Herrera
v33 was indeed marked a pass by cfbot. However, it only *builds* the test program, it does not *run* it. I guess we'll have to wait for the buildfarm to tell us more. In the meantime, I implemented PQsendQuery() as callable in pipeline mode; it does that by using the extended-query protocol dire

Re: is cfbot's apply aging intentional?

2021-03-06 Thread Alvaro Herrera
On 2021-Mar-06, e...@xs4all.nl wrote: > Is that the way it's supposed to be? I would have thought there was a > regular schedule (hourly? 3-hourly? daily?) when all patches were taken for > re-apply, and re-build, so that when a patch stops applying/building/whatever > it can be seen on the cf

Re: [PATCH] pgbench: improve \sleep meta command

2021-03-08 Thread Alvaro Herrera
On 2021-Mar-08, kuroda.hay...@fujitsu.com wrote: > Dear Fujii-san, Miyake-san > > > Isn't it better to accept even negative sleep time like currently pgbench > > does? > > Otherwise we always need to check the variable is a positive integer > > (for example, using \if command) when using it as t

Re: PATCH: Batch/pipelining support for libpq

2021-03-08 Thread 'Alvaro Herrera'
On 2021-Mar-03, 'Alvaro Herrera' wrote: > I wonder if it would make sense to get rid of conn->last_query > completely and just rely always on conn->cmd_queue_head, where the > normal (non-pipeline) would use the first entry of the command queue. > That might end up

Re: Optimising latch signals

2021-03-08 Thread Alvaro Herrera
On 2021-Mar-03, Thomas Munro wrote: > On Mon, Mar 1, 2021 at 2:29 PM Thomas Munro wrote: > > Time to watch the buildfarm to find out if my speculation about > > illumos is correct... > > I just heard that damselfly's host has been decommissioned with no > immediate plan for a replacement. That

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-09 Thread Alvaro Herrera
On 2021-Mar-07, Magnus Hagander wrote: > On Sun, Mar 7, 2021 at 3:39 PM Fujii Masao > wrote: > > > > Here's an idea: > > > > > > * hot_standby=on, before reaching consistent state > > >FATAL: database is not accepting connections > > >DETAIL: Consistent state has not yet been reached.

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-09 Thread Alvaro Herrera
On 2021-Mar-09, James Coleman wrote: > On Tue, Mar 9, 2021 at 8:47 AM Alvaro Herrera wrote: > > > > On 2021-Mar-07, Magnus Hagander wrote: > > > > > On Sun, Mar 7, 2021 at 3:39 PM Fujii Masao > > > wrote: > > > > Great, so we're agreed on

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-09 Thread Alvaro Herrera
On 2021-Mar-09, James Coleman wrote: > Yes, I think they both agreed on the "DETAIL: Hot standby mode is > disabled." message, but that alternative meant not needing to add any > new signals and pm states, correct? Ah, I see! I was thinking that you still needed the state and signal in order to

Re: libpq debug log

2021-03-10 Thread Alvaro Herrera
On 2021-Mar-10, Tom Lane wrote: > After studying this further, I think we should apply the attached > patch to remove that responsibility from pqParseInput3's subroutines. > This will allow a single trace call near the bottom of pqParseInput3 > to handle all cases that that function processes. Wo

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-11 Thread Alvaro Herrera
On 2021-Mar-05, Alvaro Herrera wrote: > I'll take the weekend to think about the issue with conn->last_query and > conn->queryclass that I mentioned yesterday; other than that detail my > feeling is that this is committable, so I'll be looking at getting this > push

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-11 Thread Alvaro Herrera
Rebase to current sources, to appease CF bot; no other changes. -- Álvaro Herrera39°49'30"S 73°17'W diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..ea3ae56991 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sg

Re: pgbench - add pseudo-random permutation function

2021-03-12 Thread Alvaro Herrera
On 2021-Mar-13, Thomas Munro wrote: > That doesn't sound like a bad option to me, if it makes this much > simpler. The main modern system without it seems to be MSVC. The > Linux, BSD, Apple, illumos, AIX systems using Clang/GCC with > Intel/AMD/ARM/PowerPC CPUs have it, and the Windows systems

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-12 Thread Alvaro Herrera
On 2021-Mar-11, Tom Lane wrote: > I think the changes in pqParseInput3() are broken. You should have > kept the else-structure as-is and inserted the check for "not really > idle" inside the else-clause that reports an error. As it stands, > after successfully processing an asynchronously-receiv

Re: pgbench - add pseudo-random permutation function

2021-03-13 Thread Alvaro Herrera
Hello, On 2021-Mar-13, Fabien COELHO wrote: > This looks like a good compromise, which can even be a little better because > we still have 64 bits ints. Yeah, we rely on those being available elsewhere. > Attached a simplified patch which does that, removing 1/3 of the code. What > do you think

Re: pgbench - add pseudo-random permutation function

2021-03-14 Thread Alvaro Herrera
On 2021-Mar-14, Fabien COELHO wrote: > + /*- > + * Apply 4 rounds of bijective transformations using key updated > + * at each stage: > + * > + * (1) whiten: partial xors on overlapping power-of-2 subsets > + * for instance with v in 0 .. 14 (i.e. with size ==

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-15 Thread Alvaro Herrera
- but it seems quite impressive nonetheless. -- Álvaro Herrera39°49'30"S 73°17'W >From 5e4fdd5246d559caf0d75ad74001f09a48ec4c0e Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Mon, 15 Mar 2021 15:05:22 -0300 Subject: [PATCH v37 1/2] Implement pipe

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-15 Thread Alvaro Herrera
On 2021-Mar-15, Justin Pryzby wrote: > Are you going to update the assertion ? > > +#if 0 > >

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-15 Thread Alvaro Herrera
On 2021-Feb-26, Alvaro Herrera wrote: > Hmm, but if we take this approach, then we're still vulnerable to the > problem that somebody can do DETACH CONCURRENTLY and cancel the wait (or > crash the server), then mess up the state before doing DETACH FINALIZE: > when they cancel t

Re: libpq debug log

2021-03-16 Thread Alvaro Herrera
On 2021-Mar-15, iwata@fujitsu.com wrote: > I create protocol message reading function for each protocol message type. > (Ex. pqTraceOutputB() read Bind message) > This makes the nesting shallower and makes the code easier to read. I'm not sure I agree with this structural change. Yes, it is

Re: Patch to fix FK-related selectivity estimates with constants

2020-10-28 Thread Alvaro Herrera
On 2020-Oct-27, Tom Lane wrote: > I had two concerns about possible extension breakage from a back-patch: > > * Changing the set of fields in ForeignKeyOptInfo is an ABI break. > We could minimize the risk by adding the new fields at the end in > the back branches, but it still wouldn't be zero-r

Re: Patch to fix FK-related selectivity estimates with constants

2020-10-28 Thread Alvaro Herrera
On 2020-Oct-28, Tom Lane wrote: > Alvaro Herrera writes: > > I think if a struct is used as a function argument somewhere or arrays > > of the struct are formed, then it's certain that changing that struct's > > size is going to cause problems. > > I grasp

Re: libpq compression

2020-10-28 Thread Alvaro Herrera
On 2020-Oct-26, Konstantin Knizhnik wrote: > + while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data > to flush or unsent data in internal compression buffer */ > { > - int r; > - > - r = secure_write(MyProcPort, bufptr, bufe

Re: Autovacuum worker doesn't immediately exit on postmaster death

2020-10-29 Thread Alvaro Herrera
On 2020-Oct-29, Stephen Frost wrote: > > It's hard to do better than that, because on most platforms there's > > no way to get a signal on parent-process death, so the only way to > > notice would be to poll the postmaster-death pipe constantly; which > > would be hugely expensive in comparison to

Re: Autovacuum worker doesn't immediately exit on postmaster death

2020-10-29 Thread Alvaro Herrera
On 2020-Oct-28, Alexander Kukushkin wrote: > Hello, > > I know, nobody in their mind should do that, but, if the postmaster > process is killed with SIGKILL signal, most backend processes > correctly notice the fact of the postmaster process absence and exit. > There is one exception though, when

Re: Autovacuum worker doesn't immediately exit on postmaster death

2020-10-29 Thread Alvaro Herrera
On 2020-Oct-29, Stephen Frost wrote: > I do think it'd be good to find a way to check every once in a while > even when we aren't going to delay though. Not sure what the best > answer there is. Maybe instead of thinking specifically in terms of vacuum, we could count buffer accesses (read from

Re: PATCH: Batch/pipelining support for libpq

2020-11-02 Thread Alvaro Herrera
In v23 I've gone over docs; discovered that PQgetResults docs were missing the new values. Added those. No significant other changes yet.

Re: PATCH: Batch/pipelining support for libpq

2020-11-02 Thread Alvaro Herrera
On 2020-Nov-02, Alvaro Herrera wrote: > In v23 I've gone over docs; discovered that PQgetResults docs were > missing the new values. Added those. No significant other changes yet. >From 3f305d05000981445fe4dbbb96c2e88ace89f439 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Dat

Re: [proposal] de-TOAST'ing using a iterator

2020-11-02 Thread Alvaro Herrera
On 2020-Nov-02, Anastasia Lubennikova wrote: > Status update for a commitfest entry. > > This entry was inactive for a very long time. > John, are you going to continue working on this? > > The last message mentions some open issues, namely backend crashes, so I move > it to "Waiting on author

Re: PATCH: Batch/pipelining support for libpq

2020-11-03 Thread Alvaro Herrera
Hi Dave, On 2020-Nov-03, Dave Cramer wrote: > On Mon, 2 Nov 2020 at 10:57, Alvaro Herrera wrote: > > > On 2020-Nov-02, Alvaro Herrera wrote: > > > > > In v23 I've gone over docs; discovered that PQgetResults docs were > > > missing the new values. A

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-11-03 Thread Alvaro Herrera
ed by 6f0b632f083b. >From 8135f82493892ad755176c8c4c8251355dae2927 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Mon, 3 Aug 2020 19:21:09 -0400 Subject: [PATCH v4] ALTER TABLE ... DETACH CONCURRENTLY --- doc/src/sgml/catalogs.sgml| 10 + doc/src/sgml/ref/alter_table.sg

Re: redundant error messages

2020-11-05 Thread Alvaro Herrera
On 2020-Nov-05, Isaac Morland wrote: > In principle, the client knows the database name. In practice, if it's > coming from PGDATABASE or via a service configuration, one may be confused > about the database; having the error message be explicit will avoid many > problems. I can easily imagine tha

Re: Fix brin_form_tuple to properly detoast data

2020-11-05 Thread Alvaro Herrera
er if we need the #define at all. I propose to remove all those #ifdef lines in your patch. The fix looks good to me. I just added a comment in 0003. >From 101638065fb13e540f58f96ff839af982d7c5344 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Thu, 5 Nov 2020 14:12:38 -0300 Subject: [PATCH 1/3] Simplify test to use a t

Re: Fix brin_form_tuple to properly detoast data

2020-11-05 Thread Alvaro Herrera
On 2020-Nov-05, Tomas Vondra wrote: > On 11/5/20 6:17 PM, Alvaro Herrera wrote: > > There are recent changes in vacuum for temp tables (commit 94bc27b57680?) > > that would maybe make this stable enough, without having to have the CIC > > there. At least, I tried it local

Re: list_free() in index_get_partition()

2020-11-06 Thread Alvaro Herrera
On 2020-Nov-06, Michael Paquier wrote: > On Thu, Nov 05, 2020 at 02:36:06PM -0600, Justin Pryzby wrote: > > Seems to be missing. > > Any code paths calling index_get_partition() is in tablecmds.c, > involving commands that are not that hot with such lookups, but that > could be an issue if this g

Re: pg_dump, ATTACH, and independently restorable child partitions

2020-11-06 Thread Alvaro Herrera
On 2020-Oct-24, Justin Pryzby wrote: > On Fri, Oct 23, 2020 at 12:29:40AM -0500, Justin Pryzby wrote: > > Now that I look, it seems like this is calling PQexec(), which sends a > > single, > > "simple" libpq message with: > > |CREATE TABLE ..; ALTER TABLE .. ATTACH PARTITION; > > ..which is tran

PG13: message style changes

2020-11-06 Thread Alvaro Herrera
(Dropped 0003 while composing this email.) Patch 0004 just adds a comment to clarify a message that I found confusing when doing the translation. -- Álvaro Herrera39°49'30"S 73°17'W >From 3ea8f3ebd421e3c9b5633fffb7f38d999f21895f Mon Sep 17 00:00:00 2001 F

Re: PG13: message style changes

2020-11-07 Thread Alvaro Herrera
Thanks for looking! Pushed.

Re: Yet another (minor) fix in BRIN

2020-11-07 Thread Alvaro Herrera
On 2020-Nov-08, Tomas Vondra wrote: > While rebasing some of the BRIN patches, I noticed some of the code in > brin_memtuple_initialize is duplicated. This happened in 8bf74967dab > which moved some of the code from brin_new_memtuple, not removing the > shared pieces. In practice this is benign, o

Re: Reduce the number of special cases to build contrib modules on windows

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-06, David Rowley wrote: > +# Handle makefile rules for when file to be added to the project > +# does not exist. Returns 1 when the original file add should be > +# skipped. > +sub FindAndAddAdditionalFiles > +{ > + my $self = shift; > + my $fname = shift; > + my ($ext) =

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Magnus Hagander wrote: > But for usability that makes less sense. For the delete script, the > wrapper (that the switch is intended for) knows more than pg_upgrade > about how to delete it, so it can do a better job, and thus it makes > sense to silence it. But for something like t

Re: Rethinking LOCK TABLE's behavior on views

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-07, Noah Misch wrote: > On Sat, Nov 07, 2020 at 11:57:20AM -0500, Tom Lane wrote: > > A completely different approach we could consider is to weaken the > > permissions requirements for LOCK on a view, say "allow it if either > > the calling user or the view owner has the needed permi

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Magnus Hagander wrote: > On Mon, Nov 9, 2020 at 3:29 PM Alvaro Herrera wrote: > > > > How about a switch like "--with-scripts=" where the list can be > > "all" to include everything (default), "none" to include nothing, or a &g

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Alvaro Herrera
On 2020-Oct-27, Craig Ringer wrote: > On Tue, Oct 27, 2020 at 12:56 AM Tom Lane wrote: > > > +1. Are we concerned about translatability of these strings? I think > > I'd vote against, as it would complicate applications, but it's worth > > thinking about it now not later. > > It's necessary no

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-10, David Rowley wrote: > On Mon, 9 Nov 2020 at 16:29, Andy Fan wrote: > > However I believe v9 > > should be no worse than v8 all the time, Is there any theory to explain > > your result? > > Nothing jumps out at me from looking at profiles. The only thing I > noticed was the t

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: > >> Do we really need exclusive lock on the ProcArray to make this flag > >> change? That seems pretty bad from a concurrency standpoint. > > > Any place where we update vacu

Re: proposal: possibility to read dumped table's name from file

2020-11-11 Thread Alvaro Herrera
On 2020-Nov-11, Pavel Stehule wrote: > I think the proposed feature is very far to be the config file for pg_dump > (it implements a option "--filter"). This is not the target. It is not > designed for this. This is just an alternative for options like -t, -T, ... > and I am sure so nobody will ge

Re: PATCH: Batch/pipelining support for libpq

2020-11-12 Thread Alvaro Herrera
(Adding previous reviewers to CC) On 2020-Nov-03, David G. Johnston wrote: > Given the caveats around blocking mode connections why not just require > non-blocking mode, in a similar fashion to how synchronous functions are > disallowed? This is a very good question. Why indeed? Does anybody h

Re: Proposition for autoname columns

2020-11-12 Thread Alvaro Herrera
On 2020-Nov-12, Tom Lane wrote: > On the whole, I'm on the side of the people who don't want to change this. > The implementation cost seems likely to greatly outweigh the value, plus > it feels more like a wart than a feature. I think if Eugen wants to spend some time with it and see how it coul

Re: error_severity of brin work item

2020-11-13 Thread Alvaro Herrera
On 2020-Nov-13, Justin Pryzby wrote: > I saw a bunch of these in my logs: > > log_time | 2020-10-25 22:59:45.619-07 > database | > left | could not open relation with OID 292103095 > left | processing work entry for relation > "ts.child.alarms_202010_alarm_clear_time_idx" > > Those hap

Re: PATCH: Batch/pipelining support for libpq

2020-11-13 Thread Alvaro Herrera
Here's a v25. I made a few more changes to the docs per David's suggestions; I also reordered the sections quite a bit. It's now like this: * Batch Mode * Using Batch Mode * Issuing Queries * Processing Results * Error Handling * Interleaving Result Processing and Query Di

Re: doc CREATE INDEX

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-16, Erik Rijkers wrote: > This one seems to have fallen by the wayside. So it did. Pushed to all three branches, thanks.

Re: Add important info about ANALYZE after create Functional Index

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-12, Bruce Momjian wrote: > For new expression indexes, it is necessary to run linkend="sql-analyze">ANALYZE or wait for > the autovacuum daemon to analyze > - the table to generate statistics about new expression indexes. > + the table to generate statistics for these

Re: Add important info about ANALYZE after create Functional Index

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-16, Justin Pryzby wrote: > I see Alvaro already patched the first issue at bcbd77133. > > The problematic language was recently introduced, and I'd reported at: > https://www.postgresql.org/message-id/20201112211143.GL30691%40telsasoft.com > And Erik at: > https://www.postgresql.org/m

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-16, Dmitry Dolgov wrote: > The same with reindex without locks: > > nsecs : count distribution >512 -> 1023 : 0|| > 1024 -> 2047 : 111345 || >

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-16 Thread Alvaro Herrera
On 2020-Nov-09, Tom Lane wrote: > Michael Paquier writes: > >> + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); > >> + MyProc->vacuumFlags |= PROC_IN_SAFE_IC; > >> + ProcGlobal->vacuumFlags[MyProc->pgxactoff] = > >> MyProc->vacuumFlags; > >> + LWLockRelease(ProcA

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-16 Thread Alvaro Herrera
it would be sane/safe to do the WaitForFoo stuff outside of any transaction. I'll have a look again tomorrow. >From 1e5560d4a3f3e3b4cb83803f90985701fb4dee8c Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Tue, 4 Aug 2020 22:04:57 -0400 Subject: [PATCH v5] Avoid spurious CREATE IND

Re: pg_proc.dat "proargmodes is not a 1-D char array"

2020-11-17 Thread Alvaro Herrera
On 2020-Sep-30, Craig Ringer wrote: > Hi all > > Random tip for future searchers. If you've modified pg_proc.dat and initdb > fails with "proargmodes is not a 1-D char array" - it could well actually > be that the length of proargmodes does not match the length of > proallargtypes given the test

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-17 Thread Alvaro Herrera
On 2020-Nov-16, Alvaro Herrera wrote: > On 2020-Nov-09, Tom Lane wrote: > > > Michael Paquier writes: > > >> +LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); > > >> +MyProc->vacuumFlags |= PROC_IN_SAFE_IC; > > >&

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-17 Thread Alvaro Herrera
ter. >From 5e2af4e082df8f839bf257d933d70bb9645f6cfe Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Tue, 17 Nov 2020 11:37:36 -0300 Subject: [PATCH v6 1/4] indexcmds.c: reorder function prototypes ... out of an overabundance of neatnikism, perhaps. --- src/backend/commands/indexcmds.c | 7 +++ 1 file changed,

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-17 Thread Alvaro Herrera
On 2020-Nov-17, Simon Riggs wrote: > As an additional optimization, if we do find a row that needs freezing > on a data block, we should simply freeze *all* row versions on the > page, not just the ones below the selected cutoff. This is justified > since writing the block is the biggest cost and

Re: Devel docs on website reloading

2020-11-18 Thread Alvaro Herrera
On 2020-Nov-18, Magnus Hagander wrote: > It would be trivial to change this so that it only actually updates > pages if they have been changed. I think this means we could also check much more frequently whether a rebuild is needed, right? We could do that every 30 mins or so, since most of the

"as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-18 Thread Alvaro Herrera
> On 2020-11-17 12:55:01 -0300, Alvaro Herrera wrote: > > ... ah, but I realize now that this means that we can use shared lock > > here, not exclusive, which is already an enormous improvement. That's > > because ->pgxactoff can only be changed with exclusive lock he

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-18 Thread Alvaro Herrera
On 2020-Nov-18, Michael Paquier wrote: > On Tue, Nov 17, 2020 at 02:14:53PM -0300, Alvaro Herrera wrote: > > diff --git a/src/backend/replication/logical/logical.c > > b/src/backend/replication/logical/logical.c > > index f1f4df7d70..4324e32656 100644 > > --- a/src/

Re: PATCH: Batch/pipelining support for libpq

2020-11-18 Thread Alvaro Herrera
On 2020-Nov-14, Daniel Verite wrote: > The patch I posted in [1] was pretty simple, but at the time, query > results were always discarded. Now that pgbench can instantiate > variables from query results, a script can do: > select 1 as var \gset > select :var; > This kind of sequence wouldn't

Re: error_severity of brin work item

2020-11-19 Thread Alvaro Herrera
On 2020-Nov-19, Justin Pryzby wrote: > On Fri, Nov 13, 2020 at 12:11:21PM -0600, Justin Pryzby wrote: > > Your patch didn't actually say "try_relation_open", so didn't work. > > But it does works if I do that, and close the table. Thanks for fixing and testing. > That patch broke the case that

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-20 Thread Alvaro Herrera
On 2020-Nov-20, Masahiko Sawada wrote: > I'm concerned that always freezing all tuples when we're going to make > the page dirty would affect the existing vacuum workload much. The > additional cost of freezing multiple tuples would be low but if we > freeze tuples we would also need to write WAL,

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-20 Thread Alvaro Herrera
Note on heap_prepare_freeze_tuple()'s fifth parameter, it's not valid to pass OldestXmin; you need a multixact limit there, not an Xid limit. I think the return value of GetOldestMultiXactId is a good choice. AFAICS this means that you'll need to add a new output argument to vacuum_set_xid_limits

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-20 Thread Alvaro Herrera
On 2020-Nov-20, Simon Riggs wrote: > On Fri, 20 Nov 2020 at 01:40, Masahiko Sawada wrote: > > Since we use the term aggressive scan in the docs, I personally don't > > feel unnatural about that. But since this option also disables index > > cleanup when not enabled explicitly, I’m concerned a bi

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-20 Thread Alvaro Herrera
On 2020-Nov-20, Robert Haas wrote: > Yeah, I think dirtying the page fewer times is a big win. However, a > page may have tuples that are not yet all-visible, and we can't freeze > those just because we are freezing others. Of course! We should only freeze tuples that are freezable. I thought t

bug in pageinspect's "tuple data" feature

2020-11-21 Thread Alvaro Herrera
If you have a sufficiently broken data page, pageinspect throws an error when trying to examine the page: ERROR: invalid memory alloc request size 18446744073709551451 This is pretty unhelpful; it would be better not to try to print the data instead of dying. With that, at least you can know wh

Re: pg_proc.dat "proargmodes is not a 1-D char array"

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-17, Tom Lane wrote: > Robert Haas writes: > > On Tue, Nov 17, 2020 at 10:32 AM Tom Lane wrote: > >> Adding the expected length to the error message might be OK though. > > > Certainly seems like we should do at least that much. The current > > message is just wrong, right? > > It's

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-18, Andres Freund wrote: > In 13 this is: > LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); > MyPgXact->vacuumFlags |= PROC_IN_VACUUM; > if (params->is_wraparound) > MyPgXact->vacuumFlags |= PROC_VACUUM_FOR_WRAPAROUND; >

Re: pg_proc.dat "proargmodes is not a 1-D char array"

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Alvaro Herrera writes: > > So let's go with this one. > > WFM. Thanks, pushed.

Re: error_severity of brin work item

2020-11-23 Thread Alvaro Herrera
apply instead of your proposed fix. >From e0b457866e503a1b70b94c540147026b652fb019 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Fri, 13 Nov 2020 13:39:31 -0300 Subject: [PATCH v3] Avoid errors in brin summarization.. ..which can happen if an index is reindexed concurrently --- src/back

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
the final form of this. >From cab95d8813cd79adb7b2a1b5501714c2dd87bec2 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Tue, 17 Nov 2020 21:09:22 -0300 Subject: [PATCH] Don't hold ProcArrayLock longer than needed in rare cases While cancelling an autovacuum worker, we hold ProcArrayLock while formatting a debugging log

walsender bug: stuck during shutdown

2020-11-23 Thread Alvaro Herrera
or the problematic condition. Apparently this problem has existed forever. Fujii-san almost patched for it in 5c6d9fc4b2b8 (2014!), but missed it by a zillionth of an inch. -- Álvaro Herrera >From aca27a0af5616bc1da4f08cbbc93b4d3c9380f60 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Mon, 23 Nov 2

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Alvaro Herrera writes: > > On 2020-Nov-19, Michael Paquier wrote: > >> By the way, it strikes me that you could just do nothing as long as > >> (log_min_messages > DEBUG1), so you could encapsulate most of the > >> logic

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Alvaro Herrera writes: > > Well, we already do this in a number of places. But I can get behind > > this: > > >> Maybe it'd be a good idea to have elog.c expose a new function > >> along the lines of "bool mes

Re: PATCH: Batch/pipelining support for libpq

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Daniel Verite wrote: > Hi, > > Here's a new version with the pgbench support included. Thanks, incorporated into my local copy.

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Here's a draft patch. Here's another of my own. Outside of elog.c it seems identical. diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 03c553e7ea..a4ab9090f9 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backen

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Alvaro Herrera wrote: > On 2020-Nov-23, Tom Lane wrote: > > > Here's a draft patch. > > Here's another of my own. Outside of elog.c it seems identical. Your version has the advantage that errstart() doesn't get a new function call. I'

Re: optimizer/clauses.h needn't include access/htup.h

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Anyway, if you're feeling motivated to explore a more wide-ranging > refactoring, by all means have a go at it. I was contemplating commands/trigger.c this morning (after Heikki split copy.c) thinking about the three pieces embedded in there -- one catalog/pg_tri

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Ah, I see I didn't cover the case in ProcSleep that you were originally on > about ... I'd just looked for existing references to log_min_messages > and client_min_messages. Yeah, it seemed bad form to add that when you had just argued against it :-) > I think i

Re: "as quickly as possible" (was: remove spurious CREATE INDEX CONCURRENTLY wait)

2020-11-23 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > I never cared that much for "is_log_level_output" either. Thinking > about renaming it to "should_output_to_log()", and then the new function > would be "should_output_to_client()". Ah, that sounds nicely symmetric and grammatical. Thanks!

Re: LogwrtResult contended spinlock

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-24, Anastasia Lubennikova wrote: > On 04.09.2020 20:13, Andres Freund wrote: > > Re general routine: On second thought, it might actually be worth having > > it. Even just for LSNs - there's plenty places where it's useful to > > ensure a variable is at least a certain size. I think

Re: walsender bug: stuck during shutdown

2020-11-24 Thread Alvaro Herrera
Hello, On 2020-Nov-24, Fujii Masao wrote: > Thanks for working on this! > Could you tell me the discussion thread where Chloe Dives reported the issue > to? > Sorry I could not find that.. It was not public -- sorry I didn't make that clear. > I'd like to see the procedure to reproduce the iss

Re: About adding a new filed to a struct in primnodes.h

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-24, Andy Fan wrote: > then we modified the copy/read/out functions for this node. In > _readFuncExpr, > we probably add something like > [ ... ] > Then we will get a compatible issue if we create a view with the node in > the older version and access the view with the new binary. W

Re: bug in pageinspect's "tuple data" feature

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-24, Michael Paquier wrote: > On Mon, Nov 23, 2020 at 09:11:26AM +0200, Heikki Linnakangas wrote: > > On 21/11/2020 21:32, Alvaro Herrera wrote: > >> This is pretty unhelpful; it would be better not to try to print the > >> data instead of dying. With that,

Re: [HACKERS] Custom compression methods

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-24, Tom Lane wrote: > Robert Haas writes: > > Oh, I thought it had been suggested in previous discussions that these > > should be treated as access methods rather than inventing a whole new > > concept just for this, and it seemed like a good idea to me. I guess I > > missed the fac

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-24, Tom Lane wrote: > David Rowley writes: > > On Wed, 25 Nov 2020 at 04:55, Tom Lane wrote: > >> walleye's been failing since this patchset went in: > >> I have no idea what to make of that, but it looks more like a compiler bug > >> than anything else. > > > I wondered if #if !def

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-23, Andres Freund wrote: > On 2020-11-23 12:30:05 -0300, Alvaro Herrera wrote: > > PROC_IN_LOGICAL_DECODING: > > Oddly enough, I think the reset of PROC_IN_LOGICAL_DECODING in > > ReplicationSlotRelease might be the most problematic one of the lot. > > T

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-24 Thread Alvaro Herrera
On 2020-Nov-23, Tom Lane wrote: > Alvaro Herrera writes: > > GetCurrentVirtualXIDs, ComputeXidHorizons, GetSnapshotData: > > > In these cases, what we need is that the code computes some xmin (or > > equivalent computation) based on a set of transactions that exclude &

<    1   2   3   4   5   6   7   8   9   10   >