Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-06 Thread Masahiko Sawada
Hi all, A customer reported that during parallel index vacuum, the oldest xmin doesn't advance. Normally, the calculation of oldest xmin (ComputeXidHorizons()) ignores xmin/xid of processes having PROC_IN_VACUUM flag in MyProc->statusFlags. But since parallel vacuum workers don’t set their statusF

Re: Fix pg_log_backend_memory_contexts() 's delay

2021-10-06 Thread bt21tanigaway
Thanks for your review. Thanks for the patch. Do we also need to do the change in HandleMainLoopInterrupts, HandleCheckpointerInterrupts, HandlePgArchInterrupts, HandleWalWriterInterrupts as we don't call CHECK_FOR_INTERRUPTS() there? Yeah, that's still some information that the user asked fo

Re: More business with $Test::Builder::Level in the TAP tests

2021-10-06 Thread Peter Eisentraut
On 06.10.21 08:28, Michael Paquier wrote: Following up with Peter E's recent commit 73aa5e0 to add some forgotten level incrementations, I got to look again at what I did wrong and why this stuff is useful. I have gone through all the TAP tests and any code paths using subroutines, to note that

Re: RfC entries in CF 2021-09

2021-10-06 Thread Etsuro Fujita
On Tue, Oct 5, 2021 at 11:28 PM Jaime Casanova wrote: > On Tue, Oct 05, 2021 at 03:24:40PM +0900, Etsuro Fujita wrote: > > On Tue, Oct 5, 2021 at 4:09 AM Jaime Casanova > > wrote: > > > - Fast COPY FROM command for the foreign tables > > > Last patch was on Jun-2021, no further activity after t

Re: a comment in joinrel.c: compute_partition_bounds()

2021-10-06 Thread Etsuro Fujita
On Fri, Sep 24, 2021 at 4:20 PM Etsuro Fujita wrote: > On Fri, Sep 24, 2021 at 3:34 PM Amit Langote wrote: > > I think there's a word missing in the following comment: > > > > /* > > * See if the partition bounds for inputs are exactly the same, in > > * which case we do

Add -java RPM generation to GDAL in pgrpms - p1

2021-10-06 Thread Guillaume FOREAU
This is an early patch to use as a discussion support, only tested on CentOS 8 on master with GDAL 3.2, to add the generation of gdal*-java RPMs.This is a follow-up to the accepted PR sent to fedora to re-enable gdal-java RPMs generation for EPEL 8 (https://src.fedoraproject.org/rpms/gdal/pull-re

Re: wrapping CF 2021-09

2021-10-06 Thread Daniel Gustafsson
> On 6 Oct 2021, at 08:41, Peter Geoghegan wrote: > > On Tue, Oct 5, 2021 at 10:21 PM Jaime Casanova > wrote: >> At this time I'm looking to close the CF. > > Thanks, Jaime. Indeed, thanks for all your work this CF! -- Daniel Gustafsson https://vmware.com/

postgres_fdw: Obsolete comments in GetConnection()

2021-10-06 Thread Etsuro Fujita
While working on something else, I noticed $SUBJECT, which I should have updated in commit 27e1f1456. :-( There are two places that need to be updated, but in the first place the second one seemed a bit redundant to me, because it says the same thing as the first one, and is placed pretty close t

Capitalization of localized month and day names (to_char() with 'TMmonth', 'TMday', etc.)

2021-10-06 Thread Magnus Holmgren
Hello and thank you very much for the best open source database engine! There's just this tiny but seemingly obvious issue that I can't believe I haven't noticed until now: to_date(now(), 'TMmonth') returns 'october' in an English locale (en_US.UTF-8 at least). Names of months and weekdays are p

Re: postgres_fdw: Obsolete comments in GetConnection()

2021-10-06 Thread Bharath Rupireddy
On Wed, Oct 6, 2021 at 2:35 PM Etsuro Fujita wrote: > > While working on something else, I noticed $SUBJECT, which I should > have updated in commit 27e1f1456. :-( There are two places that need > to be updated, but in the first place the second one seemed a bit > redundant to me, because it say

sentPtr jumping back at the beginning of logical replication

2021-10-06 Thread Ashutosh Bapat
Hi All, sentPtr reported by WAL sender should usually never jump back, it should always increase. I observed a strange behaviour with the WAL sender where sentPtr jumps back at the beginning. From code examination it looks like the following behaviour is culprit. The WAL sender reads WAL from rest

Re: can we add subscription TAP test option "vcregress subscriptioncheck" for MSVC builds?

2021-10-06 Thread Bharath Rupireddy
On Wed, Oct 6, 2021 at 7:52 AM Michael Paquier wrote: > > On Wed, Oct 06, 2021 at 07:19:04AM +0530, Bharath Rupireddy wrote: > > I was thinking of the same. +1 for "vcregress check-world" which is > > more in sync with it's peer "make check-world" instead of "vcregress > > all-taptest". I'm not su

Re: postgres_fdw: Obsolete comments in GetConnection()

2021-10-06 Thread Etsuro Fujita
Hi Bharath, On Wed, Oct 6, 2021 at 6:37 PM Bharath Rupireddy wrote: > +1 for rewording the comments. Here are my thoughts on the patch: > > 1) Just to be consistent(we are using this word in the error message, > and in other comments around there), how about > + * Determine whether to try to rees

Re: More business with $Test::Builder::Level in the TAP tests

2021-10-06 Thread Andrew Dunstan
On 10/6/21 2:28 AM, Michael Paquier wrote: > Hi all, > > Following up with Peter E's recent commit 73aa5e0 to add some > forgotten level incrementations, I got to look again at what I did > wrong and why this stuff is useful. > > I have gone through all the TAP tests and any code paths using > su

Re: postgres_fdw: Obsolete comments in GetConnection()

2021-10-06 Thread Bharath Rupireddy
On Wed, Oct 6, 2021 at 4:55 PM Etsuro Fujita wrote: > > Hi Bharath, > > On Wed, Oct 6, 2021 at 6:37 PM Bharath Rupireddy > wrote: > > +1 for rewording the comments. Here are my thoughts on the patch: > > > > 1) Just to be consistent(we are using this word in the error message, > > and in other co

Re: More business with $Test::Builder::Level in the TAP tests

2021-10-06 Thread Daniel Gustafsson
> On 6 Oct 2021, at 13:33, Andrew Dunstan wrote: > We should probably state a requirement for this somewhere. Maybe in > src/test/perl/README. +1, I think that sounds like a very good idea. -- Daniel Gustafsson https://vmware.com/

Re: Capitalization of localized month and day names (to_char() with 'TMmonth', 'TMday', etc.)

2021-10-06 Thread Juan José Santamaría Flecha
On Wed, Oct 6, 2021 at 11:09 AM Magnus Holmgren wrote: > > There's just this tiny but seemingly obvious issue that I can't believe I > haven't noticed until now: to_date(now(), 'TMmonth') returns 'october' in > an > English locale (en_US.UTF-8 at least). Names of months and weekdays are > proper

Re: Pre-allocating WAL files

2021-10-06 Thread Maxim Orlov
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, failed Spec compliant: not tested Documentation:not tested Hi! We've looked through the code and everything looks good except f

Re: TAP test for recovery_end_command

2021-10-06 Thread Amul Sul
On Wed, Oct 6, 2021 at 1:40 AM Andres Freund wrote: > > Hi, > > On 2021-09-14 10:34:09 +0530, Amul Sul wrote: > > +# recovery_end_command_file executed only on recovery end which can happen > > on > > +# promotion. > > +$standby3->promote; > > +ok(-f "$recovery_end_command_file", > > + 'recov

Re: storing an explicit nonce

2021-10-06 Thread Bruce Momjian
On Tue, Oct 5, 2021 at 04:29:25PM -0400, Bruce Momjian wrote: > On Tue, Sep 28, 2021 at 12:30:02PM +0300, Ants Aasma wrote: > > On Mon, 27 Sept 2021 at 23:34, Bruce Momjian wrote: > > We are still working on our TDE patch. Right now the focus is on refactoring > > temporary file access to make th

Re: Partition Check not updated when insert into a partition

2021-10-06 Thread Pavel Borisov
Hi, hackers! We've reviewed patch v3 and found it right. Completely agree that in case we attach/detach partition relcaches for all child relations (if they exist) need invalidation. Installcheck world succeeds after the patch. Tests from the patch fail as they should when run on the master branch

Re: using an end-of-recovery record in all cases

2021-10-06 Thread Amul Sul
On Tue, Oct 5, 2021 at 10:42 PM Robert Haas wrote: > > On Tue, Oct 5, 2021 at 12:41 PM Amul Sul wrote: > > No, InRecovery flag get cleared before this point. I think, we can use > > lastReplayedEndRecPtr what you have suggested in other thread. > > Hmm, right, that makes sense. Perhaps I should

Re: wrapping CF 2021-09

2021-10-06 Thread Jaime Casanova
On Wed, Oct 06, 2021 at 12:21:02AM -0500, Jaime Casanova wrote: > Hi everyone, > > At this time I'm looking to close the CF. > Closed > So, I will write and mark as RwF the patches currently failing on the > cfbot and move to the next CF all other patches. > > RwF: > - ALTER SYSTEM READ { ONLY

Re: Capitalization of localized month and day names (to_char() with 'TMmonth', 'TMday', etc.)

2021-10-06 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= writes: > On Wed, Oct 6, 2021 at 11:09 AM Magnus Holmgren > wrote: >> There's just this tiny but seemingly obvious issue that I can't believe I >> haven't noticed until now: to_date(now(), 'TMmonth') returns 'october' in >> an >> English locale (

Re: Compressing temporary files

2021-10-06 Thread Bruce Momjian
On Sat, Sep 11, 2021 at 05:31:37PM +0500, Andrey Borodin wrote: > How do you think: does it worth to pursue the idea? OLTP systems rarely rely > on data spilled to disk. > Are there any known good random access compressed file libs? So we could > avoid reinventing the wheel. > Maybe someone tried

Re: Improved tab completion for PostgreSQL parameters in enums

2021-10-06 Thread Tom Lane
bt21masumurak writes: > If you do tab completion in a situation like A, you will see ["on"] > instead of [on]. > A : "ALTER SYSTEM SET wal_compression TO " > I made a patch for this problem. I do not think this is an improvement. It will result in omitting quotes in some cases where they're n

Re: Triage on old commitfest entries

2021-10-06 Thread Robert Haas
On Tue, Oct 5, 2021 at 12:56 PM Jaime Casanova wrote: > > > psql - add SHOW_ALL_RESULTS option 11 > > > Last substantive discussion 2021-09, currently passing cfbot > > > > > > This got committed and reverted once already. I have to be > > > suspicious of whether this is a good desig

Re: Extension relocation vs. schema qualification

2021-10-06 Thread Robert Haas
On Tue, Oct 5, 2021 at 8:26 AM Tom Lane wrote: > I spent some time awhile ago on fixing this via new-style SQL functions > [1]. That didn't get finished for reasons explained in the thread, > but I think that's probably a more productive direction to go in. Hmm, if that'll solve the problem, +1

Re: Compressing temporary files

2021-10-06 Thread Robert Haas
On Sat, Sep 11, 2021 at 8:31 AM Andrey Borodin wrote: > I've prototyped Random Access Compressed File for fun[0]. The code is very > dirty proof-of-concept. > I compress Buffile by one block at a time. There are directory pages to store > information about the size of each compressed block. If a

Re: storing an explicit nonce

2021-10-06 Thread Robert Haas
On Tue, Oct 5, 2021 at 1:55 PM Antonin Houska wrote: > I'm just trying to make our changes to buffile.c less invasive. Or do you mean > that this module should be reworked regardless the encryption? I wasn't thinking of buffile.c specifically. I think improving that might be a really good idea, a

Re: storing an explicit nonce

2021-10-06 Thread Robert Haas
On Tue, Oct 5, 2021 at 4:29 PM Bruce Momjian wrote: > On Tue, Sep 28, 2021 at 12:30:02PM +0300, Ants Aasma wrote: > > On Mon, 27 Sept 2021 at 23:34, Bruce Momjian wrote: > > We are still working on our TDE patch. Right now the focus is on refactoring > > temporary file access to make the TDE patc

Re: Compressing temporary files

2021-10-06 Thread Tomas Vondra
Hi, On 9/11/21 2:31 PM, Andrey Borodin wrote: > Hi hackers! > > There's a lot of compression discussions nowadays. And that's cool! > Recently Naresh Chainani in private discussion shared with me the > idea to compress temporary files on disk. And I was thrilled to find > no evidence of implemen

Re[2]: [PATCH] Tracking statements entry timestamp in pg_stat_statements

2021-10-06 Thread Мельников Антон Андреевич
Hi, Andrey!   I’ve tried to apply your patch v2-0001 on current master, but i failed. There were git apply errors at: pg_stat_statements.out:941 pg_stat_statements.sql:385   Best Regards, Anton Melnikov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company  

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Pavel Borisov
Hi, hackers! We've looked through the initial patch and the exclusion of temporary tables from pg_amcheck seems the right thing. Also it is not the matter anyone disagrees here, and we propose to commit it alone. Supplementary things/features might be left for further discussion but refusing to ch

Re: storing an explicit nonce

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 9:35 AM Bruce Momjian wrote: > The above text isn't very clear. What I am saying is that currently > torn pages can be tolerated by hint bit writes because only a single > byte is changing. If we use a block cipher like AES-XTS, later 16-byte > encrypted blocks would be ch

Re: storing an explicit nonce

2021-10-06 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, Oct 5, 2021 at 1:24 PM Robert Haas wrote: > > On Mon, Oct 4, 2021 at 10:00 PM Stephen Frost wrote: > > > I do want to point out, as I think I did when we discussed this but want > > > to be sure it's also captured here- I don't th

Re: Role Self-Administration

2021-10-06 Thread Robert Haas
On Tue, Oct 5, 2021 at 3:41 PM Mark Dilger wrote: > If roles have owners, then DROP ROLE bob CASCADE drops bob, bob's objects, > roles owned by bob, their objects and any roles they own, recursively. Roles > which bob merely has admin rights on are unaffected, excepting that they are > adminis

Re: storing an explicit nonce

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 11:22 AM Stephen Frost wrote: > Seems like it'd be easier to achieve that by having something that looks > very close to how write() looks, but just happens to have the option to > run the data through a stream cipher and maybe does better error > handling for us. Making th

Re: Running tests under valgrind is getting slower at an alarming pace

2021-10-06 Thread Tom Lane
Andres Freund writes: > After a recent migration of the skink and a few other animals (sorry for the > false reports on BF, I forgot to adjust a path), I looked at the time it takes > to complete a valgrind run: > 9.6: Consumed 4h 53min 18.518s CPU time > 10: Consumed 5h 32min 50.839s CPU time >

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, * Mark Dilger (mark.dil...@enterprisedb.com) wrote: > > On Oct 5, 2021, at 10:20 AM, Stephen Frost wrote: > > On Tue, Oct 5, 2021 at 13:17 Mark Dilger > > wrote: > > > On Oct 5, 2021, at 10:14 AM, Stephen Frost wrote: > > > > > > What does the “ownership” concept actually buy us th

Re: Running tests under valgrind is getting slower at an alarming pace

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 1:57 AM Andres Freund wrote: > After a recent migration of the skink and a few other animals (sorry for the > false reports on BF, I forgot to adjust a path), I looked at the time it takes > to complete a valgrind run: > > 9.6: Consumed 4h 53min 18.518s CPU time > 10: Consum

Re: Use simplehash.h instead of dynahash in SMgr

2021-10-06 Thread Yura Sokolov
Good day, David and all. В Вт, 05/10/2021 в 11:07 +1300, David Rowley пишет: > On Mon, 4 Oct 2021 at 20:37, Jaime Casanova > wrote: > > Based on your comments I will mark this patch as withdrawn at midday > > of > > my monday unless someone objects to that. > > I really think we need a hash tabl

Re: Role Self-Administration

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 9:01 AM, Stephen Frost wrote: > > I can see how what you describe as the behavior you'd like to see of > DROP ROLE ... CASCADE could be useful... However, at least in the > latest version of the standard that I'm looking at, when a > DROP ROLE ... CASCADE is executed, wh

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2021-10-06 Thread Yura Sokolov
I've made some remarks in related thread: https://www.postgresql.org/message-id/flat/0A3221C70F24FB45833433255569204D1FB976EF@G01JPEXMBYT05 The new status of this patch is: Waiting on Author

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 8:14 AM, Pavel Borisov wrote: > > We've looked through the initial patch and the exclusion of temporary tables > from pg_amcheck seems the right thing. Also it is not the matter anyone > disagrees here, and we propose to commit it alone. Thanks for reviewing! I expect

Re: [Proposal] Global temporary tables

2021-10-06 Thread Andrew Bille
On master with the v54 patches applied the following script leads to crash: export ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:disable_coredump=0:strict_string_checks=1:check_initialization_order=1:strict_init_order=1 initdb -D data pg_ctl -w -t 5 -D data -l server.log start psql -c "create global

Re: Pre-allocating WAL files

2021-10-06 Thread Bossart, Nathan
On 10/6/21, 5:20 AM, "Maxim Orlov" wrote: > We've looked through the code and everything looks good except few minor > things: > 1). Using dedicated bg worker seems not optimal, it introduces a lot of > redundant code for little single action. > We'd join initial proposal of Andres to implem

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-10-06 Thread Bossart, Nathan
On 10/6/21, 12:13 AM, "Masahiko Sawada" wrote: > A customer reported that during parallel index vacuum, the oldest xmin > doesn't advance. Normally, the calculation of oldest xmin > (ComputeXidHorizons()) ignores xmin/xid of processes having > PROC_IN_VACUUM flag in MyProc->statusFlags. But since

Re: Running tests under valgrind is getting slower at an alarming pace

2021-10-06 Thread Andres Freund
Hi, On 2021-10-06 12:09:36 -0400, Robert Haas wrote: > Is the problem here that we're adding a lot of new new test cases? Or > is the problem that valgrind runs are getting slower for the same > number of test cases? I don't know precisely. It's probably a combination of several factors. I do thi

Re: storing an explicit nonce

2021-10-06 Thread Bruce Momjian
On Wed, Oct 6, 2021 at 11:01:25AM -0400, Robert Haas wrote: > On Tue, Oct 5, 2021 at 4:29 PM Bruce Momjian wrote: > > On Tue, Sep 28, 2021 at 12:30:02PM +0300, Ants Aasma wrote: > > > On Mon, 27 Sept 2021 at 23:34, Bruce Momjian wrote: > > > We are still working on our TDE patch. Right now the f

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2021-10-06 Thread Jeremy Schneider
On 10/5/21 17:43, Bruce Momjian wrote: > On Tue, Oct 5, 2021 at 03:30:07PM -0700, Jeremy Schneider wrote: >> Specifically exposing pg_waldump functionality in SQL could speed up >> finding bugs in the PG logical replication code. We found and fixed a >> few over this past year, but there are more

Re: [PATCH] Added TRANSFORM FOR for COMMENT tab completion

2021-10-06 Thread Suraj Khamkar
Hello, I reviewed your patch. At a first glance, I have the below comments. 1. The below change crosses the 80-character limit in a line. Please maintain the same. - "LARGE OBJECT", "TABLESPACE", "TEXT SEARCH", "ROLE"); + "LARGE OBJECT", "TABLESPACE", "TEXT SEARCH", "TRANSFORM FOR"

Re: Running tests under valgrind is getting slower at an alarming pace

2021-10-06 Thread Tom Lane
Andres Freund writes: > One thing I think would really help is having the total time for each run > visible in an animals run history. That way we could pinpoint regressions > reasonably efficiently, right now that's not easily possible without writing > nontrivial queries to the buildfarm databas

Re: storing an explicit nonce

2021-10-06 Thread Bruce Momjian
On Wed, Oct 6, 2021 at 12:54:49PM -0400, Bruce Momjian wrote: > On Wed, Oct 6, 2021 at 11:01:25AM -0400, Robert Haas wrote: > > On Tue, Oct 5, 2021 at 4:29 PM Bruce Momjian wrote: > > > On Tue, Sep 28, 2021 at 12:30:02PM +0300, Ants Aasma wrote: > > > > On Mon, 27 Sept 2021 at 23:34, Bruce Momji

Re: storing an explicit nonce

2021-10-06 Thread Bruce Momjian
On Wed, Oct 6, 2021 at 11:17:59AM -0400, Robert Haas wrote: > If you enable checksums or set wal_log_hints=on, then you might incur > a some write-ahead log records that would otherwise be avoided, and > those records will include full page images. This can happen once per > page per checkpoint cy

Re: Running tests under valgrind is getting slower at an alarming pace

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 12:47 PM Andres Freund wrote: > There's probably some of that. > > The fact that the tap test infrastructure does all communication with the > server via psql each only execute only a single query is a problem - > connection startup is expensive. Ageed. safe_psql() is a poo

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 9:25 AM Mark Dilger wrote: > Thanks for reviewing! > > I expect to post a new version shortly. Not sure how much it matters, but I have some thoughts on the return value of pg_amcheck. (I'm mostly going into this now because it seems related to how we discuss these issues g

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 10:16 AM, Peter Geoghegan wrote: > > A return value of 0 cannot be said to indicate that the database is > not corrupt; Nor can a non-zero value be said to indicate that the database is corrupt. These invocations will still return a non-zero exit status: pg_amch

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2021-10-06 Thread Bruce Momjian
On Wed, Oct 6, 2021 at 09:56:34AM -0700, Jeremy Schneider wrote: > On 10/5/21 17:43, Bruce Momjian wrote: > > On Tue, Oct 5, 2021 at 03:30:07PM -0700, Jeremy Schneider wrote: > >> Specifically exposing pg_waldump functionality in SQL could speed up > >> finding bugs in the PG logical replication

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, * Mark Dilger (mark.dil...@enterprisedb.com) wrote: > > On Oct 6, 2021, at 9:01 AM, Stephen Frost wrote: > > I can see how what you describe as the behavior you'd like to see of > > DROP ROLE ... CASCADE could be useful... However, at least in the > > latest version of the standard th

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2021-10-06 Thread Alvaro Herrera
On 2021-Oct-06, Jeremy Schneider wrote: > Well this whole conversation is just theoretical anyway until the code > is shared. :) But if Bharath is writing functions to decode WAL, then > wouldn't we just have pg_waldump use these same functions in order to > avoid duplicating code? Actually, a

Re: Role Self-Administration

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 10:20 AM, Stephen Frost wrote: > > Consistency is not having dangling pointers around to things which no > longer exist- FK reference kind of things. Object management is about > actual *removal* of full blown objects like roles, tables, etc. DROP > TABLE ... CASCADE doe

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 10:19 AM Mark Dilger wrote: > > A return value of 0 cannot be said to indicate that the database is > > not corrupt; > > Nor can a non-zero value be said to indicate that the database is corrupt. I never said otherwise. I think it's perfectly fine that there are multiple no

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 10:39 AM, Peter Geoghegan wrote: > >> The differences in the upcoming version are >> >> 1) --all no longer means "all relations" but rather "all checkable relations" > > Clearly pg_amcheck never checked all relations, because it never > checked indexes that are not B-Tre

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, * Mark Dilger (mark.dil...@enterprisedb.com) wrote: > > On Oct 6, 2021, at 10:20 AM, Stephen Frost wrote: > > > > Consistency is not having dangling pointers around to things which no > > longer exist- FK reference kind of things. Object management is about > > actual *removal* of fu

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 10:57 AM Mark Dilger wrote: > > Clearly pg_amcheck never checked all relations, because it never > > checked indexes that are not B-Tree indexes. I'm pretty sure that I > > can poke big holes in almost any positivist statement like that with > > little effort. > > There is a

Re: Role Self-Administration

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 11:09 AM, Stephen Frost wrote: > > After all, it says > "REOKVE R FROM A DB", not "DROP ROLE A CASCADE". Wait, are you arguing what DROP ROLE A CASCADE should do based on what the spec says REVOKE R FROM A DB should do? If so, I'd say that's irrelevant. I'm not propo

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 1:57 PM Mark Dilger wrote: > > To me #2 sounds like a tautology. It could almost be phrased as > > "pg_amcheck does not check the things that it cannot check". > > I totally disagree. It is uncomfortable to me that `pg_amcheck > --parent-check` will now silently not perfor

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, * Mark Dilger (mark.dil...@enterprisedb.com) wrote: > > On Oct 6, 2021, at 11:09 AM, Stephen Frost wrote: > > After all, it says > > "REOKVE R FROM A DB", not "DROP ROLE A CASCADE". > > Wait, are you arguing what DROP ROLE A CASCADE should do based on what the > spec says REVOKE R F

Re: Running tests under valgrind is getting slower at an alarming pace

2021-10-06 Thread Andres Freund
Hi, On 2021-10-06 12:58:34 -0400, Tom Lane wrote: > Andres Freund writes: > > One thing I think would really help is having the total time for each run > > visible in an animals run history. That way we could pinpoint regressions > > reasonably efficiently, right now that's not easily possible wi

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 11:32 AM Robert Haas wrote: > All of the decisions we're talking about here really have to do with > determining the user's intent. I think that if the user says > pg_amcheck --all, there's a good argument that they don't want us to > check unlogged relations on a standby wh

Re: Role Self-Administration

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 2:48 PM Stephen Frost wrote: > What I'm saying above is that the command explicitly listed there > *isn't* 'DROP ROLE A DB', even though that is something which the spec > *could* have done, had they wished to. Given that they didn't, it seems > very clear that making such

Re: storing an explicit nonce

2021-10-06 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Oct 5, 2021 at 04:29:25PM -0400, Bruce Momjian wrote: > > On Tue, Sep 28, 2021 at 12:30:02PM +0300, Ants Aasma wrote: > > > On Mon, 27 Sept 2021 at 23:34, Bruce Momjian wrote: > > > We are still working on our TDE patch. Right now th

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 11:55 AM Peter Geoghegan wrote: > I am pretty sure that I agree with you about all these details. We > need to tease them apart some more. I think that what I've said boils down to this: * pg_amcheck shouldn't attempt to verify temp relations, on the grounds that this is f

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Oct 6, 2021 at 2:48 PM Stephen Frost wrote: > > What I'm saying above is that the command explicitly listed there > > *isn't* 'DROP ROLE A DB', even though that is something which the spec > > *could* have done, had they wished to.

Re: using extended statistics to improve join estimates

2021-10-06 Thread Tomas Vondra
Hi, attached is an improved version of this patch, addressing some of the points mentioned in my last message: 1) Adds a couple regression tests, testing various join cases with expressions, additional conditions, etc. 2) Adds support for expressions, so the join clauses don't need to reference

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 2:56 PM Peter Geoghegan wrote: > --heapallindexed doesn't complicate things for us at all. It changes > nothing about the locking considerations. It's just an additive thing, > some extra checks with the same basic underlying requirements. Maybe > you meant to say --parent-c

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 12:28 PM, Peter Geoghegan wrote: > > I think that what I've said boils down to this: > > * pg_amcheck shouldn't attempt to verify temp relations, on the > grounds that this is fundamentally not useful, and not something that > could ever be sensibly interpreted as "just d

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 12:33 PM Robert Haas wrote: > To me, it doesn't matter which specific option we're talking about. If > I tell pg_amcheck to pass a certain flag to the underlying functions, > then it should do that. If the behavior needs to be changed, it should > be changed in those underly

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2021-10-06 Thread Jelte Fennema
We actually ran into an issue caused by this in production, where a PQcancel connection was open on the client for a 2+ days because the server had restarted at the wrong moment in the cancel handshake. The client was now indefinitely waiting for the server to send an EOF back, and because keepa

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2021-10-06 Thread Jelte Fennema
Ugh forgot to attach the patch. Here it is. From: Jelte Fennema Sent: Wednesday, October 6, 2021 21:56 To: Zhihong Yu Cc: pgsql-hack...@postgresql.org Subject: Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings We

Re: Role Self-Administration

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 11:48 AM, Stephen Frost wrote: > > In the spec, under , there is a 'General Rules' > section (as there is with most statements) and in that section it says > that for every authorization identifier (that is, some privilege, be it > a GRANT of SELECT rights on an object, or

Re: storing an explicit nonce

2021-10-06 Thread Bruce Momjian
On Wed, Oct 6, 2021 at 03:17:00PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, Oct 5, 2021 at 04:29:25PM -0400, Bruce Momjian wrote: > > > On Tue, Sep 28, 2021 at 12:30:02PM +0300, Ants Aasma wrote: > > > > On Mon, 27 Sept 2021 at 23:34, Bruc

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 12:36 PM Mark Dilger wrote: > The user may not know that the system has changed. > > For example, if I see errors in the logs suggesting corruption in a relation > named "mark" and run pg_amcheck --relation=mark, I expect that to check the > relation. If that relation is

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 3:56 PM Peter Geoghegan wrote: > I agree, with the stipulation that the caller (in this case > pg_amcheck) is required to know certain basic things about the > relation in order to get useful behavior. For example, if you use > bt_index_check() with a GIN index, you're going

Re: Role Self-Administration

2021-10-06 Thread Robert Haas
On Wed, Oct 6, 2021 at 3:29 PM Stephen Frost wrote: > Does that mean that we also get to change what a specific set of > commands, which are all well-defined in the standard, do even when that > goes against what an SQL compliant implementation would do? I really > don't think so. If this was *n

Re: parallelizing the archiver

2021-10-06 Thread Magnus Hagander
On Tue, Oct 5, 2021 at 5:32 AM Bossart, Nathan wrote: > On 10/4/21, 8:19 PM, "Stephen Frost" wrote: > > It's also been discussed, at least around the water cooler (as it were > > in pandemic times- aka our internal slack channels..) that the existing > > archive command might be reimplemented as

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2021-10-06 Thread Robert Haas
On Mon, Oct 4, 2021 at 12:44 PM Shruthi Gowda wrote: > Thanks for the inputs, Robert. In the v4 patch, an unused OID (i.e, 4) > is fixed for the template0 and the same is removed from unused oid > list. > > In addition to the review comment fixes, I have removed some code that > is no longer neede

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, On Wed, Oct 6, 2021 at 16:01 Mark Dilger wrote: > > On Oct 6, 2021, at 11:48 AM, Stephen Frost wrote: > > > > In the spec, under , there is a 'General Rules' > > section (as there is with most statements) and in that section it says > > that for every authorization identifier (that i

Re: Role Self-Administration

2021-10-06 Thread Stephen Frost
Greetings, On Wed, Oct 6, 2021 at 16:28 Robert Haas wrote: > On Wed, Oct 6, 2021 at 3:29 PM Stephen Frost wrote: > > Does that mean that we also get to change what a specific set of > > commands, which are all well-defined in the standard, do even when that > > goes against what an SQL complian

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 1:15 PM Robert Haas wrote: > > Where I might go further than you or Mark (not sure) is on this: I > > also think that it's the caller's job to not call the functions with > > temp relations, or (in the case of the index verification stuff) with > > !indisready or !indisvalid

Re: ALTER INDEX .. RENAME allows to rename tables/views as well

2021-10-06 Thread Bruce Momjian
I can confirm this bug in git head, and I think it should be fixed. --- On Mon, Oct 4, 2021 at 10:23:23AM +, Onder Kalaci wrote: > Hi hackers, > > I realized a subtle behavior with ALTER INDEX .. RENAME. It seems like

Re: using extended statistics to improve join estimates

2021-10-06 Thread Zhihong Yu
On Wed, Oct 6, 2021 at 12:33 PM Tomas Vondra wrote: > Hi, > > attached is an improved version of this patch, addressing some of the > points mentioned in my last message: > > 1) Adds a couple regression tests, testing various join cases with > expressions, additional conditions, etc. > > 2) Adds

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Pavel Borisov
> > > It is the most consistent with the general design of the system, for > reasons that are pretty deeply baked into the system. I'm reminded of > the fact that REINDEX CONCURRENTLY's completion became blocked due to > similar trepidations. Understandably so. I may mistake, but I recall the fac

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 1:49 PM, Peter Geoghegan wrote: > >> The analogy here is: are we trying to verify that the relations are >> valid? Or are we just trying to verify that they are as valid as we >> can expect them to be? > > I think that we do the latter (or something much closer to the lat

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 2:45 PM, Mark Dilger wrote: > > and db3 is in recovery. > they're scattered across different databases, some in recovery, some not. What I mean here is that, since pg_amcheck might run for many hours, and database may start in recovery but then exit recovery, or may b

Re: using extended statistics to improve join estimates

2021-10-06 Thread Tomas Vondra
On 10/6/21 23:03, Zhihong Yu wrote: Hi, +       conditions2 = statext_determine_join_restrictions(root, rel, mcv); + +       /* if the new statistics covers more conditions, use it */ +       if (list_length(conditions2) > list_length(conditions1)) +       { +           mcv = stat; It seems con

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Peter Geoghegan
On Wed, Oct 6, 2021 at 2:45 PM Mark Dilger wrote: > I think the disagreements are about something else. Informally speaking, you could say that pg_amcheck and amcheck verify relations. More formally speaking, both amcheck (whether called by pg_amcheck or some other thing) can only prove the prese

Re: ALTER INDEX .. RENAME allows to rename tables/views as well

2021-10-06 Thread Tom Lane
"Bossart, Nathan" writes: > On 10/6/21, 1:52 PM, "Bruce Momjian" wrote: >> I can confirm this bug in git head, and I think it should be fixed. > Here's a patch that ERRORs if the object type and statement type do > not match. Interestingly, some of the regression tests were relying > on this be

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-06 Thread Mark Dilger
> On Oct 6, 2021, at 3:20 PM, Peter Geoghegan wrote: > >> I think the disagreements are about something else. > > Informally speaking, you could say that pg_amcheck and amcheck verify > relations. More formally speaking, both amcheck (whether called by > pg_amcheck or some other thing) can on

  1   2   >