Re: POC: Better infrastructure for automated testing of concurrency issues

2021-02-22 Thread Peter Geoghegan
removed by..."). Same is true of the closely related heap_prepare_freeze_tuple()/heap_tuple_needs_freeze() code. -- Peter Geoghegan

Re: 64-bit XIDs in deleted nbtree pages

2021-02-24 Thread Peter Geoghegan
On Mon, Feb 22, 2021 at 2:54 PM Peter Geoghegan wrote: > Good point. I think that the structure should make the page deletion > triggering condition have only secondary importance -- it is only > described at all to be complete and exhaustive. The > vacuum_cleanup_index_scale_fa

Re: 64-bit XIDs in deleted nbtree pages

2021-02-24 Thread Peter Geoghegan
onsidered within btvacuumcleanup()" bug. (Though I'm less confident on this second point about a backpatchable fix.) -- Peter Geoghegan

Re: cursor sensitivity misunderstanding

2021-02-25 Thread Peter Eisentraut
On 18.02.21 19:14, Peter Eisentraut wrote: On 18.02.21 17:11, David G. Johnston wrote: The OP was doing a course based on Oracle and was confused regarding our behavior.  The documentation failed to help me provide a useful response, so I'd agree there is something here that needs reworki

Re: SSL SNI

2021-02-25 Thread Peter Eisentraut
es from host names that are less ad hoc, I would welcome them. From bef8152b6f4ad2ed2ccd8158088a35b2cf625491 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Thu, 25 Feb 2021 16:40:32 +0100 Subject: [PATCH v2] Set SNI for SSL connections from the client This allows an SNI-aware proxy to route connections. Discussion:

Re: SSL SNI

2021-02-25 Thread Peter Eisentraut
On 17.02.21 00:01, Jacob Champion wrote: On Mon, 2021-02-15 at 15:09 +0100, Peter Eisentraut wrote: The question I had was whether this should be an optional behavior, or conversely a behavior that can be turned off, or whether it should just be turned on all the time. Personally I think there

Re: 64-bit XIDs in deleted nbtree pages

2021-02-25 Thread Peter Geoghegan
code doesn't trust the statistics too much (it's okay for VACUUM VERBOSE output only). Right now we can get a pg_class.reltuples that is "exactly wrong" -- it would actually be a big improvement if it was "approximately correct". Another new concern for me (another concer

Re: SSL SNI

2021-02-25 Thread Peter Eisentraut
On 26.02.21 03:40, Greg Stark wrote: This still doesn't seem like it is IPv6-ready. Do you mean the IPv6 detection code is not correct? What is the problem? > Is there any harm in> having SNI with an IPv6 address there if it gets through? I doubt it.

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-26 Thread Peter Smith
ng_table_states signal handler) only to be immediately/wrongly overwritten as table_states_valid = true in this FetchTableStates code. -- Kind Regards, Peter Smith. Fujitsu Australia

Re: SEARCH and CYCLE clauses

2021-02-26 Thread Peter Eisentraut
On 22.02.21 14:45, Vik Fearing wrote: On 2/22/21 1:28 PM, Peter Eisentraut wrote: On 22.02.21 11:05, Vik Fearing wrote: This looks good to me, except that you forgot to add the feature stamp.   Attached is a small diff to apply on top of your patch to fix that. The feature code is from SQL

Re: psql - add SHOW_ALL_RESULTS option

2021-02-27 Thread Peter Eisentraut
.com >From 22ac191084db1b6ab155202a09bc1a6fedde794f Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Sat, 27 Feb 2021 11:50:50 +0100 Subject: [PATCH v10] psql: Show all query results by default Author: Author: Fabien COELHO Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.21.19041

Re: macOS SIP, next try

2021-02-28 Thread Peter Eisentraut
owing the same route. I suggest the attached documentation patch and then close this issue. From 7efb0ec3e15f37f9c5e12845aeccd9cd8693c01d Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Mon, 1 Mar 2021 07:58:17 +0100 Subject: [PATCH] doc: Update information on macOS SIP On more recent ver

Re: 64-bit XIDs in deleted nbtree pages

2021-03-01 Thread Peter Geoghegan
y Postgres 13 bug fix commit 48e12913). These two issues make removing vacuum_cleanup_index_scale_factor tempting, even in the back branches -- it might actually be the more conservative approach, at least for Postgres 13. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-01 Thread Peter Geoghegan
age-id/flat/20130108024957.GA4751%40tornado.leadboat.com Of course, this effort to eliminate the "tupgone = true"/XLOG_HEAP2_CLEANUP_INFO special case didn't go anywhere at the time. [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=7cde6b13a9b630e2f04d91e2f17dedc2afee21c6 [2] https://coverage.postgresql.org/src/backend/access/heap/vacuumlazy.c.gcov.html -- Peter Geoghegan

Re: 64-bit XIDs in deleted nbtree pages

2021-03-01 Thread Peter Geoghegan
On Mon, Mar 1, 2021 at 1:40 PM Peter Geoghegan wrote: > > Since it seems not a bug I personally think we don't need to do > > anything for back branches. But if we want not to trigger an index > > scan by vacuum_cleanup_index_scale_factor, we could change the default &g

Re: 64-bit XIDs in deleted nbtree pages

2021-03-01 Thread Peter Geoghegan
n only truly fix the problem in Postgres 14 anyway, and should just accept that? OTOH scanning the indexes for no reason when autovacuum_vacuum_insert_scale_factor triggers an autovacuum VACUUM does seem *particularly* silly. So I don't know what to think. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-01 Thread Peter Geoghegan
On Mon, Mar 1, 2021 at 7:00 PM Peter Geoghegan wrote: > I think that you're right. However, in practice it isn't harmful > because has_dead_tuples is only used when "all_visible = true", and > only to detect corruption (which should never happen). I think that it >

Re: pg_upgrade version checking questions

2021-03-02 Thread Peter Eisentraut
On 23.02.21 17:14, Daniel Gustafsson wrote: This exports validate_exec to reduce duplication, and implements a custom find_other_exec-like function in pg_upgrade to check each binary for the version number. Keeping a local copy of validate_exec is easy to do if it's deemed not worth it to export

Re: SQL-standard function body

2021-03-02 Thread Peter Eisentraut
On 11.02.21 09:02, Peter Eisentraut wrote: Here is an updated patch to get it building again. Another updated patch to get things building again.  I've also fixed the last TODO I had in there in qualifying function arguments as necessary in ruleutils.c. Updated patch to resolve

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-02 Thread Peter Geoghegan
intelligent about overwriting pg_class stats for indexes? I define "real index vacuuming" as calling any indexes ambulkdelete() routine. 2. Does anybody anticipate any other issues? Possibly an issue that resembles this existing known VACUUM ANALYZE issue? Thanks -- Peter Geoghegan

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2021-03-02 Thread Peter Geoghegan
updating pg_attribute, for example (which is something that the tests do already). -- Peter Geoghegan

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2021-03-02 Thread Peter Geoghegan
ally don't like that the new unique checking functionality merely warns that the index *might* be corrupt. False positives are always unacceptable within amcheck, and I think that this is a false positive. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-02 Thread Peter Geoghegan
at the > > time. > > I'll look at that thread. I'm not sure if it's super valuable to look at the thread. But it is reassuring to see that Noah shared the intuition that the "tupgone = true" case was kind of bad, even back in 2013. It's one part of my "mental map" of VACUUM. -- Peter Geoghegan

Re: pg_upgrade version checking questions

2021-03-03 Thread Peter Eisentraut
On 02.03.21 22:51, Daniel Gustafsson wrote: The commit message says something about "to ensure the health of the target cluster", which doesn't make sense to me. Maybe find a better wording. Reworded in the attached updated version. The name find_exec() seems not very accurate. It doesn't

Re: macOS SIP, next try

2021-03-03 Thread Peter Eisentraut
On 01.03.21 15:44, Tom Lane wrote: Peter Eisentraut writes: I have since learned that there is a way to disable only the part of SIP that is relevant for us. This seems like a useful compromise, and it appears that a number of other open-source projects are following the same route. I

Re: authtype parameter in libpq

2021-03-03 Thread Peter Eisentraut
On 26.02.21 21:02, Daniel Gustafsson wrote: When looking at disallowing SSL compression I found the parameter "authtype" which was deprecated in commit d5bbe2aca55bc8 on January 26 1998. While I do think there is a case to be made for the backwards compatibility having run its course on this one

Re: OpenSSL 3.0.0 compatibility

2021-03-03 Thread Peter Eisentraut
This thread is still in the commit fest, but I don't see any actual proposed patch still pending. Most of the activity has moved into other threads. Could you update the status of this CF entry, and perhaps also on the status of OpenSSL compatibility in general?

Re: Let people set host(no)ssl settings from initdb

2021-03-03 Thread Peter Eisentraut
On 01.01.21 14:12, Magnus Hagander wrote: That said, I agree with not adding it as an option to initdb. You'll quickly get to the point where you specify the whole pg_hba file on the commandline to initdb -- and most people today who actually care that much about it would have their pg_hba.conf

Re: Disallow SSL compression?

2021-03-03 Thread Peter Eisentraut
On 03.03.21 11:31, Daniel Gustafsson wrote: On 26 Feb 2021, at 20:34, Daniel Gustafsson wrote: Attached is a v2 which retains the sslcompression parameter for backwards compatibility. And now a v3 which fixes an oversight in postgres_fdw as well as adds an SSL TAP test to cover deprecated

Re: [patch] bit XOR aggregate functions

2021-03-03 Thread Peter Eisentraut
On 10.02.21 06:42, Kyotaro Horiguchi wrote: We already had CREATE AGGREATE at the time, so BIT_XOR can be thought as it falls into the same category with BIT_AND and BIT_OR, that is, we may have BIT_XOR as an intrinsic aggregation? I think the use of BIT_XOR is quite separate from BIT_AND and B

Re: contrib/cube - binary input/output handlers

2021-03-03 Thread Peter Eisentraut
On 24.02.21 04:18, Kohei KaiGai wrote: This patch adds cube_send / cube_recv handlers on the contrib/cube data type. Once this patch was applied to, the libpq client can obtain the table data using binary mode. Seems reasonable. But you need to write an extension upgrade script and bump the e

Re: Pg14, pg_dumpall and "password_encryption=true"

2021-03-03 Thread Peter Eisentraut
On 18.01.21 07:18, Michael Paquier wrote: This would be interpreting setconfig='{password_encryption=on}' as "opt out of future password security increases". I expect that will tend not to match the intent of the person entering the setting. That said, if v14 were already behaving this way, I w

Re: [PATCH] Add --create-only option to pg_dump/pg_dumpall

2021-03-03 Thread Peter Eisentraut
On 01.03.21 11:12, Michael Banck wrote: postgres@kohn:~$ pg_dump --create-only -p 65432 -d test -h /tmp | egrep -v '^($|--|SET)' SELECT pg_catalog.set_config('search_path', '', false); CREATE DATABASE test WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE = 'de_DE.UTF-8'; ALTER DATABASE test O

Re: Catalog version access

2021-03-03 Thread Peter Eisentraut
On 22.02.21 08:00, Vik Fearing wrote: On 2/22/21 3:24 AM, Andres Freund wrote: Imagine trying to run regular tests of HEAD, where the tests require a large database to be loaded. Re-loading the data for every [few] commits is prohibitively time consuming, and even just running pg_upgrade is pain

Re: Extensibility of the PostgreSQL wire protocol

2021-03-03 Thread Peter Eisentraut
I think, the way the abstractions are chosen in this patch, it is still very much tied to how the libpq protocol works. For example, there is a cancel key and a ready-for-query message. Some of the details of the simple and the extended query are exposed. So you could create a protocol th

Re: Feedback on table expansion hook (including patch)

2021-03-04 Thread Peter Eisentraut
On 07.05.20 10:11, Erik Nordström wrote: I am looking for feedback on the possibility of adding a table expansion hook to PostgreSQL (see attached patch). The motivation for this is to allow extensions to optimize table expansion. In particular, TimescaleDB does its own table expansion in order

Re: WIP: document the hook system

2021-03-04 Thread Peter Eisentraut
On 15.01.21 08:28, Peter Eisentraut wrote: On 2020-12-31 04:28, David Fetter wrote: This could probably use a lot of filling in, but having it in the actual documentation beats needing to know folklore even to know that the capability is there. This patch seems quite short of a state where

Re: pg_amcheck contrib application

2021-03-04 Thread Peter Geoghegan
we know. The user should always see the unvarnished truth. pg_amcheck should not presume to suppress errors from lower level code, except perhaps in well-scoped special cases. -- Peter Geoghegan

Re: 011_crash_recovery.pl intermittently fails

2021-03-04 Thread Peter Geoghegan
.mqmrnpkaqrdtm...@alap3.anarazel.de -- Peter Geoghegan

Re: Increase value of OUTER_VAR

2021-03-06 Thread Peter Eisentraut
On 04.03.21 20:01, Tom Lane wrote: Just as a proof of concept, I tried the attached, and it passes check-world. So if there's anyplace trying to stuff OUTER_VAR and friends into bitmapsets, it's pretty far off the beaten track. The main loose ends that'd have to be settled seem to be: (1) What

Re: [patch] bit XOR aggregate functions

2021-03-06 Thread Peter Eisentraut
On 05.03.21 13:42, Alexey Bashtanov wrote: Thanks for your reviews. I've updated my patch to the current master and added a documentation line suggesting using the new function as a checksum. committed

Re: Enhance traceability of wal_level changes for backup management

2021-03-06 Thread Peter Eisentraut
On 28.01.21 01:44, osumi.takami...@fujitsu.com wrote: (1) writing the time or LSN in the control file to indicate when/where wal_level is changed to 'minimal' from upper level to invalidate the old backups or make alerts to users. I attached the first patch which implementes this idea. It was al

Tablesync early exit

2021-03-06 Thread Peter Smith
-yT5%3D9GDEW84TF%2BA%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-07 Thread Peter Smith
On Sun, Mar 7, 2021 at 3:00 PM Amit Kapila wrote: > > On Sun, Mar 7, 2021 at 7:35 AM Peter Smith wrote: > > > > Please find attached the latest patch set v51* > > > > Few more comments on v51-0006-Fix-apply-worker-empty-prepare: > =

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-07 Thread Peter Smith
On Mon, Mar 8, 2021 at 4:19 PM Amit Kapila wrote: > > On Mon, Mar 8, 2021 at 10:04 AM Peter Smith wrote: > > > > On Sun, Mar 7, 2021 at 3:00 PM Amit Kapila wrote: > > > > > > On Sun, Mar 7, 2021 at 7:35 AM Peter Smith wrote: > > > > >

Re: macOS SIP, next try

2021-03-08 Thread Peter Eisentraut
On 05.03.21 01:36, Tom Lane wrote: Hmm. So I tried this, ie "csrutil enable --without debug" in the recovery system, and after rebooting what I see is $ csrutil status System Integrity Protection status: unknown (Custom Configuration). Configuration: Apple Internal: disabled

Re: authtype parameter in libpq

2021-03-08 Thread Peter Eisentraut
On 04.03.21 16:06, Daniel Gustafsson wrote: authtype is completely dead in terms of reading back the value, to the point of it being a memleak if it indeed was found in as an environment variable. But I tend to think we should remove them both altogether (modulo ABI and API preservation). No

Re: Boundary value check in lazy_tid_reaped()

2021-03-08 Thread Peter Eisentraut
On 21.01.21 14:11, Masahiko Sawada wrote: Agreed. bsearch with bound check showed a reasonable improvement in my evaluation in terms of performance. Regarding memory efficiency, we can experiment with other methods later. I've attached the patch that adds a bound check for encoded itermpointers

Re: pg_upgrade failing for 200+ million Large Objects

2021-03-08 Thread Peter Eisentraut
On 07.03.21 09:43, Tharakan, Robins wrote: Attached is a proof-of-concept patch that allows Postgres to perform pg_upgrade if the instance has Millions of objects. It would be great if someone could take a look and see if this patch is in the right direction. There are some pending tasks (such a

Re: Occasional tablespace.sql failures in check-world -jnn

2021-03-08 Thread Peter Eisentraut
On 09.12.20 08:55, Michael Paquier wrote: It's not clear to me why we have this logic in the makefile at all? Somebody taught pg_regress to do so, but only on windows... See convert_sourcefiles_in(). ... Because we may still introduce this problem again if some new stuff uses src/test/pg_regre

Re: Yet another fast GiST build

2021-03-08 Thread Peter Geoghegan
On Mon, Mar 8, 2021 at 6:41 AM Ibrar Ahmed wrote: > The patch (0001-Add-bool-column-for-LP_DEAF-flag-to-GiST-pageinspect.patch) > does not apply successfully and has multiple hanks failed. That's because it was committed. -- Peter Geoghegan

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-08 Thread Peter Geoghegan
On Tue, Mar 2, 2021 at 6:01 PM Peter Geoghegan wrote: > 1. Any objections to the idea of teaching VACUUM ANALYZE to > distinguish between the cases where VACUUM ran and performed "real > index vacuuming", to make it more intelligent about overwriting > pg_class stats for in

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-08 Thread Peter Geoghegan
this seems defensible now, all things considered. There are other things that are slightly broken but will be fixed by the first patch. But I'm really just worried about these two cases in Postgres 13. Thanks for weighing in -- Peter Geoghegan From 29079ec92eed077c8d4d65e4c3db16ce41578657 Mon S

Re: 64-bit XIDs in deleted nbtree pages

2021-03-08 Thread Peter Geoghegan
ad tuples [1]. I should be able to commit that for Postgres 14. (I will probably finish off my other patch to make nbtree VACUUM recycle pages deleted during the same VACUUM operation last of all.) [1] https://postgr.es/m/CAD21AoAtZb4+HJT_8RoOXvu4HM-Zd4HKS3YSMCH6+-W=bdy...@mail.gmail.com -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-08 Thread Peter Geoghegan
cality of access if allowed to occur at all. So we need to recognize those heap pages where it's possible to preserve a kind of pristine state over time. Heap pages that are subject to constant churn from updates exist -- most apps have some of those. But they're also usually a small minority of all heap pages, even within the same heap relation. -- Peter Geoghegan

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-08 Thread Peter Smith
start using numeric instead for the new feature > getting added? This may or may not become a problem sometime in the future, but I think the feedback is not really specific to the current patch set so I am skipping it at this time. If you want, maybe create it as a separate thread, Is it OK? Kind Regards, Peter Smith. Fujitsu Australia

Re: New IndexAM API controlling index vacuum strategies

2021-03-08 Thread Peter Geoghegan
On Tue, Mar 2, 2021 at 8:49 PM Masahiko Sawada wrote: > On Tue, Mar 2, 2021 at 2:34 PM Peter Geoghegan wrote: > > lazy_vacuum_table_and_indexes() should probably not skip index > > vacuuming when we're close to exceeding the space allocated for the > > LVDeadTuples

Re: Tablesync early exit

2021-03-08 Thread Peter Smith
On Sun, Mar 7, 2021 at 1:33 PM Amit Kapila wrote: > > On Sun, Mar 7, 2021 at 7:26 AM Peter Smith wrote: > > > > Hi hackers. > > > > I propose a small optimization can be added to the tablesync replication > > code. > > > > This proposa

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-09 Thread Peter Smith
e tablesync to achieve SYNCDONE state. See wait_for_relation_state_change(rstate->relid, SUBREL_STATE_SYNCDONE); Now, notice the wait_for_relation_state_change already has CHECK_FOR_INTERRUPTS(); So, AFAIK it isn't necessary to put another CHECK_FOR_INTERRUPTS at the outer loop. Thoughts? -- Kind Regards, Peter Smith. Fujitsu Australia.

Re: SQL-standard function body

2021-03-09 Thread Peter Eisentraut
utility statements. I think it's still useful that way for now. From 29de4ec1ae12d3f9c1f6a31cf626a19b2421ae7a Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Tue, 9 Mar 2021 13:25:39 +0100 Subject: [PATCH v9] SQL-standard function body This adds support for writing CREATE FUNCTION and

Re: authtype parameter in libpq

2021-03-09 Thread Peter Eisentraut
On 08.03.21 10:57, Peter Eisentraut wrote: On 04.03.21 16:06, Daniel Gustafsson wrote: authtype is completely dead in terms of reading back the value, to the point of it being a memleak if it indeed was found in as an environment variable. But I tend to think we should remove them both

Re: Lowering the ever-growing heap->pd_lower

2021-03-09 Thread Peter Geoghegan
ch would imply moving in-use > line pointers); as far as I know there is nothing that relies on only > growing page->pd_lower, and nothing keeping us from shrinking it > whilst holding a pin on the page. Sounds like a good idea to me. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-09 Thread Peter Geoghegan
On Mon, Mar 8, 2021 at 7:34 PM Peter Geoghegan wrote: > > One possible > > consequence that I'm concerned about is sequential scan performance. > > For an index scan, you just jump to the line pointer you want and then > > go get the tuple, but a sequential scan

Re: Lowering the ever-growing heap->pd_lower

2021-03-09 Thread Peter Geoghegan
Rather, it represents that the bytes in question are considered safe to copy around but not safe to rely on being any particular value. So Valgrind will complain if the bytes in question influence control flow, directly or indirectly. Obviously the code should also be audited. Even then, there may still be bugs. I think that we need to bite the bullet here -- line pointer bloat is a significant problem. -- Peter Geoghegan

Re: Lowering the ever-growing heap->pd_lower

2021-03-09 Thread Peter Geoghegan
On Tue, Mar 9, 2021 at 1:54 PM Peter Geoghegan wrote: > It occurs to me that we should also mark the hole in the middle of the > page (which includes the would-be LP_UNUSED line pointers at the end > of the original line pointer array space) as undefined to Valgrind

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-09 Thread Peter Geoghegan
This approach is unorthodox, but it has a kind of a precedent -- the recheck_on_update storage param was disabled on the Postgres 11 branch by commit 5d28c9bd. More importantly, it just happens to make sense, given the specifics here. -- Peter Geoghegan v2-0002-VACUUM-ANALYZE-Always-update-pg_clas

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-09 Thread Peter Smith
ther streaming spool file code does - e.g. notice apply_handle_stream_commit function simply closes its own fd using BufFileClose; it doesn’t delegate stream_close_file() to do it. -- Kind Regards, Peter Smith. Fujitsu Australia

Re: Boundary value check in lazy_tid_reaped()

2021-03-10 Thread Peter Eisentraut
On 10.03.21 02:29, Masahiko Sawada wrote: There is no noticeable effect of inlining lazy_tid_reaped(). So it would be better to not do that. Attached the patch that doesn't inline lazy_tid_reaped(). committed

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-10 Thread Peter Geoghegan
On Tue, Mar 9, 2021 at 7:42 PM Peter Geoghegan wrote: > My current plan is to commit everything within the next day or two. > This includes backpatching to Postgres 13 only. Pushed, thanks. -- Peter Geoghegan

Re: 64-bit XIDs in deleted nbtree pages

2021-03-10 Thread Peter Geoghegan
On Mon, Mar 1, 2021 at 7:25 PM Peter Geoghegan wrote: > Attached is v8 of the patch series, which has new patches. No real > changes compared to v7 for the first patch, though. Here is another bitrot-fix-only revision, v9. Just the recycling patch again. I'll commit this when we get

Re: [PATCH] Identify LWLocks in tracepoints

2021-03-10 Thread Peter Eisentraut
On 10.03.21 06:38, Craig Ringer wrote: On Wed, 3 Mar 2021 at 20:50, David Steele <mailto:da...@pgmasters.net>> wrote: On 1/22/21 6:02 AM, Peter Eisentraut wrote: This patch set no longer applies: http://cfbot.cputube.org/patch_32_2927.log <http://cfbot.cputube.org/p

Re: pg_amcheck contrib application

2021-03-11 Thread Peter Eisentraut
(I'm still not a fan of adding more client-side tools whose sole task is to execute server-side functionality in a slightly filtered way, but it seems people are really interested in this, so ...) I want to register, if we are going to add this, it ought to be in src/bin/. If we think it's a

Re: OpenSSL 3.0.0 compatibility

2021-03-11 Thread Peter Eisentraut
On 10.03.21 09:23, Daniel Gustafsson wrote: On 3 Mar 2021, at 14:55, Peter Eisentraut wrote: This thread is still in the commit fest, but I don't see any actual proposed patch still pending. Most of the activity has moved into other threads. The doc changes in the patch proposed on

Re: cursor sensitivity misunderstanding

2021-03-11 Thread Peter Eisentraut
On 09.03.21 00:22, David G. Johnston wrote: I came up with the attached patch to sort this out a bit.  It does not change any cursor behavior.  But the documentation now uses the terms more correctly and explains the differences between SQL and the PostgreSQL implementation better

Re: OpenSSL 3.0.0 compatibility

2021-03-11 Thread Peter Eisentraut
27;), '0123456', 'abcd', 'des'); - decrypt_iv - - foo +decrypt_iv +-- + \177\177\177\177\177\177\177\177 (1 row) The attached patch appears to address these cases. From 1b9cf580e9e441806def681eea71ce6fd2228206 Mo

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Peter Geoghegan
therefore much easier to model. I hope to do more of that. [1] https://github.com/wieck/benchmarksql/blob/29b62435dc5c9eaf178983b43818fcbba82d4286/run/sql.postgres/extraCommandsBeforeLoad.sql#L1 -- Peter Geoghegan

Re: OpenSSL 3.0.0 compatibility

2021-03-11 Thread Peter Eisentraut
On 11.03.21 11:41, Daniel Gustafsson wrote: .. and apply the padding changes as proposed in a patch upthread like this (these work for all OpenSSL versions I've tested, and I'm rather more puzzled as to why we got away with not having them in the past): Yes, before proceeding with this, we s

Re: Enhance traceability of wal_level changes for backup management

2021-03-12 Thread Peter Eisentraut
On 08.03.21 03:45, osumi.takami...@fujitsu.com wrote: OK. The basic idea is to enable backup management tools to recognize wal_level drop between*snapshots*. When you have a snapshot of the cluster at one time and another one at different time, with this new parameter, you can see if anything tha

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-12 Thread Peter Smith
text applies to the code above or below it) TIA. Kind Regards, Peter Smith. Fujitsu Australia

Re: pg_amcheck contrib application

2021-03-12 Thread Peter Geoghegan
On Fri, Mar 12, 2021 at 10:10 AM Robert Haas wrote: > Committed that way with some small adjustments. Let's see what the > buildfarm thinks. Thank you both, Mark and Robert. This is excellent work! -- Peter Geoghegan

Re: pg_amcheck contrib application

2021-03-12 Thread Peter Geoghegan
On Fri, Mar 12, 2021 at 10:32 AM Peter Geoghegan wrote: > Thank you both, Mark and Robert. This is excellent work! FYI I see these compiler warnings just now: pg_amcheck.c:1653:4: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement] 1653 |DatabaseInfo *

Re: pg_amcheck contrib application

2021-03-12 Thread Peter Geoghegan
ster branch (plus all new major version branches going forward). That would give us some degree of amcheck test coverage in the back branches right away. It might even detect cross-version inconsistencies. Or even pg_upgrade bugs. -- Peter Geoghegan

Re: amcheck hardening

2021-03-13 Thread Peter Geoghegan
de has the opportunity to run. Higher level code is always going to be at risk of making assumptions about the data not being corrupt, because there is so much more of it, and also because it tends to roughly look like idiomatic AM code. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-13 Thread Peter Geoghegan
xes() calls. If at that point we decide to do index vacuuming throughout the entire vacuum operation, then we will not allow the table to accumulate many more TIDs than we can expect to fit in an entire maintenance_work_mem space. Under this scheme, the "maintenance_work_mem threshold" can be thought of as an implicit thing (it's not a constant/multiplier or anything) -- it is >= 100% of maintenance_work_mem, in effect. -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-14 Thread Peter Geoghegan
On Sat, Mar 13, 2021 at 7:23 PM Peter Geoghegan wrote: > In other words, I am not worried about debt, exactly. Debt is normal > in moderation. Healthy, even. I am worried about bankruptcy, perhaps > following a rare and extreme event. It's okay to be imprecise, but all > of the

Re: New IndexAM API controlling index vacuum strategies

2021-03-14 Thread Peter Geoghegan
he way.) [1] https://postgr.es/m/CAD21AoAtZb4+HJT_8RoOXvu4HM-Zd4HKS3YSMCH6+-W=bdy...@mail.gmail.com [2] https://postgr.es/m/23885.1555357618%40sss.pgh.pa.us [3] https://postgr.es/m/20130108024957.GA4751%40tornado.leadboat.com [4] https://postgr.es/m/16814.1555348381%40sss.pgh.pa.us [5] https://postgr.es/m/CAH2-Wznpywm4qparkQxf2ns3c7BugC4x7VzKjiB8OCYswwu-=g...@mail.gmail.com -- Peter Geoghegan v2-0001-Skip-index-vacuuming-dynamically.patch Description: Binary data

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-14 Thread Peter Geoghegan
' -- that change to the SQL will make the query just test indexes from the public schema. Do "SET client_min_messages=DEBUG1 " to get a kind of rudimentary progress indicator, if that seems useful to you. The docs have further information on what this bt_index_parent_check function

Re: recovering from "found xmin ... from before relfrozenxid ..."

2021-03-14 Thread Peter Geoghegan
P2_CLEANUP_INFO record for recovery conflicts can go -- we can get by with only XLOG_HEAP2_CLEAN records from pruning, IIRC. Have I missed something? The special case in question seems pretty awful to me, so I have to wonder why somebody else didn't remove it long ago... -- Peter Geoghegan

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-15 Thread Peter Smith
ssert(sizeof(hentry->name) == szpath) ' will be better. > Thanks for your feedback comment. But today Amit suggested [ak0315] that the current psf logic should all be replaced, after which the function you commented about will no longer exist. [ak0315] https://www.postgresql.org/message-id/CAA4eK1LVEdPYnjdajYzu3k6KEii1%2BF0jdQ6sWnYugiHcSGZD6Q%40mail.gmail.com Kind Regards, Peter Smith. Fujitsu Australia

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Peter Geoghegan
ant violated for > index "some_other_index" That indicates corruption. Can you tie this back to the crash? Is it the same index? -- Peter Geoghegan

Re: New IndexAM API controlling index vacuum strategies

2021-03-15 Thread Peter Geoghegan
determined that it is not HEAPTUPLE_DEAD * (else we should be removing the tuple, not freezing it). Does that need work too? > See > https://postgr.es/m/20200724165514.dnu5hr4vvgkssf5p%40alap3.anarazel.de > for some discussion around the fragility. That's a good reference, thanks. [1] https://www.postgresql.org/message-id/20130130020456.GE3524%40tornado.leadboat.com -- Peter Geoghegan

Re: Support tab completion for upper character inputs in psql

2021-03-15 Thread Peter Eisentraut
On 09.02.21 15:48, Tang, Haiying wrote: I'm still confused about the APPROPRIATE behavior of tab completion. It seems ALTER table/tablespace SET/RESET is already case-insensitive. For example # alter tablespace dbspace set(e[tab] # alter tablespace dbspace set(effective_io_concurrency # alter

Re: New IndexAM API controlling index vacuum strategies

2021-03-15 Thread Peter Geoghegan
On Mon, Mar 15, 2021 at 12:58 PM Peter Geoghegan wrote: > > I'm not comfortable with this change without adding more safety > > checks. If there's ever a case in which the HEAPTUPLE_DEAD case is hit > > and the xid needs to be frozen, we'll either cause erro

pg_subscription - substream column?

2021-03-15 Thread Peter Smith
ommit/464824323e57dc4b397e8b05854d779908b55304 Kind Regards, Peter Smith. Fujitsu Australia

Re: New IndexAM API controlling index vacuum strategies

2021-03-15 Thread Peter Geoghegan
t I'd not bet > my (nonexistant) farm on it. Well if we can solve the problem by simply doing pruning once again in the event of a HEAPTUPLE_DEAD return value from the lazy_scan_heap() HTSV call, then the comment becomes 100% true (which is not the case even today). -- Peter Geoghegan

Re: pg_subscription - substream column?

2021-03-16 Thread Peter Smith
On Tue, Mar 16, 2021 at 7:20 PM Amit Kapila wrote: > > On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't > > have any mention of the substream column. > > > > Accidental omis

Re: dynamic result sets support in extended query protocol

2021-03-16 Thread Peter Eisentraut
On 15.03.21 14:56, David Steele wrote: Hi Peter, On 12/30/20 9:33 AM, Peter Eisentraut wrote: On 2020-10-09 20:46, Andres Freund wrote: Is there really a good reason for forcing the client to issue NextResult, Describe, Execute for each of the dynamic result sets? It's not like th

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
ge the on-disk representation to not have any posting list tuples from deduplication): alter index idx_id_mtime set (deduplicate_items = off); reindex index idx_id_mtime; -- Peter Geoghegan

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
base cannot crash even with corrupt data, but we do try to avoid it whenever possible. I may be able to harden _bt_swap_posting(), to make failures like this a little more friendly. It's an infrequently hit code path, so we can easily afford to make the code more careful/less trusting. -- Peter Geoghegan

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:08 AM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would

<    1   2   3   4   5   6   7   8   9   10   >