Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-03-27 Thread Amit Kapila
On Wed, Mar 4, 2020 at 9:14 AM Dilip Kumar wrote: > > On Wed, Mar 4, 2020 at 3:16 AM Tomas Vondra > wrote: > > > > > > The first thing I realized that WAL-logging of assignments in v12 does > > both the "old" logging (using dedicated message) and "new" with > > toplevel-XID embedded in the first

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-27 Thread David Rowley
On Sat, 28 Mar 2020 at 17:12, Laurenz Albe wrote: > In the light of that, I have no objections. Thank you. Pushed. David

proposal - psql output file write mode

2020-03-27 Thread Pavel Stehule
Hi I am playing with pspg and inotify support. It is working pretty well and now can be nice if forwarding to output file can be configured little bit more. Now, only append mode is supported. But append mode doesn't work with pspg pager. So I propose new pset option "file_output_mode" with two po

Re: error context for vacuum to include block number

2020-03-27 Thread Amit Kapila
On Sat, Mar 28, 2020 at 7:04 AM Justin Pryzby wrote: > > On Sat, Mar 28, 2020 at 06:59:10AM +0530, Amit Kapila wrote: > > On Sat, Mar 28, 2020 at 6:46 AM Justin Pryzby wrote: > > > > > > On Sat, Mar 28, 2020 at 06:28:38AM +0530, Amit Kapila wrote: > > > > > Hm, but I caused a crash *without* addi

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-27 Thread Laurenz Albe
On Sat, 2020-03-28 at 11:59 +1300, David Rowley wrote: > On Fri, 27 Mar 2020 at 22:40, Laurenz Albe wrote: > > The new meaning of -2 should be documented, other than that it looks > > good to me. > > But the users don't need to know anything about -2. It's not possible > to explicitly set the val

Re: Online checksums verification in the backend

2020-03-27 Thread Masahiko Sawada
On Wed, 18 Mar 2020 at 19:11, Julien Rouhaud wrote: > > On Wed, Mar 18, 2020 at 07:06:19AM +0100, Julien Rouhaud wrote: > > On Wed, Mar 18, 2020 at 01:20:47PM +0900, Michael Paquier wrote: > > > On Mon, Mar 16, 2020 at 09:21:22AM +0100, Julien Rouhaud wrote: > > > > On Mon, Mar 16, 2020 at 12:29:2

Re: [PATCH] Check operator when creating unique index on partition table

2020-03-27 Thread Guancheng Luo
> On Mar 26, 2020, at 01:00, Tom Lane wrote: > > Guancheng Luo writes: >> I found that things could go wrong in some cases, when the unique index and >> the partition key use different opclass. > > I agree that this is an oversight, but it seems like your solution is > overcomplicated and pro

Re: [PATCH] Btree BackwardScan race condition on Standby during VACUUM

2020-03-27 Thread Peter Geoghegan
On Mon, Mar 16, 2020 at 7:08 AM Michail Nikolaev wrote: > I was sure I have broken something in btree and spent a lot of time > trying to figure what. > And later... I realized what it is bug in btree since a very old times... > Because of much faster scans with LP_DEAD support on a standby it >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-27 Thread Tomas Vondra
On Fri, Mar 27, 2020 at 09:36:55PM -0400, James Coleman wrote: On Fri, Mar 27, 2020 at 9:19 PM Tomas Vondra wrote: On Fri, Mar 27, 2020 at 12:51:34PM -0400, James Coleman wrote: >In a previous email I'd summarized remaining TODOs I'd found. Here's >an updated listed with several resolved. > >R

Re: Make mesage at end-of-recovery less scary.

2020-03-27 Thread James Coleman
On Thu, Mar 26, 2020 at 12:41 PM Robert Haas wrote: > > On Wed, Mar 25, 2020 at 8:53 AM Peter Eisentraut > wrote: > > HINT: This is to be expected if this is the end of the WAL. Otherwise, > > it could indicate corruption. > > First, I agree that this general issue is a problem, because it's co

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Tom Lane wrote: > That being the case, I'd think a better design principle is "make your > new code look like the code around it", which would tend to weigh against > introducing StringInfo uses into pgbench when there's none there now and > a bunch of PQExpBuffer instead. So I ca

Re: [PATCH] Btree BackwardScan race condition on Standby during VACUUM

2020-03-27 Thread Peter Geoghegan
On Fri, Mar 27, 2020 at 8:58 AM Michail Nikolaev wrote: > I have spent some time trying to find any possible race condition > between btree_xlog_split and _bt_walk_left… But I can’t find any. > Also, I have tried to cause any issue by putting pg_sleep put into > btree_xlog_split (between releasing

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Sat, Mar 28, 2020 at 06:59:10AM +0530, Amit Kapila wrote: > On Sat, Mar 28, 2020 at 6:46 AM Justin Pryzby wrote: > > > > On Sat, Mar 28, 2020 at 06:28:38AM +0530, Amit Kapila wrote: > > > > Hm, but I caused a crash *without* adding CHECK_FOR_INTERRUPTS, just > > > > kill+sleep. The kill() coul

Re: error context for vacuum to include block number

2020-03-27 Thread Amit Kapila
On Sat, Mar 28, 2020 at 6:46 AM Justin Pryzby wrote: > > On Sat, Mar 28, 2020 at 06:28:38AM +0530, Amit Kapila wrote: > > > Hm, but I caused a crash *without* adding CHECK_FOR_INTERRUPTS, just > > > kill+sleep. The kill() could come from running pg_cancel_backend(). And > > > the > > > sleep()

Re: improve transparency of bitmap-only heap scans

2020-03-27 Thread Amit Kapila
On Wed, Mar 25, 2020 at 5:44 PM James Coleman wrote: > > On Tue, Mar 24, 2020 at 11:02 PM Amit Kapila wrote: > > > > On Wed, Mar 25, 2020 at 12:44 AM Tom Lane wrote: > > > > > > I took a quick look through this patch. While I see nothing to complain > > > about implementation-wise, I'm a bit be

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-27 Thread Tomas Vondra
On Fri, Mar 27, 2020 at 12:51:34PM -0400, James Coleman wrote: In a previous email I'd summarized remaining TODOs I'd found. Here's an updated listed with several resolved. Resolved: 2. Not marked in the patch, but in nodeIncrementalSort.c ExecIncrementalSort() I wonder if perhaps we should mov

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Sat, Mar 28, 2020 at 06:28:38AM +0530, Amit Kapila wrote: > > Hm, but I caused a crash *without* adding CHECK_FOR_INTERRUPTS, just > > kill+sleep. The kill() could come from running pg_cancel_backend(). And > > the > > sleep() just encourages a context switch, which can happen at any time. >

Re: error context for vacuum to include block number

2020-03-27 Thread Amit Kapila
On Sat, Mar 28, 2020 at 12:34 AM Justin Pryzby wrote: > > On Fri, Mar 27, 2020 at 11:50:30AM +0530, Amit Kapila wrote: > > > > The crash scenario I'm trying to avoid would be like statement_timeout > > > > or other > > > > asynchronous event occurring between two non-atomic operations. > > > > >

Re: SLRU statistics

2020-03-27 Thread Tomas Vondra
Hi, here is a bit improved version of the patch - I've been annoyed by how the resetting works (per-entry timestamp, but resetting all entries) so I've added a new function pg_stat_reset_slru() that allows resetting either all entries or just one entry (identified by name). So SELECT pg_stat

Re: Make MemoryContextMemAllocated() more precise

2020-03-27 Thread Jeff Davis
On Wed, 2020-03-18 at 15:41 -0700, Jeff Davis wrote: > In an off-list discussion, Andres suggested that MemoryContextStats > could be refactored to achieve this purpose, perhaps with flags to > avoid walking through the blocks and freelists when those are not > needed. Attached refactoring patch.

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-27 Thread Justin Pryzby
On Thu, Mar 26, 2020 at 11:01:06PM -0500, Justin Pryzby wrote: > > Another issue is this: > > > +VACUUM ( FULL [, ...] ) [ TABLESPACE > > class="parameter">new_tablespace ] [ > > class="parameter">table_and_columns [, ...] ] > > As you mentioned in your v1 patch, in the other cases, "tablespace >

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread Tom Lane
Fabien COELHO writes: >>> Ok. I find it strange to mix PQExpBuffer & StringInfo in the same file. >> Agreed, but we'd rather use StringInfo going forward. However, I don't >> think >> that puts you on the hook for updating all the PQExpBuffer references. >> Unless you want to... > I cannot sa

Re: Possible copy and past error? (\usr\backend\commands\analyze.c)

2020-03-27 Thread Tom Lane
Ranier Vilela writes: > Can someone check if there is a copy and paste error, at file: > \usr\backend\commands\analyze.c, at lines 2225 and 2226? > int num_mcv = stats->attr->attstattarget; > int num_bins = stats->attr->attstattarget; No, that's intentional I believe. Those are independent varia

Re: allow online change primary_conninfo

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Alvaro Herrera wrote: > On 2020-Mar-27, Alvaro Herrera wrote: > > > I pushed the wal_receiver_create_temp_slot bugfix, because I realized > > after looking for long enough at WalReceiverMain() that the code was > > beyond saving. I'll be pushing the rest of this later today. > >

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2020-03-27 Thread Tom Lane
Fabien COELHO writes: >> It does not look like the remainder of this patch is going to be committed >> and I don't think it makes sense to keep moving the patch indefinitely. >> Unless something changes by the end of this CF I'll mark it Returned With >> Feedback. > I'd be rather unclear about

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-27 Thread David Rowley
On Fri, 27 Mar 2020 at 22:40, Laurenz Albe wrote: > The new meaning of -2 should be documented, other than that it looks > good to me. But the users don't need to know anything about -2. It's not possible to explicitly set the value to -2. This is just the reset value of the reloption which means

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread Fabien COELHO
Ok. I find it strange to mix PQExpBuffer & StringInfo in the same file. Agreed, but we'd rather use StringInfo going forward. However, I don't think that puts you on the hook for updating all the PQExpBuffer references. Unless you want to... I cannot say that I "want" to fix something w

Re: psql FETCH_COUNT feature does not work with combined queries

2020-03-27 Thread Fabien COELHO
Assuming there's no one willing to fix the behavior (and that seems unlikely given we're in the middle of a 2020-01 commitfest) it makes sense to at least document the behavior. That being said, I think the proposed patch may be a tad too brief. I don't think we need to describe the exact behav

Re: fix for BUG #3720: wrong results at using ltree

2020-03-27 Thread Nikita Glukhov
On 24.01.2020 21:29, Tomas Vondra wrote: Hi Nikita, This patch seems inactive / stuck in "waiting on author" since November. It's marked as bugfix, so it'd be good to get it committed instead of just punting it to the next CF. I did a quick review, and I came mostly with the same two complaint

Re: pgbench - rework variable management

2020-03-27 Thread David Steele
On 3/27/20 6:25 PM, Fabien COELHO wrote: Patch v5 is a rebase with some adjustements. This patch is failing on the Windows build: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.85698 I'm not sure if this had been fixed in v3 and this is a new issue or if it has bee

Re: backup manifests

2020-03-27 Thread Bruce Momjian
On Fri, Mar 27, 2020 at 06:38:33PM -0400, Stephen Frost wrote: > Greetings, > > On Fri, Mar 27, 2020 at 18:36 Bruce Momjian wrote: > > On Thu, Mar 26, 2020 at 12:34:52PM -0400, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > This is where I feel like I'm t

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, On Fri, Mar 27, 2020 at 18:36 Bruce Momjian wrote: > On Thu, Mar 26, 2020 at 12:34:52PM -0400, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > This is where I feel like I'm trying to make decisions in a vacuum. If > > > we had a few more people weighing in

Re: backup manifests

2020-03-27 Thread Bruce Momjian
On Thu, Mar 26, 2020 at 12:34:52PM -0400, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > This is where I feel like I'm trying to make decisions in a vacuum. If > > we had a few more people weighing in on the thread on this point, I'd > > be happy to go with whatever the con

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 6:07 PM, Andres Freund wrote: Hi, On 2020-03-27 16:57:46 -0400, Stephen Frost wrote: I really don't know what to say to this. WAL is absolutely critical to a backup being valid. pgBackRest doesn't have a way to *just* validate a backup today, unfortunately, but we're planning to su

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2020-03-27 Thread Fabien COELHO
This patch is registered in 2020-01, but the last message in the thread seems to be from 2019/05/23. The patch itself seems to be OK (it applies fine etc.) What do we need to get it over the line, instead of just moving it to the next one CF over and over? It does not look like the remainder

Re: Conflict handling for COPY FROM

2020-03-27 Thread Tom Lane
Surafel Temesgen writes: > [ conflict-handling-copy-from-v16.patch ] I took a quick look at this patch, since it was marked "ready for committer", but I don't see how it can possibly be considered committable. 1. Covering only the errors that are thrown in DoCopy itself doesn't seem to me to pas

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread David Steele
On 3/27/20 6:13 PM, Fabien COELHO wrote: Hello David, I'd prefer not to expand the use of pqexpbuffer in more places, and instead rather see this use StringInfo, now that's also available to frontend programs. Franckly, one or the other does not matter much to me. FWIW, I agree with Andre

Re: pgbench - rework variable management

2020-03-27 Thread Fabien COELHO
Patch v5 is a rebase with some adjustements. This patch is failing on the Windows build: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.85698 I'm not sure if this had been fixed in v3 and this is a new issue or if it has been failing all along. Either way, it should b

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 16:57:46 -0400, Stephen Frost wrote: > > I really don't know what to say to this. WAL is absolutely critical to > > a backup being valid. pgBackRest doesn't have a way to *just* validate > > a backup today, unfortunately, but

Re: backup manifests

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Stephen Frost wrote: > I don't think we should just compare all permissions or ownership with > some arbitrary idea of what we think they should be, even though if you > use pg_basebackup in 'plain' format, you actually end up with > differences, today, from what the source system

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread Fabien COELHO
Hello David, I'd prefer not to expand the use of pqexpbuffer in more places, and instead rather see this use StringInfo, now that's also available to frontend programs. Franckly, one or the other does not matter much to me. FWIW, I agree with Andres with regard to using StringInfo. Ok. I

Possible copy and past error? (\usr\backend\commands\analyze.c)

2020-03-27 Thread Ranier Vilela
Hi, Can someone check if there is a copy and paste error, at file: \usr\backend\commands\analyze.c, at lines 2225 and 2226? int num_mcv = stats->attr->attstattarget; int num_bins = stats->attr->attstattarget; If they really are the same values, it could be changed to: int num_mcv = stats->attr->

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 17:44:07 -0400, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > > > > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund > > > > wrote: > > > > > Hm. Sh

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 16:57:46 -0400, Stephen Frost wrote: > I really don't know what to say to this. WAL is absolutely critical to > a backup being valid. pgBackRest doesn't have a way to *just* validate > a backup today, unfortunately, but we're planning to support it in the > future and we will a

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 17:07:42 -0400, Stephen Frost wrote: > I had suggested up-thread, and I'm still fine with, having > pg_validatebackup scan the WAL and check the internal checksums. I'd > prefer an option that uses hashes to check when the user has asked for > hashes with SHA256 or something, bu

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 17:44:07 -0400, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > > > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > > > > Hm. Should this warn if the directory's permissions are set too openly > > >

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > > > Hm. Should this warn if the directory's permissions are set too openly > > > (world writable?)? > > > > I don't think so, but i

Re: Internal key management system

2020-03-27 Thread Bruce Momjian
On Wed, Mar 25, 2020 at 05:51:08PM +0900, Masahiko Sawada wrote: > On Tue, 24 Mar 2020 at 23:15, Bruce Momjian wrote: > > > > On Tue, Mar 24, 2020 at 02:29:57PM +0900, Masahiko Sawada wrote: > > > That seems to work fine. > > > > > > So we will have pg_cryptokeys within PGDATA and each key is stor

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2020-03-27 14:34:19 -0400, Robert Haas wrote: > > I think #2 is an interesting idea and could possibly reduce the danger > > of user confusion on this point considerably - because, let's face it, > > not everyone is going to read the docu

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: > > > Seems better to (later?) add support for generating manifests for WAL > > > files, and then have a tool that can verify all the manifests required > > > to restore a base backup.

Re: allow online change primary_conninfo

2020-03-27 Thread Sergei Kornilov
Hello Thank you! > I think I should set aside your new draft for now I agree, this patch definitely needs a bit more time to review. (currently it applies on top of v13 patch cleanly) > but I think we should still get it in pg13 to avoid having the change the > semantics of the > walreceiver

Re: pgsql: Provide a TLS init hook

2020-03-27 Thread Tom Lane
Andrew Dunstan writes: > On 3/27/20 11:09 AM, Tom Lane wrote: >> It seems like what the buildfarm would like is a way to invoke TAP tests >> and traditional-framework tests separately, so that it could apply special >> tooling to the former. I'd have no objection to making that possible. > Exact

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:59:56 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-03-27 14:34:56 -0400, Tom Lane wrote: > >> Andres Freund writes: > >>> Tom, while imo not a fix of the right magnitude here: Are you planning / > >>> hoping to work again on your postmaster latch patch? > > >>

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 3:55 PM, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: I think that what we have seen so far is that all of the SHA-n algorithms that PostgreSQL supports are about equally slow, so it doesn't really matter which one you pick there from a performance point of view.

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 15:20:27 -0400, Robert Haas wrote: > On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > > Are you planning to include a specification of the manifest file format > > anywhere? I looked through the patches and didn't find anything. > > I thought about that. I think it would

Re: some AppVeyor files

2020-03-27 Thread Mike Palmiotto
On Tue, Mar 24, 2020 at 10:27 PM Thomas Munro wrote: > Thanks! I added a link to this thread to a Wiki page that tries to > collect information on this topic[1]. Another thing you could be > interested in is the ability to test on several different MSVC > versions (I tried to find some appveyor.

Re: proposal \gcsv

2020-03-27 Thread Pavel Stehule
čt 26. 3. 2020 v 19:41 odesílatel Erik Rijkers napsal: > On 2020-03-26 18:49, Pavel Stehule wrote: > > Hi > > > > [psql-gfmt.patch] > > This seems useful and works well; I haven't found any errors. Well done. > > However, I have a suggestion that is perhaps slightly outside of this > patch but fu

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-27 Thread Justin Pryzby
On Thu, Mar 26, 2020 at 11:01:06PM -0500, Justin Pryzby wrote: > > Another issue is this: > > > +VACUUM ( FULL [, ...] ) [ TABLESPACE > > class="parameter">new_tablespace ] [ > > class="parameter">table_and_columns [, ...] ] > > As you mentioned in your v1 patch, in the other cases, "tablespace

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 3:29 PM, Robert Haas wrote: On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: Seems better to (later?) add support for generating manifests for WAL files, and then have a tool that can verify all the manifests required to restore a base backup. I'm not trying to expand on the f

Re: Add A Glossary

2020-03-27 Thread Justin Pryzby
On Fri, Mar 20, 2020 at 11:32:25PM +0100, Jürgen Purtz wrote: > > > +Archiver > > Can you change that to archiver process ? > > I prefer the short term without the addition of 'process' - concerning > 'Archiver' as well as the other cases. But I'm not an native English > speaker. I didn't lik

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:34:19 -0400, Robert Haas wrote: > I think #2 is an interesting idea and could possibly reduce the danger > of user confusion on this point considerably - because, let's face it, > not everyone is going to read the documentation. However, I'm having a > hard time figuring out e

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 15:29:02 -0400, Robert Haas wrote: > On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: > > > Seems better to (later?) add support for generating manifests for WAL > > > files, and then have a tool that can verify all the manifests required > > > to restore a base backup. > >

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 3:20 PM, Robert Haas wrote: On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: Hm. Is it a great choice to include the checksum for the manifest inside the manifest itself? With a cryptographic checksum it seems like it could make a ton of sense to store the checksum somewhere "sa

Re: backup manifests

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:13:17 -0400, Robert Haas wrote: > On Thu, Mar 26, 2020 at 4:44 PM Stephen Frost wrote: > > > I mean, the property that I care about is the one where it detects > > > better than 999,999,999 errors out of every 1,000,000,000, regardless > > > of input length. > > > > Throwing

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Mar 26, 2020 at 4:44 PM Stephen Frost wrote: > > Is it actually possible, today, in PG, to have a 4GB WAL record? > > Judging this based on the WAL record size doesn't seem quite right. > > I'm not sure. I mean, most records are q

Re: allow online change primary_conninfo

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Alvaro Herrera wrote: > I pushed the wal_receiver_create_temp_slot bugfix, because I realized > after looking for long enough at WalReceiverMain() that the code was > beyond saving. I'll be pushing the rest of this later today. So here's the next one. I still have to go over the

Re: backup manifests

2020-03-27 Thread David Steele
On 3/27/20 1:53 PM, Robert Haas wrote: On Thu, Mar 26, 2020 at 4:37 PM David Steele wrote: I know you and Stephen have agreed on a number of doc changes, would it be possible to get a new patch with those included? I finally have time to do a review of this tomorrow. I saw some mistakes in the

Re: backup manifests

2020-03-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Mar 26, 2020 at 4:37 PM David Steele wrote: > > I agree with Stephen that this should be done, but I agree with you that > > it can wait for a future commit. However, I do think: > > > > 1) It should be called out rather plainly in

Re: pgsql: Provide a TLS init hook

2020-03-27 Thread Andrew Dunstan
On 3/27/20 11:09 AM, Tom Lane wrote: > >> Right now the explicit TAP test code in the buildfarm knows how to collect >> all the relevant output. The installcheck code doesn't know about that for >> TAP tests. > It seems like what the buildfarm would like is a way to invoke TAP tests > and trad

Re: backup manifests

2020-03-27 Thread Robert Haas
On Fri, Mar 27, 2020 at 11:26 AM Stephen Frost wrote: > > Seems better to (later?) add support for generating manifests for WAL > > files, and then have a tool that can verify all the manifests required > > to restore a base backup. > > I'm not trying to expand on the feature set here or move the

Re: BUG #16040: PL/PGSQL RETURN QUERY statement never uses a parallel plan

2020-03-27 Thread Tom Lane
Hamid Akhtar writes: > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed > Implements feature: tested, passed > Spec compliant: tested, passed > Documentation:tested, passed > All good with this patch.

Re: allow online change primary_conninfo

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Sergei Kornilov wrote: > Hello > > > I realized that the reason the tests broke after Sergei's patch is that > > recovery/t/001_stream_rep.pl's get_slot_xmins() is broken for temp > > walreceiver slots, since it's using the non-temp name it tries to give > > to the slot, rather th

Re: WAL usage calculation patch

2020-03-27 Thread Julien Rouhaud
On Fri, Mar 27, 2020 at 8:21 PM Kirill Bychik wrote: > > > > > I'm attaching a v5 with fp records only for temp tables, so there's > > > > no risk of > > > > instability. As I previously said I'm fine with your two patches, > > > > so unless > > > > you have objections on th

Re: WAL usage calculation patch

2020-03-27 Thread Kirill Bychik
> > > I'm attaching a v5 with fp records only for temp tables, so there's > > > no risk of > > > instability. As I previously said I'm fine with your two patches, so > > > unless > > > you have objections on the fpi test for temp tables or the > > > documentation > > >>>

Re: A bug when use get_bit() function for a long bytea string

2020-03-27 Thread Tom Lane
"Daniel Verite" writes: > So aside from the integer overflow bug, isn't there the issue that the > "offset" argument of get_bit() and set_bit() should have been an > int8 in the first place? Good point, but a fix for that wouldn't be back-patchable. It does suggest that we should just make all t

Re: backup manifests

2020-03-27 Thread Robert Haas
On Fri, Mar 27, 2020 at 2:29 AM Andres Freund wrote: > s/ye'/yes/ Ugh, sorry. Fixed in the version posted earlier. > Are you planning to include a specification of the manifest file format > anywhere? I looked through the patches and didn't find anything. I thought about that. I think it would

Re: Improve handling of parameter differences in physical replication

2020-03-27 Thread Sergei Kornilov
Hello > I think we can set wait event WAIT_EVENT_RECOVERY_PAUSE here. +1, since we added this in recoveryPausesHere. PS: do we need to add a prototype for the RecoveryRequiredIntParameter function in top of xlog.c? regards, Sergei

Re: error context for vacuum to include block number

2020-03-27 Thread Justin Pryzby
On Fri, Mar 27, 2020 at 11:50:30AM +0530, Amit Kapila wrote: > > > The crash scenario I'm trying to avoid would be like statement_timeout or > > > other > > > asynchronous event occurring between two non-atomic operations. > > > > > +if (errinfo->phase==VACUUM_ERRCB_PHASE_VACUUM_INDEX && > > erri

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Tom Lane
Andres Freund writes: > On 2020-03-27 14:34:56 -0400, Tom Lane wrote: >> Andres Freund writes: >>> Tom, while imo not a fix of the right magnitude here: Are you planning / >>> hoping to work again on your postmaster latch patch? >> Um ... -ESWAPPEDOUT. What are you thinking of? > https://postg

Re: backup manifests

2020-03-27 Thread Robert Haas
On Thu, Mar 26, 2020 at 4:44 PM Stephen Frost wrote: > Is it actually possible, today, in PG, to have a 4GB WAL record? > Judging this based on the WAL record size doesn't seem quite right. I'm not sure. I mean, most records are quite small, but I think if you set REPLICA IDENTITY FULL on a table

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Andres Freund
Hi, On 2020-03-27 14:34:56 -0400, Tom Lane wrote: > Andres Freund writes: > > I've locally fixed the issue by computing the stack base address anew > > for postmaster children. Currently in InitPostmasterChild(). > > > I'd like to get that change upstream. The rr hackers have fixed a number > > o

Re: backup manifests

2020-03-27 Thread Robert Haas
On Thu, Mar 26, 2020 at 4:37 PM David Steele wrote: > I agree with Stephen that this should be done, but I agree with you that > it can wait for a future commit. However, I do think: > > 1) It should be called out rather plainly in the documentation. > 2) If there are files in pg_wal then pg_valid

Re: Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Tom Lane
Andres Freund writes: > I've locally fixed the issue by computing the stack base address anew > for postmaster children. Currently in InitPostmasterChild(). > I'd like to get that change upstream. The rr hackers have fixed a number > of other issues that could be hit with postgres, but they could

Re: BUG #16040: PL/PGSQL RETURN QUERY statement never uses a parallel plan

2020-03-27 Thread Hamid Akhtar
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed All good with this patch. -- Highgo Software (Canada/China

Re: allow online change primary_conninfo

2020-03-27 Thread Sergei Kornilov
Hello In other words, patches in reverse order: 0001 will change primary_conninfo and primary_slot_name to reloadable 0002 will move wal_receiver_create_temp_slot logic to startup process (without changing to PGC_POSTMASTER) 0003 is new draft patch: wal_receiver_create_temp_slot will use the give

Reinitialize stack base after fork (for the benefit of rr)?

2020-03-27 Thread Andres Freund
Hi, I've found rr [1] very useful to debug issues in postgres. The ability to hit a bug, and then e.g. identify a pointer with problematic contents, set a watchpoint on its contents, and reverse-continue is extremely powerful. Unfortunately, when running postgres, it currently occasionally trigge

Re: backup manifests

2020-03-27 Thread Robert Haas
On Fri, Mar 27, 2020 at 1:06 AM Andres Freund wrote: > > Like, suppose we change the default from CRC-32C to SHA-something. On > > the upside, the error detection rate will increase from 99.999+% > > to something much closer to 100%. > > FWIW, I don't buy the relevancy of 99.999+% at all.

Re: plan cache overhead on plpgsql expression

2020-03-27 Thread Tom Lane
I wrote: > Amit Langote writes: >> One thing -- I don't get the division between >> CachedPlanAllowsSimpleValidityCheck() and CachedPlanIsSimplyValid(). >> Maybe I am missing something, but could there not be just one >> function, possibly using whether expr_simple_expr is set or not to >> skip or

Re: A bug when use get_bit() function for a long bytea string

2020-03-27 Thread Daniel Verite
Ashutosh Bapat wrote: > I think we need a similar change in byteaGetBit() and byteaSetBit() > as well. get_bit() and set_bit() as SQL functions take an int4 as the "offset" argument representing the bit number, meaning that the maximum value that can be passed is 2^31-1. But the maximum t

Re: Unix-domain socket support on Windows

2020-03-27 Thread Andrew Dunstan
On 2/12/20 3:32 AM, Peter Eisentraut wrote: > Here is another patch set to enable this functionality. > > 0001 enables Unix-domain sockets on Windows, but leaves them turned > off by default at run time, using the mechanism introduced by > a9cff89f7e. This is relatively straightforward, except pe

Re: WIP/PoC for parallel backup

2020-03-27 Thread Asif Rehman
On Wed, Mar 25, 2020 at 12:22 PM Rajkumar Raghuwanshi < rajkumar.raghuwan...@enterprisedb.com> wrote: > Hi Asif, > > While testing further I observed parallel backup is not able to take > backup of standby server. > > mkdir /tmp/archive_dir > echo "archive_mode='on'">> data/postgresql.conf > echo

Re: Patch: to pass query string to pg_plan_query()

2020-03-27 Thread legrand legrand
Tom Lane-2 wrote > Fujii Masao < > masao.fujii@.nttdata > > writes: >> Does anyone object to this patch? I'm thinking to commit it separetely >> at first before committing the planning_counter_in_pg_stat_statements >> patch. > > I took a quick look through v9-0001-Pass-query-string-to-the-planne

Re: Allow auto_explain to log plans before queries are executed

2020-03-27 Thread legrand legrand
Kyotaro Horiguchi-4 wrote > At Thu, 27 Feb 2020 06:27:24 +0100, Pavel Stehule < > pavel.stehule@ > > wrote in >> odesílatel Kyotaro Horiguchi < > horikyota.ntt@ > > >> napsal: > > If we need a live plan dump of a running query, We could do that using > some kind of inter-backend triggering. (

Re: INSERT ... OVERRIDING USER VALUE vs GENERATED ALWAYS identity columns

2020-03-27 Thread Vik Fearing
On 3/27/20 9:33 AM, Dean Rasheed wrote: > On Fri, 27 Mar 2020 at 11:29, Peter Eisentraut > wrote: >> >> I have re-read everything and >> expanded your patch a bit with additional documentation and comments in >> the tests. > > I looked that over, and it all looks good to me. I concur. And it ma

Re: pgbench - rework variable management

2020-03-27 Thread David Steele
Hi Fabien, On 1/9/20 5:04 PM, Fabien COELHO wrote: Patch v4 is a just a rebase. Patch v5 is a rebase with some adjustements. This patch is failing on the Windows build: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.85698 I'm not sure if this had been fixed in v3 an

Re: INSERT ... OVERRIDING USER VALUE vs GENERATED ALWAYS identity columns

2020-03-27 Thread Dean Rasheed
On Fri, 27 Mar 2020 at 11:29, Peter Eisentraut wrote: > > We appear to have lost track of this. Ah yes, indeed! > I have re-read everything and > expanded your patch a bit with additional documentation and comments in > the tests. I looked that over, and it all looks good to me. Regards, Dean

Re: pgbench - refactor init functions with buffers

2020-03-27 Thread David Steele
On 11/6/19 12:48 AM, Fabien COELHO wrote: Hello Andres, Attached v3 shorten some lines and adds "append_tablespace". A v4 which just extends the patch to newly added 'G'. I'd prefer not to expand the use of pqexpbuffer in more places, and instead rather see this use StringInfo, now that's

Re: [PATCH] Btree BackwardScan race condition on Standby during VACUUM

2020-03-27 Thread Michail Nikolaev
Hello. > Probably, patch in this thread should fix this in btree_xlog_split() too? I have spent some time trying to find any possible race condition between btree_xlog_split and _bt_walk_left… But I can’t find any. Also, I have tried to cause any issue by putting pg_sleep put into btree_xlog_spl

Re: psql FETCH_COUNT feature does not work with combined queries

2020-03-27 Thread David Steele
Hi Fabien, On 1/6/20 5:20 PM, Tomas Vondra wrote: On Fri, Jul 26, 2019 at 04:13:23PM +, Fabien COELHO wrote: FETCH_COUNT does not work with combined queries, and probably has never worked since 2006. For the record, this bug has already been reported & discussed by Daniel Vérité a few

Re: shared-memory based stats collector

2020-03-27 Thread Alvaro Herrera
On 2020-Mar-27, Kyotaro Horiguchi wrote: > +/* > + * XLogArchiveWakeupEnd - Set up archiver wakeup stuff > + */ > +void > +XLogArchiveWakeupStart(void) > +{ > + Latch *old_latch PG_USED_FOR_ASSERTS_ONLY; > + > + SpinLockAcquire(&XLogCtl->info_lck); > + old_latch = XLogCtl->archiverWake

  1   2   >