Re: Don't choke on files that are removed while pg_rewind runs.

2020-07-14 Thread Michael Paquier
On Tue, Jul 14, 2020 at 12:18:41PM +0200, Daniel Gustafsson wrote: > I don't have strong opinions either, both approaches will work, so feel free > to > go ahead with the proposed change. Thanks. I have just gone with the solution to not change the query, and applied it down to 9.5. Please note

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-14 Thread Masahiko Sawada
On Tue, 14 Jul 2020 at 09:08, Masahiro Ikeda wrote: > > > I've attached the latest version patches. I've incorporated the review > > comments I got so far and improved locking strategy. > > Thanks for updating the patch! > I have three questions about the v23 patches. > > > 1. messages related to

Re: Parallel copy

2020-07-14 Thread Amit Kapila
On Sun, Jul 12, 2020 at 5:48 PM Bharath Rupireddy wrote: > > > > > Hi Bharath, > > > > I was looking forward to review this patch-set but unfortunately it is > > showing a reject in copy.c, and might need a rebase. > > I was applying on master over the commit- > > cd22d3cdb9bd9963c694c01a8c0232bba

Re: Default setting for enable_hashagg_disk

2020-07-14 Thread Peter Geoghegan
On Mon, Jul 13, 2020 at 9:47 AM Alvaro Herrera wrote: > I'm in favor of hash_mem_multiplier. I think a >1 default is more > sensible than =1 in the long run, but if strategic vote is what we're > doing, then I support the =1 option. Attached is a WIP patch implementing hash_mem_multiplier, with

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-14 Thread vignesh C
On Wed, Jul 15, 2020 at 8:03 AM Amit Langote wrote: > > Hi Vignesh, > > On Tue, Jul 14, 2020 at 10:23 PM vignesh C wrote: > > On Tue, Jul 14, 2020 at 11:19 AM Amit Langote > > wrote: > > > Sounds fine to me. Although CopyLoadRawBuf() does not seem to a > > > candidate for rigorous code optimiz

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 10:47 AM Dilip Kumar wrote: > > On Sun, Jul 12, 2020 at 9:56 PM Dilip Kumar wrote: > > > > On Mon, Jul 6, 2020 at 11:43 AM Dilip Kumar wrote: > > > > > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila > > > wrote: > > > > > > > > On Sun, Jul 5, 2020 at 4:47 PM Dilip Kumar

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:00 PM Amit Kapila wrote: > > On Mon, Jul 13, 2020 at 3:04 PM Dilip Kumar wrote: > > > > On Mon, Jul 13, 2020 at 2:56 PM Amit Kapila wrote: > > > > > > On Mon, Jul 13, 2020 at 2:32 PM Dilip Kumar wrote: > > > > > > > > On Mon, Jul 13, 2020 at 11:10 AM Amit Kapila > >

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread David Rowley
On Wed, 15 Jul 2020 at 14:51, Amit Kapila wrote: > > On Wed, Jul 15, 2020 at 5:55 AM David Rowley wrote: > > If we've not seen any performance regressions within 1 week, then I > > propose that we (pending final review) push this to allow wider > > testing. > > I think Soumyadeep has reported a r

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-07-14 Thread Justin Pryzby
On Sun, Jun 21, 2020 at 08:53:25PM -0500, Justin Pryzby wrote: > I'm still waiting to hear feedback from a commiter if this is a good idea to > put this into the system catalog. Right now, ts_debug is the only nontrivial > function. I'm still missing feedback from committers about the foundation

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread Amit Kapila
On Wed, Jul 15, 2020 at 5:55 AM David Rowley wrote: > > On Tue, 14 Jul 2020 at 19:13, Thomas Munro wrote: > > > > On Fri, Jun 26, 2020 at 3:33 AM Robert Haas wrote: > > > On Tue, Jun 23, 2020 at 11:53 PM David Rowley > > > wrote: > > > > In summary, based on these tests, I don't think we're ma

Re: GSSENC'ed connection stalls while reconnection attempts.

2020-07-14 Thread Kyotaro Horiguchi
At Tue, 14 Jul 2020 13:31:31 +0900 (JST), Kyotaro Horiguchi wrote in > Agreed to separate the change from this issue. I also don't think > that change in behavior dramatically improve the situation since we > should have had a bunch of trouble when a write actually failed in the > normal case.

Re: Binary support for pgoutput plugin

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 22:28:48 -0400, Tom Lane wrote: > Andres Freund writes: > > What is the gain in having these checks? recv functions need to be safe > > against arbitrary input, so a type crosscheck doesn't buy additional > > safety in that regard. Not that a potential attacker couldn't just > >

Re: Is it useful to record whether plans are generic or custom?

2020-07-14 Thread Fujii Masao
On 2020/07/14 21:24, torikoshia wrote: On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +   

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Amit Kapila
On Tue, Jul 14, 2020 at 3:37 PM Petr Jelinek wrote: > > On 14/07/2020 11:36, Dilip Kumar wrote: > > On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > >> > >> I am not sure if I can follow the discussion here very well, but if I > >> understand correctly I'd like to clarify two things: > >> -

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Amit Kapila
On Wed, Jul 15, 2020 at 12:32 AM Robert Haas wrote: > > On Sat, Jul 11, 2020 at 8:37 AM Dilip Kumar wrote: > > I have just notice that the parallelism is off even for the select > > part of the query mentioned in the $subject. I see the only reason it > > is not getting parallel because we block

Re: WIP: BRIN multi-range indexes

2020-07-14 Thread Alexander Korotkov
On Mon, Jul 13, 2020 at 5:59 PM Tomas Vondra wrote: > >> > If we really want to print something nicer, I'd say it needs to be a > >> > special function in the BRIN opclass. > >> > >> If that can be done, then +1. We just need to ensure that the function > >> knows and can verify the type of index

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-14 Thread Amit Langote
Hi Vignesh, On Tue, Jul 14, 2020 at 10:23 PM vignesh C wrote: > On Tue, Jul 14, 2020 at 11:19 AM Amit Langote wrote: > > Sounds fine to me. Although CopyLoadRawBuf() does not seem to a > > candidate for rigorous code optimization as it does not get called > > that often. > > I thought we could

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
Andres Freund writes: > What is the gain in having these checks? recv functions need to be safe > against arbitrary input, so a type crosscheck doesn't buy additional > safety in that regard. Not that a potential attacker couldn't just > change the content anyways? You're confusing security issue

Re: Binary support for pgoutput plugin

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 19:46:52 -0400, Tom Lane wrote: > Andres Freund writes: > > There's also send/receive functions that do not work across systems, > > unfortunately :(. In particular record and array send functions embed > > type oids and their receive functions verify that they match the local >

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread David Rowley
On Tue, 14 Jul 2020 at 19:13, Thomas Munro wrote: > > On Fri, Jun 26, 2020 at 3:33 AM Robert Haas wrote: > > On Tue, Jun 23, 2020 at 11:53 PM David Rowley wrote: > > > In summary, based on these tests, I don't think we're making anything > > > worse in regards to synchronize_seqscans if we cap t

Re: avoid bitmapOR-ing indexes with scan condition inconsistent with partition constraint

2020-07-14 Thread Justin Pryzby
Added here: https://commitfest.postgresql.org/29/2644/ And updated tests to pass following: |commit 689696c7110f148ede8004aae50d7543d05b5587 |Author: Tom Lane |Date: Tue Jul 14 18:56:49 2020 -0400 | |Fix bitmap AND/OR scans on the inside of a nestloop partition-wise join. >From 5c323ffaa8c7

Refactoring relkind as an enum

2020-07-14 Thread Mark Dilger
> On Jul 14, 2020, at 4:12 PM, Alvaro Herrera wrote: > > This patch is way too large. Probably in part explained by the fact > that you seem to have run pgindent and absorbed a lot of unrelated > changes. This makes the patch essentially unreviewable. I did not run pgindent, but when changi

Re: Cache lookup errors with functions manipulation object addresses

2020-07-14 Thread Michael Paquier
On Tue, Jul 07, 2020 at 11:08:30AM -0400, Alvaro Herrera wrote: g> That's a holdover from old times, when we thought functions were > procedures. That's no longer the case. Thanks, so "routine" it is. I have done an extra round of review of this patch, and noticed two things: - In getObjectDescr

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
Andres Freund writes: > There's also send/receive functions that do not work across systems, > unfortunately :(. In particular record and array send functions embed > type oids and their receive functions verify that they match the local > system. Which basically means that if there's any differen

Re: Postgres is not able to handle more than 4k tables!?

2020-07-14 Thread Alvaro Herrera
On 2020-Jul-10, Konstantin Knizhnik wrote: > @@ -3076,6 +3080,10 @@ relation_needs_vacanalyze(Oid relid, > instuples = tabentry->inserts_since_vacuum; > anltuples = tabentry->changes_since_analyze; > > + rel = RelationIdGetRelation(relid); > +

Re: Towards easier AMs: Cleaning up inappropriate use of name "relkind"

2020-07-14 Thread Alvaro Herrera
This patch is way too large. Probably in part explained by the fact that you seem to have run pgindent and absorbed a lot of unrelated changes. This makes the patch essentially unreviewable. I think you should define a RelationGetRelkind() static function that returns the appropriate datatype wi

Re: Postgres is not able to handle more than 4k tables!?

2020-07-14 Thread Bruce Momjian
On Fri, Jul 10, 2020 at 02:10:20AM +, tsunakawa.ta...@fujitsu.com wrote: > > There were many proposed patches which help to improve this > > situation. But as far as this patches increase performance only > > at huge servers with large number of cores and show almost no > > improvement (or eve

Re: Default setting for enable_hashagg_disk

2020-07-14 Thread Peter Geoghegan
On Tue, Jul 14, 2020 at 12:46 PM Robert Haas wrote: > - I thought the problem we were trying to solve here was that, in v12, > if the planner thinks that your hashagg will fit in memory when really > it doesn't, you will get good performance because we'll cheat; in v13, > you'll get VERY bad perfo

Re: [patch] demote

2020-07-14 Thread Jehan-Guillaume de Rorthais
On Tue, 14 Jul 2020 12:49:51 -0700 Andres Freund wrote: > Hi, > > On 2020-07-14 17:26:37 +0530, Amul Sul wrote: > > On Mon, Jul 13, 2020 at 8:35 PM Jehan-Guillaume de Rorthais > > wrote: > > > > > > Hi, > > > > > > Another summary + patch + tests. > > > > > > This patch supports 2PC. The goal

Re: SQL/JSON: functions

2020-07-14 Thread Andrew Dunstan
On 7/14/20 1:00 PM, Andrew Dunstan wrote: > On 7/5/20 1:29 PM, Justin Pryzby wrote: >> On Mon, Mar 23, 2020 at 08:28:52PM +0300, Nikita Glukhov wrote: >>> Attached 47th version of the patches. >> The patch checker/cfbot says this doesn't apply ; could you send a rebasified >> version ? >> > To ke

Re: Rethinking opclass member checks and dependency strength

2020-07-14 Thread Anastasia Lubennikova
On 31.03.2020 23:45, Tom Lane wrote: I wrote: Still haven't got a better naming idea, but in the meantime here's a rebase to fix a conflict with 612a1ab76. Maybe "amadjustmembers" will work? I've looked through the patch and noticed this comment: +            default: +                /* Pro

Re: Multi-byte character case-folding

2020-07-14 Thread Bruce Momjian
On Mon, Jul 6, 2020 at 08:32:22PM -0400, Tom Lane wrote: > Yes, a GUC changing this would be a headache. It would be just as much of > a compatibility and security hazard as standard_conforming_strings (which > indeed I've been thinking of proposing that we get rid of; it's hung > around long eno

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 3:42 PM Andres Freund wrote: > The "found xmin ... from before relfrozenxid ..." cases should all be > fixable without needing such a function, and without it making fixing > them significantly easier, no? As far as I understand your suggested > solution, you need the tid(s

Re: Binary support for pgoutput plugin

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 14:08:53 -0400, Dave Cramer wrote: > On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: > > > So I started looking through this seriously, and my first question > > is why do the docs and code keep saying that "base types" are sent > > in binary? Why not just "data"? Are there any

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Alvaro Herrera
On 2020-Jul-14, Andres Freund wrote: > Hi, > > On 2020-07-14 13:20:25 -0400, Alvaro Herrera wrote: > > Just having the block number is already a tremendous step forward; with > > that you can ask the customer to set a pageinspect dump of tuple > > headers, and then the problem is obvious. Now i

Re: [patch] demote

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 17:26:37 +0530, Amul Sul wrote: > On Mon, Jul 13, 2020 at 8:35 PM Jehan-Guillaume de Rorthais > wrote: > > > > Hi, > > > > Another summary + patch + tests. > > > > This patch supports 2PC. The goal is to keep them safe during demote/promote > > actions so they can be committed/r

Re: Default setting for enable_hashagg_disk

2020-07-14 Thread Robert Haas
On Mon, Jul 13, 2020 at 2:50 PM Peter Geoghegan wrote: > Primarily in favor of escape hatch: > > Jeff, > DavidR, > Pavel, > Andres, > Robert ??, > Amit ?? > > Primarily in favor of hash_mem/hash_mem_multiplier: > > PeterG, > Tom, > Alvaro, > Tomas, > Justin, > DavidG, > Jonathan Katz > > There are

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 07:51:27 -0400, Robert Haas wrote: > On Tue, Jul 14, 2020 at 3:08 AM Peter Eisentraut > wrote: > > In the meantime, if you're wizard enough to deal with this kind of > > thing, you could also clone the module from the PG14 tree and build it > > against older versions manually. >

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andres Freund
Hi, On 2020-07-14 13:20:25 -0400, Alvaro Herrera wrote: > On 2020-Jul-13, Andres Freund wrote: > > > Hi, > > > > On 2020-07-13 17:12:18 -0400, Robert Haas wrote: > > > 1. There's nothing to identify the tuple that has the problem, and no > > > way to know how many more of them there might be. Ba

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Andres Freund
Hi, On 2020-07-13 21:18:10 -0400, Robert Haas wrote: > On Mon, Jul 13, 2020 at 9:10 PM Andres Freund wrote: > > > What if clog has been truncated so that the xmin can't be looked up? > > > > That's possible, but probably only in cases where xmin actually > > committed. > > Isn't that the normal

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2020-07-14 Thread Andres Freund
Hi, On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote: > Totally unasked for, here's a rebase of this patch series. I didn't do > anything other than rebasing to current master, solving a couple of very > trivial conflicts, fixing some whitespace complaints by git apply, and > running tests to v

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Robert Haas
On Sat, Jul 11, 2020 at 8:37 AM Dilip Kumar wrote: > I have just notice that the parallelism is off even for the select > part of the query mentioned in the $subject. I see the only reason it > is not getting parallel because we block the parallelism if the query > type is not SELECT. I don't se

Re: warnings for invalid function casts

2020-07-14 Thread Peter Eisentraut
On 2020-07-07 18:08, Tom Lane wrote: Peter Eisentraut writes: On 2020-07-04 16:16, Tom Lane wrote: I'm for a typedef. There is *nothing* readable about "(void (*) (void))", and the fact that it's theoretically incorrect for the purpose doesn't exactly aid intelligibility either. With a typed

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
Dave Cramer writes: > On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: >> So I started looking through this seriously, and my first question >> is why do the docs and code keep saying that "base types" are sent >> in binary? Why not just "data"? Are there any cases where we >> don't use binary for

RE: Index Skip Scan (new UniqueKeys)

2020-07-14 Thread Floris Van Nee
> > > - the uniquekeys that is built, still contains some redundant keys, that are > normally eliminated from the path keys lists. > > I guess you're talking about: > > + if (EC_MUST_BE_REDUNDANT(ec)) > + continue; > > Can you add some test cases to your changes to show the effect of it? It >

Re: Binary support for pgoutput plugin

2020-07-14 Thread Dave Cramer
On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: > So I started looking through this seriously, and my first question > is why do the docs and code keep saying that "base types" are sent > in binary? Why not just "data"? Are there any cases where we > don't use binary format, if the subscription r

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2020-07-14 Thread Anastasia Lubennikova
On 01.07.2020 12:38, Daniel Gustafsson wrote: This patch incurs a compiler warning, which probably is quite simple to fix: heapam.c: In function ‘heap_multi_insert’: heapam.c:2349:4: error: ‘recptr’ may be used uninitialized in this function [-Werror=maybe-uninitialized] visibilitymap_set(

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Alvaro Herrera
On 2020-Jul-13, Andres Freund wrote: > Hi, > > On 2020-07-13 17:12:18 -0400, Robert Haas wrote: > > 1. There's nothing to identify the tuple that has the problem, and no > > way to know how many more of them there might be. Back-patching > > b61d161c146328ae6ba9ed937862d66e5c8b035a would help wit

Re: Binary support for pgoutput plugin

2020-07-14 Thread Tom Lane
So I started looking through this seriously, and my first question is why do the docs and code keep saying that "base types" are sent in binary? Why not just "data"? Are there any cases where we don't use binary format, if the subscription requests it? If there's not a concrete reason to use tha

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-07-14 Thread Bruce Momjian
On Tue, Jul 14, 2020 at 03:38:49PM +0530, Bharath Rupireddy wrote: > Approach #4: > A postgres_fdw foreign server level option: connection idle time, the > amount of idle time for that server cached entry, after which the > cached entry goes away. Probably the backend, before itself going to > idle

Re: Index Skip Scan (new UniqueKeys)

2020-07-14 Thread Dmitry Dolgov
> On Sun, Jul 12, 2020 at 12:48:47PM +, Floris Van Nee wrote: > > > > Good point, thanks for looking at this. With the latest planner version > > there > > are indeed more possibilities to use skipping. It never occured to me that > > some of those paths will still rely on index scan returning

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 6:09 AM Bharath Rupireddy wrote: > Thanks all for the ideas. There have been various points/approaches > discussed in the entire email chain so far. > I would like to summarize all of them here, so that we can agree on > one of the options and proceed further with this feat

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread David G. Johnston
On Tue, Jul 14, 2020 at 7:21 AM Pavel Stehule wrote: > út 14. 7. 2020 v 16:09 odesílatel David G. Johnston < > david.g.johns...@gmail.com> napsal: > >> On Tue, Jul 14, 2020 at 6:56 AM Pavel Stehule >> wrote: >> >>> út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < >>> david.g.johns...@gmail.

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-14 Thread Alexey Kondratov
On 2020-07-14 15:27, Ashutosh Bapat wrote: On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov wrote: I built a simple two node multi-tenant schema for tests, which can be easily set up with attached scripts. It creates three tables (companies, users, documents) distributed over two nodes. Every

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Tue, Jul 14, 2020 at 4:09 PM Robert Haas wrote: > > On Tue, Jul 14, 2020 at 8:25 AM Magnus Hagander > > wrote: > > > I don't think that it necessarily has to be. As long as we're talking > > about adding something and not actually ch

Re: [Proposal] Global temporary tables

2020-07-14 Thread Pavel Stehule
pá 10. 7. 2020 v 11:04 odesílatel wenjing zeng napsal: > HI all > > I started using my personal email to respond to community issue. > > > > 2020年7月7日 下午6:05,Pavel Stehule 写道: > > Hi > > >> GTT Merge the latest PGMaster and resolves conflicts. >> >> >> > I tested it and it looks fine. I think it

Re: Creating foreign key on partitioned table is too slow

2020-07-14 Thread Daniel Gustafsson
> On 24 Mar 2020, at 16:26, Alvaro Herrera wrote: > It > would be great if Kato Sho can try the original test case with my latest > patch (the one in https://postgr.es/m/20191113214544.GA16060@alvherre.pgsql ) > and let us know if it improves things. Hi!, Are you able to test Alvaros latest pat

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Magnus Hagander
On Tue, Jul 14, 2020 at 4:09 PM Robert Haas wrote: > On Tue, Jul 14, 2020 at 8:25 AM Magnus Hagander > wrote: > > I don't think that it necessarily has to be. As long as we're talking > about adding something and not actually changing their existing packages, > getting this into both yum and apt

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread Pavel Stehule
út 14. 7. 2020 v 16:09 odesílatel David G. Johnston < david.g.johns...@gmail.com> napsal: > On Tue, Jul 14, 2020 at 6:56 AM Pavel Stehule > wrote: > >> út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < >> david.g.johns...@gmail.com> napsal: >> >>> Further comments welcome so I'm putting it ba

Re: [Proposal] Global temporary tables

2020-07-14 Thread Pavel Stehule
po 13. 7. 2020 v 13:59 odesílatel wenjing zeng napsal: > > > 2020年7月10日 下午5:03,wenjing zeng 写道: > > HI all > > I started using my personal email to respond to community issue. > > > > 2020年7月7日 下午6:05,Pavel Stehule 写道: > > Hi > > >> GTT Merge the latest PGMaster and resolves conflicts. >> >> >>

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 8:25 AM Magnus Hagander wrote: > I don't think that it necessarily has to be. As long as we're talking about > adding something and not actually changing their existing packages, getting > this into both yum and apt shouldn't be *that* hard, if it's coordinated well > wi

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread David G. Johnston
On Tue, Jul 14, 2020 at 6:56 AM Pavel Stehule wrote: > út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < > david.g.johns...@gmail.com> napsal: > >> Further comments welcome so I'm putting it back into needs review for the >> moment while I work on the refactor. >> > > attached fixed patch > >

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Mon, Jul 13, 2020 at 9:29 PM Fujii Masao wrote: > But updating this tool can fit to the release schedule and > policy of PostgreSQL? > > While investigating the problem by using this tool, we may want to > add new feature into the tool because it's necessary for the investigation. > But users w

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread Pavel Stehule
út 14. 7. 2020 v 15:55 odesílatel David G. Johnston < david.g.johns...@gmail.com> napsal: > On Tue, Jul 14, 2020 at 5:40 AM Justin Pryzby > wrote: > >> On Tue, Jul 14, 2020 at 07:25:56AM +0200, Pavel Stehule wrote: >> > út 14. 7. 2020 v 0:37 odesílatel David G. Johnston < >> david.g.johns...@gmai

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread David G. Johnston
On Tue, Jul 14, 2020 at 5:40 AM Justin Pryzby wrote: > On Tue, Jul 14, 2020 at 07:25:56AM +0200, Pavel Stehule wrote: > > út 14. 7. 2020 v 0:37 odesílatel David G. Johnston < > david.g.johns...@gmail.com> napsal: > > > On Mon, Jul 13, 2020 at 2:12 AM Pavel Stehule > wrote: > > I think so now all

Re: Binary support for pgoutput plugin

2020-07-14 Thread Dave Cramer
On Tue, 14 Jul 2020 at 09:26, Daniel Gustafsson wrote: > > On 13 Jul 2020, at 15:11, Dave Cramer wrote: > > I took another look at the updated version today. Since there now were > some > unused variables and (I believe) unnecessary checks (int size and > endianness > etc) left, I took the libe

Re: Improvements in Copy From

2020-07-14 Thread vignesh C
On Tue, Jul 14, 2020 at 12:17 PM vignesh C wrote: > > On Tue, Jul 14, 2020 at 11:13 AM David Rowley wrote: > > > > On Tue, 14 Jul 2020 at 17:22, David Rowley wrote: > > > > > > On Thu, 2 Jul 2020 at 00:46, vignesh C wrote: > > > > b) CopyMultiInsertInfoNextFreeSlot had an unused function parame

Re: Binary support for pgoutput plugin

2020-07-14 Thread Daniel Gustafsson
> On 13 Jul 2020, at 15:11, Dave Cramer wrote: I took another look at the updated version today. Since there now were some unused variables and (I believe) unnecessary checks (int size and endianness etc) left, I took the liberty to fix those. I also fixed some markup in the catalog docs, did s

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-14 Thread vignesh C
On Tue, Jul 14, 2020 at 11:19 AM Amit Langote wrote: > > > Sounds fine to me. Although CopyLoadRawBuf() does not seem to a > candidate for rigorous code optimization as it does not get called > that often. > I thought we could include that change as we are making changes around that code. Rest o

Re: Retry Cached Remote Connections for postgres_fdw in case remote backend gets killed/goes away

2020-07-14 Thread Bharath Rupireddy
On Tue, Jul 14, 2020 at 6:13 PM Ashutosh Bapat wrote: > > Has this been added to CF, possibly next CF? > I have not added yet. Request to get it done in this CF, as the final patch for review(v3 patch) is ready and shared. We can target it to the next CF if there are major issues with the patch.

Re: Retry Cached Remote Connections for postgres_fdw in case remote backend gets killed/goes away

2020-07-14 Thread Ashutosh Bapat
Has this been added to CF, possibly next CF? On Tue, Jul 14, 2020 at 7:27 AM Bharath Rupireddy wrote: > > > > > You could get a backend's PID using PQbackendPID and then kill it by > > calling pg_terminate_backend() to kill the remote backend to automate > > scenario of remote backend being kille

Re: DROP relation IF EXISTS Docs and Tests - Bug Fix

2020-07-14 Thread Justin Pryzby
On Tue, Jul 14, 2020 at 07:25:56AM +0200, Pavel Stehule wrote: > út 14. 7. 2020 v 0:37 odesílatel David G. Johnston > napsal: > > On Mon, Jul 13, 2020 at 2:12 AM Pavel Stehule > > wrote: > I think so now all changes are correct and valuable. I''l mark this patch > as ready for commit This is f

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-14 Thread Ashutosh Bapat
On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov wrote: > > Hi Hackers, > > The idea of achieving Postgres scaling via sharding using postgres_fdw + > partitioning got a lot of attention last years. Many optimisations have > been done in this direction: partition pruning, partition-wise > aggrega

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Magnus Hagander
On Tue, Jul 14, 2020 at 1:52 PM Robert Haas wrote: > On Tue, Jul 14, 2020 at 4:59 AM Magnus Hagander > wrote: > > The countersable of this is pg_resetwal. The number of people who have > broken their database with that tool is not small. > > Very true. > > > That said, we could have a separate "

Re: Is it useful to record whether plans are generic or custom?

2020-07-14 Thread torikoshia
On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +    TupleDescInitEntry(tupdesc, (AttrNumber) 8

Re: [patch] demote

2020-07-14 Thread Amul Sul
On Mon, Jul 13, 2020 at 8:35 PM Jehan-Guillaume de Rorthais wrote: > > Hi, > > Another summary + patch + tests. > > This patch supports 2PC. The goal is to keep them safe during demote/promote > actions so they can be committed/rollbacked later on a primary. See tests. > Wondering is it necessary

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 4:59 AM Magnus Hagander wrote: > The countersable of this is pg_resetwal. The number of people who have broken > their database with that tool is not small. Very true. > That said, we could have a separate "class" of extensions/tools in the > distribution, and encourage

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Robert Haas
On Tue, Jul 14, 2020 at 3:08 AM Peter Eisentraut wrote: > In the meantime, if you're wizard enough to deal with this kind of > thing, you could also clone the module from the PG14 tree and build it > against older versions manually. But what if you are NOT a wizard, and a wizard is giving you dir

RE: problem with RETURNING and update row movement

2020-07-14 Thread osumi.takami...@fujitsu.com
Amit san Hello. I've tested your patch. This patch can be applied comfortably and make check-world has produced no failure. I didn't do performance test because this patch doesn't have an effect on it in my opinion. I reproduced the bug you described before, which can be prevented by your pat

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Peter Eisentraut
On 2020-07-14 02:41, Robert Haas wrote: I think that our normal back-patching rules are based primarily on the risk of breaking things, and a new contrib module carries a pretty negligible risk of breaking anything that works today. I think that all feature code ought to go through a beta cycle

Re: TAP tests and symlinks on Windows

2020-07-14 Thread Peter Eisentraut
On 2020-07-10 13:58, Andrew Dunstan wrote: After much frustration and gnashing of teeth here's a patch that allows almost all the TAP tests involving symlinks to work as expected on all Windows build environments, without requiring an additional Perl module. I have tested this on a system that is

Re: max_slot_wal_keep_size and wal_keep_segments

2020-07-14 Thread David Steele
On 7/14/20 12:00 AM, Fujii Masao wrote: The patch was no longer applied cleanly because of recent commit. So I updated the patch. Attached. Barring any objection, I will commit this patch. This doesn't look right: + the most recent megabytes + WAL files plus one WAL file are How about:

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-14 Thread Julien Rouhaud
On Tue, Jul 14, 2020 at 07:11:02PM +0900, Atsushi Torikoshi wrote: > Hi, > > v9 patch fails to apply to HEAD, could you check and rebase it? Thanks for the notice, v10 attached! > And here are minor typos. > > 79 +* utility statements. Note that we don't compute a queryId > for prepar

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Jochem van Dieten
On Tue, Jul 14, 2020 at 3:26 AM Tom Lane wrote: > I think you're attacking a straw man. I'm well aware of how open source > works, thanks. What I'm saying is that contrib is mostly seen to be > reasonably harmless stuff. Sure, you can overwrite data you didn't want > to with adminpack's pg_file

Re: Don't choke on files that are removed while pg_rewind runs.

2020-07-14 Thread Daniel Gustafsson
> On 13 Jul 2020, at 14:18, Michael Paquier wrote: > That sounds like a good idea with an extra qual in the first part of > the inner CTE, if coupled with a check to make sure that we never > get a NULL result. Now there is IMO an argument to not complicate > more this query as it is not like a

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-14 Thread Atsushi Torikoshi
Hi, v9 patch fails to apply to HEAD, could you check and rebase it? And here are minor typos. 79 +* utility statements. Note that we don't compute a queryId for prepared 80 +* statemets related utility, as those will inherit from the underlying 81 +* statements's one

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-07-14 Thread Bharath Rupireddy
Thanks all for the ideas. There have been various points/approaches discussed in the entire email chain so far. I would like to summarize all of them here, so that we can agree on one of the options and proceed further with this feature. The issue this feature is trying to solve: In postgres_fdw,

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Petr Jelinek
On 14/07/2020 11:36, Dilip Kumar wrote: On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: Hi, On 14/07/2020 10:29, Amit Kapila wrote: On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Amit Kapila
On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > > Hi, > > On 14/07/2020 10:29, Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > >> > >> On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila > >> wrote: > >>> > >>> On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar > >>> wro

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Dilip Kumar
On Tue, Jul 14, 2020 at 2:47 PM Petr Jelinek wrote: > > Hi, > > On 14/07/2020 10:29, Amit Kapila wrote: > > On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > >> > >> On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila > >> wrote: > >>> > >>> On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar > >>> wro

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Petr Jelinek
Hi, On 14/07/2020 10:29, Amit Kapila wrote: On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar wrote: On Thu, Jul 9, 2020 at 6:55 PM Amit Kapila wrote: On Thu, Jul 9, 2020 at 6:14 PM Pet

Re: Avoid useless retrieval of defaults and check constraints in pg_dump -a

2020-07-14 Thread Julien Rouhaud
On Sun, Jul 12, 2020 at 4:29 PM Justin Pryzby wrote: > > On Sun, Jul 12, 2020 at 07:48:50AM +0200, Julien Rouhaud wrote: > > Currently, getTableAttrs() always retrieves info about columns defaults and > > check constraints, while this will never be used if --data-only option if > > used. > > This

Re: Online checksums verification in the backend

2020-07-14 Thread Julien Rouhaud
On Sun, Jul 12, 2020 at 12:34:03PM -0500, Justin Pryzby wrote: > Small language fixes in comments and user-facing docs. Thanks a lot! I just fixed a small issue (see below), PFA updated v10. > > diff --git a/src/backend/storage/page/checksum.c > b/src/backend/storage/page/checksum.c > index eb

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-14 Thread Magnus Hagander
On Tue, Jul 14, 2020 at 3:26 AM Tom Lane wrote: > Robert Haas writes: > > On Mon, Jul 13, 2020 at 8:58 PM Tom Lane wrote: > >> The more that I think about it, the more I think that the proposed > >> functions are tools for wizards only, and so I'm getting hesitant > >> about having them in cont

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-07-14 Thread Bharath Rupireddy
On Tue, Jul 7, 2020 at 10:24 PM Andres Freund wrote: > > On 2020-07-07 09:44:41 -0400, Tom Lane wrote: > > Bharath Rupireddy writes: > > > Firstly, pg_start_backup registers nonexclusive_base_backup_cleanup as > > > on_shmem_exit call back which will > > > add this function to the before_shmem_ex

Re: replication_origin and replication_origin_lsn usage on subscriber

2020-07-14 Thread Amit Kapila
On Tue, Jul 14, 2020 at 12:05 PM Dilip Kumar wrote: > > On Tue, Jul 14, 2020 at 11:14 AM Amit Kapila wrote: > > > > On Tue, Jul 14, 2020 at 11:00 AM Dilip Kumar wrote: > > > > > > On Thu, Jul 9, 2020 at 6:55 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Jul 9, 2020 at 6:14 PM Petr Jeline

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-14 Thread Masahiro Ikeda
I've attached the latest version patches. I've incorporated the review comments I got so far and improved locking strategy. I want to ask a question about streaming replication with 2PC. Are you going to support 2PC with streaming replication? I tried streaming replication using v23 patches. I

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Rushabh Lathia
On Sat, Jul 11, 2020 at 6:07 PM Dilip Kumar wrote: > I have just notice that the parallelism is off even for the select > part of the query mentioned in the $subject. I see the only reason it > is not getting parallel because we block the parallelism if the query > type is not SELECT. I don't s

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-14 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 4:23 PM Amit Kapila wrote: > > On Sat, Jul 11, 2020 at 6:07 PM Dilip Kumar wrote: > > > > I have just notice that the parallelism is off even for the select > > part of the query mentioned in the $subject. I see the only reason it > > is not getting parallel because we bl

Re: Parallel Seq Scan vs kernel read ahead

2020-07-14 Thread Thomas Munro
On Fri, Jun 26, 2020 at 3:33 AM Robert Haas wrote: > On Tue, Jun 23, 2020 at 11:53 PM David Rowley wrote: > > In summary, based on these tests, I don't think we're making anything > > worse in regards to synchronize_seqscans if we cap the maximum number > > of blocks to allocate to each worker at

  1   2   >