Re: doc review for v14

2020-12-28 Thread Michael Paquier
On Tue, Dec 29, 2020 at 01:59:58PM +1300, Thomas Munro wrote: > LGTM. Thanks, I have done this one then. -- Michael signature.asc Description: PGP signature

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-28 Thread Masahiko Sawada
Hi Ajin, On Wed, Dec 23, 2020 at 6:38 PM Ajin Cherian wrote: > > On Tue, Dec 22, 2020 at 8:59 PM Amit Kapila wrote: > > > > On Tue, Dec 22, 2020 at 2:51 PM Ajin Cherian wrote: > > > > > > On Sat, Dec 19, 2020 at 2:13 PM Amit Kapila > > > wrote: > > > > > > > Okay, I have changed the rollback_

Re: New IndexAM API controlling index vacuum strategies

2020-12-28 Thread Peter Geoghegan
On Mon, Dec 28, 2020 at 11:20 PM Peter Geoghegan wrote: > > That's a very good result in terms of skipping lazy_vacuum_heap(). How > > much the table and indexes bloated? Also, I'm curious about that which > > tests in choose_vacuum_strategy() turned vacuum_heap on: 130 test or > > test if mainten

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-12-28 Thread Masahiko Sawada
Hi Simon, On Mon, Nov 30, 2020 at 1:53 AM Simon Riggs wrote: > > On Fri, 20 Nov 2020 at 15:29, Alvaro Herrera wrote: > > > > Note on heap_prepare_freeze_tuple()'s fifth parameter, it's not valid to > > pass OldestXmin; you need a multixact limit there, not an Xid limit. I > > think the return v

Re: Detecting File Damage & Inconsistencies

2020-12-28 Thread Masahiko Sawada
Hi Simon, On Wed, Nov 18, 2020 at 2:14 AM Simon Riggs wrote: > > On Fri, 13 Nov 2020 at 11:24, Simon Riggs wrote: > > > > On Fri, 13 Nov 2020 at 00:50, tsunakawa.ta...@fujitsu.com > > wrote: > > > > > > From: Simon Riggs > > > > If a rogue user/process is suspected, this would allow you to ide

Re: New IndexAM API controlling index vacuum strategies

2020-12-28 Thread Peter Geoghegan
On Mon, Dec 28, 2020 at 10:26 PM Masahiko Sawada wrote: > The second test checking if maintenane_work_mem space is running out > also makes sense to me. Perhaps another idea would be to compare the > number of collected garbage tuple to the total number of heap tuples > so that we do lazy_vacuum_h

Re: New IndexAM API controlling index vacuum strategies

2020-12-28 Thread Masahiko Sawada
On Tue, Dec 29, 2020 at 7:06 AM Peter Geoghegan wrote: > > On Sun, Dec 27, 2020 at 11:41 PM Peter Geoghegan wrote: > > I experimented with this today, and I think that it is a good way to > > do it. I like the idea of choose_vacuum_strategy() understanding that > > heap pages that are subject to

Re: New IndexAM API controlling index vacuum strategies

2020-12-28 Thread Masahiko Sawada
On Mon, Dec 28, 2020 at 4:42 PM Peter Geoghegan wrote: > > On Sun, Dec 27, 2020 at 10:55 PM Masahiko Sawada > wrote: > > > As you said, the next question must be: How do we teach lazy vacuum to > > > not do what gets requested by amvacuumcleanup() when it cannot respect > > > the wishes of one i

Re: HOT chain bug in latestRemovedXid calculation

2020-12-28 Thread Peter Geoghegan
On Tue, Dec 22, 2020 at 9:52 AM Peter Geoghegan wrote: > ISTM that heap_compute_xid_horizon_for_tuples() calculates > latestRemovedXid for index deletion callers without sufficient care. > The function only follows line pointer redirects, which is necessary > but not sufficient to visit all releva

Re: Let's start using setenv()

2020-12-28 Thread Tom Lane
Thomas Munro writes: > On Tue, Dec 29, 2020 at 4:21 PM Tom Lane wrote: >> I also changed our unsetenv() emulations to make them return an int >> error indicator, as per POSIX. I have no desire to change the call >> sites to check for errors, but it seemed like our compatibility stubs >> should b

Re: Let's start using setenv()

2020-12-28 Thread Thomas Munro
On Tue, Dec 29, 2020 at 4:21 PM Tom Lane wrote: > Back in 2001 we decided to prefer putenv() over setenv() because the > latter wasn't in POSIX (SUSv2) at the time. That decision has been > overtaken by events: more recent editions of POSIX not only provide > setenv() but recommend it over putenv

Let's start using setenv()

2020-12-28 Thread Tom Lane
Back in 2001 we decided to prefer putenv() over setenv() because the latter wasn't in POSIX (SUSv2) at the time. That decision has been overtaken by events: more recent editions of POSIX not only provide setenv() but recommend it over putenv(). So I think it's time to change our policy and prefer

Re: Parallel Full Hash Join

2020-12-28 Thread Thomas Munro
On Mon, Dec 28, 2020 at 9:49 PM Masahiko Sawada wrote: > On Thu, Nov 5, 2020 at 7:34 AM Melanie Plageman > wrote: > > I've attached a patch with the corrections I mentioned upthread. > > I've gone ahead and run pgindent, though, I can't say that I'm very > > happy with the result. > > > > I'm sti

Re: doc review for v14

2020-12-28 Thread Thomas Munro
On Thu, Dec 24, 2020 at 9:12 PM Michael Paquier wrote: > On Mon, Dec 21, 2020 at 10:11:53PM -0600, Justin Pryzby wrote: > > Specifies the amount of memory that should be allocated at server > > -startup time for use by parallel queries. When this memory region > > is > > +

Re: doc review for v14

2020-12-28 Thread Michael Paquier
On Mon, Dec 28, 2020 at 11:42:03AM +0100, Magnus Hagander wrote: > Not as much "tightly controlled" as "nobody's really bothered to grant any > permissions". Magnus, do I have an access to that? This is the second time I am crossing an issue with this issue, but I don't really know if I should ac

Re: Proposed patch for key managment

2020-12-28 Thread Fabien COELHO
I want to repeat here what I said in another thread: I think ultimately we will need three commands to control the keys. First, there is the cluster_key_command, which we have now. Second, I think we will need an optional command which returns random bytes --- this would allow users to get ra

Re: New IndexAM API controlling index vacuum strategies

2020-12-28 Thread Peter Geoghegan
On Sun, Dec 27, 2020 at 11:41 PM Peter Geoghegan wrote: > I experimented with this today, and I think that it is a good way to > do it. I like the idea of choose_vacuum_strategy() understanding that > heap pages that are subject to many non-HOT updates have a "natural > extra capacity for LP_DEAD

Re: trailing junk in numeric literals

2020-12-28 Thread Tom Lane
Peter Eisentraut writes: > I was surprised to find that this doesn't error: > => select 100a; >a > - > 100 > I suspect this and similar cases used to error before aliases without AS > were introduced. But now this seems possibly problematic. Should we > try to handle this better? M

trailing junk in numeric literals

2020-12-28 Thread Peter Eisentraut
I was surprised to find that this doesn't error: => select 100a; a - 100 I suspect this and similar cases used to error before aliases without AS were introduced. But now this seems possibly problematic. Should we try to handle this better?

Re: Better client reporting for "immediate stop" shutdowns

2020-12-28 Thread Tom Lane
Andres Freund writes: > I don't think it needs to be done right now, but I again want to suggest > it'd be nice if we split log levels into a bitmask. If we bits, separate > from the log level, for do-not-log-to-client and do-not-log-to-server > some of this code would imo look nicer. Hmm, maybe.

Re: Better client reporting for "immediate stop" shutdowns

2020-12-28 Thread Andres Freund
Hi, On 2020-12-28 13:25:14 -0500, Tom Lane wrote: > The most straightforward way to do that is to introduce a new error > level. Having to renumber existing levels is a bit of a pain, but > I'm not aware of anything that should break in source-code terms. > We make similar ABI breaks in every maj

Re: Better client reporting for "immediate stop" shutdowns

2020-12-28 Thread Tom Lane
Andres Freund writes: > On 2020-12-26 13:37:15 -0500, Tom Lane wrote: >>> I'd like to not log all these repeated messages into the server >>> log. It's quite annoying to have to digg through thousands of lines of >>> repeated "terminating connection..." >> Hm. That's an orthogonal issue, but cer

Re: pgsql: Fix memory leak in plpgsql's CALL processing.

2020-12-28 Thread Tom Lane
I wrote: > so it looks like a memory clobber --- probably some data structure is > being built in the wrong memory context. Will look closer shortly. Brain-dead typo ... where's my brown paper bag? regards, tom lane

Re: pgsql: Add pg_alterckey utility to change the cluster key

2020-12-28 Thread Bruce Momjian
On Mon, Dec 28, 2020 at 10:09:11AM -0400, Fabien COELHO wrote: > Yep, my point is that it should be possible to have the whole key management > outside of postgres. I think this kind of discussion has to happen in a different thread, parhsps: https://www.postgresql.org/message-id/flat/20

Re: pgsql: Fix memory leak in plpgsql's CALL processing.

2020-12-28 Thread Tom Lane
Peter Eisentraut writes: > This change has broken the following test case (reduced from actual > production code): Thanks for the report. What I see here is ERROR: unsupported target type: 2139062143 so it looks like a memory clobber --- probably some data structure is being built in the wro

Re: pgsql: Add pg_alterckey utility to change the cluster key

2020-12-28 Thread Fabien COELHO
Hello Bruce, I put the thread back on hackers. The first two keys are stored in pg_cryptokeys/ in the data directory, while the third one is retrieved using a GUC for validation at server startup for the other two. Do we necessarily have to store the first level keys within the data directo

Re: a misbehavior of partition row movement (?)

2020-12-28 Thread Arne Roland
While I'd agree that simply deleting with "on delete cascade" on tuple rerouting is a strong enough violation of the spirit of partitioning changing that behavior, I am surprised by the idea to make a differentiation between fks and other triggers. The way user defined triggers work, is a violat

Re: New Table Access Methods for Multi and Single Inserts

2020-12-28 Thread Bharath Rupireddy
On Fri, Dec 25, 2020 at 8:10 AM Justin Pryzby wrote: > On Thu, Dec 24, 2020 at 05:48:42AM +0530, Bharath Rupireddy wrote: > > I'm not posting the updated 0002 to 0004 patches, I plan to do so > > after a couple of reviews happen on the design of the APIs in 0001. > > > > Thoughts? > > Are you fami

Re: EXPLAIN/EXPLAIN ANALYZE REFRESH MATERIALIZED VIEW

2020-12-28 Thread Bharath Rupireddy
On Tue, Dec 22, 2020 at 7:01 PM Bharath Rupireddy wrote: > Currently, $subject is not allowed. We do plan the mat view query > before every refresh. I propose to show the explain/explain analyze of > the select part of the mat view in case of Refresh Mat View(RMV). It > will be useful for the user

Re: On login trigger: take three

2020-12-28 Thread Masahiko Sawada
On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule wrote: > > > > so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule > napsal: >> >> Hi >> >> >>> Thank you. >>> I have applied all your fixes in on_connect_event_trigger-12.patch. >>> >>> Concerning enable_client_connection_trigger GUC, I think that it is

Re: Parallel Inserts in CREATE TABLE AS

2020-12-28 Thread Bharath Rupireddy
On Mon, Dec 28, 2020 at 11:24 AM vignesh C wrote: > Test comments are detailed in a few cases and in few others it is not > detailed for similar kinds of parallelism selected tests. I felt we could > make the test comments consistent across the file. Modified the test case description in the v1

Re: Parallel Inserts in CREATE TABLE AS

2020-12-28 Thread Bharath Rupireddy
On Mon, Dec 28, 2020 at 1:16 AM Zhihong Yu wrote: > For v16-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch: > > + if (ignore && > + (root->parse->CTASParallelInsInfo & > +CTAS_PARALLEL_INS_TUP_COST_CAN_IGN)) > > I wonder why CTAS_PARALLEL_INS_TUP_COST_CAN

Re: [PATCH] Identify LWLocks in tracepoints

2020-12-28 Thread Masahiko Sawada
Hi Craig, On Sat, Dec 19, 2020 at 2:00 PM Craig Ringer wrote: > > Hi all > > The attached patch set follows on from the discussion in [1] "Add LWLock > blocker(s) information" by adding the actual LWLock* and the numeric tranche > ID to each LWLock related TRACE_POSTGRESQL_foo tracepoint. > > T

Re: OpenSSL connection setup debug callback issue

2020-12-28 Thread Masahiko Sawada
Hi Daniel, On Thu, Dec 10, 2020 at 10:43 PM Daniel Gustafsson wrote: > > I went looking at the SSL connection state change information callback we > install when setting up connections with OpenSSL, and I wasn't getting the > state changes I expected. Turns out we install it at the tail end of s

Re: [Doc Patch] Clarify that CREATEROLE roles can GRANT default roles

2020-12-28 Thread Masahiko Sawada
Hi Michael, On Sat, Nov 28, 2020 at 7:50 AM Michael Banck wrote: > > Hi, > > https://www.postgresql.org/docs/current/default-roles.html mentions the > "Administrator" several times, but does not specify it further. I > understand that it is mentioned elsewhere [1], but I think it would be > benef

Re: Rename of triggers for partitioned tables

2020-12-28 Thread Masahiko Sawada
Hi Arne, On Fri, Nov 27, 2020 at 9:20 AM Arne Roland wrote: > > I'm sorry for the spam. Here a patch with updated comments, I missed one > before. > You sent in your patch, recursive_tgrename.2.patch to pgsql-hackers on Nov 27, but you did not post it to the next CommitFest[1]. If this was int

Re: Disable WAL logging to speed up data loading

2020-12-28 Thread Simon Riggs
On Fri, 25 Dec 2020 at 07:09, Michael Paquier wrote: > > On Thu, Dec 03, 2020 at 03:52:47AM +, tsunakawa.ta...@fujitsu.com wrote: > > The code looks good, and the performance seems to be nice, so I > > marked this ready for committer. > > FWIW, I am extremely afraid of this proposal because th

Re: postgres_fdw - cached connection leaks if the associated user mapping/foreign server is dropped

2020-12-28 Thread Fujii Masao
On 2020/12/24 23:45, Fujii Masao wrote: On 2020/12/24 23:30, Bharath Rupireddy wrote: On Thu, Dec 24, 2020 at 7:43 PM Fujii Masao wrote: Even when we're in the midst of transaction, if that transaction has not used the cached connections yet, we close them immediately. So, to make the com

Re: [doc] adding way to examine the plan type of prepared statements

2020-12-28 Thread Masahiko Sawada
Hi Torikoshi-san, On Thu, Nov 19, 2020 at 3:19 PM torikoshia wrote: > > On 2020-11-18 11:04, torikoshia wrote: > > Hi, > > > > > > Currently, EXPLAIN is the only way to know whether the plan is generic > > or custom according to the manual of PREPARE. > > > > https://www.postgresql.org/docs/dev

Re: doc review for v14

2020-12-28 Thread Magnus Hagander
On Sun, Dec 27, 2020 at 9:26 PM Justin Pryzby wrote: > On Thu, Dec 24, 2020 at 05:12:02PM +0900, Michael Paquier wrote: > > 0001-pgindent-typos.not-a-patch touches pg_bsd_indent. > > I'm hoping that someone will apply it there, but I realize that access to > its > repository is tightly controlled

Re: Table AM modifications to accept column projection lists

2020-12-28 Thread Masahiko Sawada
Hi Soumyadeep, On Sat, Nov 14, 2020 at 3:02 AM Soumyadeep Chakraborty wrote: > > Hello, > > This patch introduces a set of changes to the table AM APIs, making them > accept a column projection list. That helps columnar table AMs, so that > they don't need to fetch all columns from disk, but only

Re: PATCH: Report libpq version and configuration

2020-12-28 Thread Masahiko Sawada
Hi Craig, On Wed, Nov 11, 2020 at 1:11 PM Craig Ringer wrote: > > On Tue, Nov 10, 2020 at 2:22 PM Craig Ringer > wrote: >> >> >> The main things I'd really like to get in place are a way to get the version >> as an ELF data symbol, and a simple way to ID the binary. >> >> So the minimal change

Re: Disable WAL logging to speed up data loading

2020-12-28 Thread Masahiko Sawada
On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com wrote: > > Hello Sawada-San > > > On Monday, December 28, 2020 2:29 PM Masahiko Sawada > wrote: > > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > I've made a new patch v05 that took in comments to f

Re: pgsql: Fix memory leak in plpgsql's CALL processing.

2020-12-28 Thread Peter Eisentraut
This change has broken the following test case (reduced from actual production code): CREATE OR REPLACE PROCEDURE prc_inout_test_inner(INOUT x integer) LANGUAGE plpgsql AS $procedure$ BEGIN COMMIT; x := 2; END; $procedure$; CREATE OR REPLACE PROCEDURE prc_inout_test_main() LANGUAGE plpgsql

Re: Parallel copy

2020-12-28 Thread vignesh C
On Tue, Nov 3, 2020 at 2:28 PM Amit Kapila wrote: > > On Mon, Nov 2, 2020 at 12:40 PM Heikki Linnakangas wrote: > > > > On 02/11/2020 08:14, Amit Kapila wrote: > > > On Fri, Oct 30, 2020 at 10:11 PM Heikki Linnakangas > > > wrote: > > >> > > >> In this design, you don't need to keep line bounda

RE: [HACKERS] logical decoding of two-phase transactions

2020-12-28 Thread osumi.takami...@fujitsu.com
Hi, Amit-San On Thursday, Dec 24, 2020 2:35 PM Amit Kapila wrote: > On Wed, Dec 23, 2020 at 3:08 PM Ajin Cherian wrote: > > > > > Can you please update the patch for the points we agreed upon? > > > > Changed and attached. > > > > Thanks, I have looked at these patches again and it seems patch

Re: [PATCH] TAP test showing that pg_replication_slot_advance() works on standby

2020-12-28 Thread Masahiko Sawada
Hi Craig, On Tue, Nov 10, 2020 at 2:32 PM Craig Ringer wrote: > > Hi all > > The attached patch adds a test to the TAP logical decoding suite to show that > pg_replication_slot_advance() works on a server in standby mode. > > I didn't add a full demonstration of how to do failover with logical s

Re: Parallel Full Hash Join

2020-12-28 Thread Masahiko Sawada
Hi Melanie, On Thu, Nov 5, 2020 at 7:34 AM Melanie Plageman wrote: > > I've attached a patch with the corrections I mentioned upthread. > I've gone ahead and run pgindent, though, I can't say that I'm very > happy with the result. > > I'm still not quite happy with the name > BarrierArriveAndDeta

Lazy JIT IR code generation to increase JIT speed with partitions

2020-12-28 Thread Luc Vlaming
Hi, I would like to propose a small patch to the JIT machinery which makes the IR code generation lazy. The reason for postponing the generation of the IR code is that with partitions we get an explosion in the number of JIT functions generated as many child tables are involved, each with the

Re: Add table AM 'tid_visible'

2020-12-28 Thread Masahiko Sawada
On Tue, Nov 3, 2020 at 5:23 PM Jinbao Chen wrote: > > Hi Andres, > > > > > Yea, it's something we should improve. Have you checked if this has > > > performance impact for heap? Should we also consider planning costs? > > Since the visibility map is very small, all pages of the visibility map will

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-28 Thread Tang, Haiying
Hi Amit, >I think one table with a varying amount of data is sufficient for the vacuum >test. >I think with more number of tables there is a greater chance of variation. >We have previously used multiple tables in one of the tests because of >the Truncate operation (which uses DropRelFileNodes