Re: Parallel copy

2020-07-12 Thread Rafia Sabih
On Sat, 11 Jul 2020 at 08:55, Bharath Rupireddy wrote: > > Thanks Vignesh for the review. Addressed the comments in 0006 patch. > > > > > we can create a local variable and use in place of > > cstate->pcdata->curr_data_block. > > Done. > > > + if (cstate->raw_buf_index + sizeof(fld_count) >=

Re: WIP: BRIN multi-range indexes

2020-07-12 Thread Sascha Kuhl
Thanks, I see there is some understanding, though. Tomas Vondra schrieb am So., 12. Juli 2020, 01:30: > On Sat, Jul 11, 2020 at 03:32:43PM +0200, Sascha Kuhl wrote: > >Tomas Vondra schrieb am Sa., 11. Juli > 2020, > >13:24: > > > >> On Fri, Jul 10, 2020 at 04:44:41PM +0200, Sascha Kuhl wrote: >

Re: Towards easier AMs: Cleaning up inappropriate use of name "relkind"

2020-07-12 Thread Michael Paquier
On Sat, Jul 11, 2020 at 03:32:55PM -0400, Tom Lane wrote: > Mark Dilger writes: >> There are code paths where relkind is sometimes '\0' under normal, >> non-exceptional conditions. This happens in >> >> allpaths.c: set_append_rel_size >> rewriteHandler.c: view_query_is_auto_updatable >

Re: Default setting for enable_hashagg_disk

2020-07-12 Thread Tomas Vondra
On Sat, Jul 11, 2020 at 08:47:54PM -0400, Tom Lane wrote: Tomas Vondra writes: I don't know, but one of the main arguments against simply suggesting people to bump up work_mem (if they're hit by the hashagg spill in v13) was that it'd increase overall memory usage for them. It seems strange to

Re: Default setting for enable_hashagg_disk

2020-07-12 Thread Tomas Vondra
On Sat, Jul 11, 2020 at 10:26:22PM -0700, David G. Johnston wrote: On Saturday, July 11, 2020, Tom Lane wrote: "David G. Johnston" writes: > On Sat, Jul 11, 2020 at 5:47 PM Tom Lane wrote: >> It seems like a lot of the disagreement here is focused on Peter's >> proposal to make hash_mem_mult

Re: A patch for get origin from commit_ts.

2020-07-12 Thread Michael Paquier
On Fri, Jul 10, 2020 at 10:06:06AM +0900, Michael Paquier wrote: > Please note that I have switched the patch as ready for committer. So > I'll try to get that done, with roident as attribute name. If > somebody prefers a different name or has an objection, please feel > free to chime in. Hearin

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-12 Thread vignesh C
On Fri, Jul 10, 2020 at 8:51 AM Amit Langote wrote: > > Hi Bharath, > Here the numbers with the updated patch: > > HEADpatched (v2) > foo5 8.56.1 > foo10 149.4 > foo20 25 16.7 > Patch applies cleanly, make check & make check-world pass

Re: calling procedures is slow and consumes extra much memory against calling function

2020-07-12 Thread Pavel Stehule
so 11. 7. 2020 v 7:38 odesílatel Pavel Stehule napsal: > > > čt 9. 7. 2020 v 8:28 odesílatel Amit Khandekar > napsal: > >> On Wed, 17 Jun 2020 at 13:54, Pavel Stehule >> wrote: >> > >> > >> > >> > st 17. 6. 2020 v 7:52 odesílatel Amit Khandekar >> napsal: >> >> >> >> On Wed, 10 Jun 2020 at 17:

Re: Avoid useless retrieval of defaults and check constraints in pg_dump -a

2020-07-12 Thread Justin Pryzby
On Sun, Jul 12, 2020 at 07:48:50AM +0200, Julien Rouhaud wrote: > Currently, getTableAttrs() always retrieves info about columns defaults and > check constraints, while this will never be used if --data-only option if > used. > This seems like a waste of resources, so here's a patch to skip those

Re: Improving psql slash usage help message

2020-07-12 Thread Tom Lane
Hamid Akhtar writes: > On Sun, Apr 26, 2020 at 1:03 AM David G. Johnston < > david.g.johns...@gmail.com> wrote: >> On Sat, Apr 25, 2020 at 12:29 PM Hamid Akhtar >> wrote: >>> "\dE" displays the list with a "List of relations" heading whereas "\det" >>> displays "List of foreign tables". So, to di

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-12 Thread Dilip Kumar
On Mon, Jul 6, 2020 at 11:43 AM Dilip Kumar wrote: > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila wrote: > > > > On Sun, Jul 5, 2020 at 4:47 PM Dilip Kumar wrote: > > > > > > On Sat, Jul 4, 2020 at 11:35 AM Amit Kapila > > > wrote: > > > > > > > > > > > 9. > > > > +ReorderBufferHandleConcurre

Re: Online checksums verification in the backend

2020-07-12 Thread Justin Pryzby
Small language fixes in comments and user-facing docs. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 88efb38556..39596db193 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26162,7 +26162,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), '

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-12 Thread Andrey Lepikhov
22.06.2020 17:11, Ashutosh Bapat пишет: On Wed, 17 Jun 2020 at 11:54, Andrey V. Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 6/15/20 10:26 AM, Ashutosh Bapat wrote: > Thanks Andrey for the patch. I am glad that the patch has taken care > of some corner cases already

Re: explain HashAggregate to report bucket and memory stats

2020-07-12 Thread Daniel Gustafsson
> On 9 Apr 2020, at 00:24, Jeff Davis wrote: > > On Wed, 2020-04-08 at 16:00 -0500, Justin Pryzby wrote: >> 90% of the initial goal of this patch was handled by instrumentation >> added by >> "hash spill to disk" (1f39bce02), but this *also* adds: >> >> - separate accounting for tuples vs hashta

Re: Warn when parallel restoring a custom dump without data offsets

2020-07-12 Thread Tom Lane
David Gilman writes: > On Thu, Jul 02, 2020 at 05:25:21PM -0400, Tom Lane wrote: >> I guess I'm completely confused about the purpose of these patches. >> Far from coping with the situation of an unseekable file, they appear >> to change pg_restore so that it fails altogether if it can't seek >> i

Re: should INSERT SELECT use a BulkInsertState?

2020-07-12 Thread Daniel Gustafsson
> On 4 Jun 2020, at 19:30, Andres Freund wrote: > On 2020-05-08 02:25:45 -0500, Justin Pryzby wrote: >> Seems to me it should, at least conditionally. At least if there's a >> function >> scan or a relation or .. > > Well, the problem is that this can cause very very significant > regressions.

Re: ALTER TABLE validate foreign key dependency problem

2020-07-12 Thread Simon Riggs
On Sun, 12 Jul 2020 at 05:51, David Rowley wrote: > > I also considered that we could just delay all foreign key validations > > until phase 3, but I ended up just doing then only when a rewrite is > > pending. > > I still wonder if it's best to delay the validation of the foreign key > regardle

Re: Parallel grouping sets

2020-07-12 Thread Daniel Gustafsson
> On 25 Mar 2020, at 15:35, Pengzhou Tang wrote: > Thanks a lot, the patch has a memory leak in the lookup_hash_entries, it uses > a list_concat there > and causes a 64-byte leak for every tuple, has fixed that. > > Also, resolved conflicts and rebased the code. While there hasn't been a revie

Re: Libpq support to connect to standby server as priority

2020-07-12 Thread Daniel Gustafsson
> On 6 Jul 2020, at 14:19, Greg Nancarrow wrote: > >> This patch no longer applies, can you please submit a rebased version? I've >> marked the entry as Waiting on Author in the meantime. >> > > Here's a rebased version of the patch. Thanks, but now the tests no longer work as the nodes in th

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-12 Thread Thomas Munro
On Mon, Jul 13, 2020 at 1:13 AM vignesh C wrote: > On Fri, Jul 10, 2020 at 8:51 AM Amit Langote wrote: > > Here the numbers with the updated patch: > > > > HEADpatched (v2) > > foo5 8.56.1 > > foo10 149.4 > > foo20 25 16.7 > > > > Patc

Re: WIP: BRIN multi-range indexes

2020-07-12 Thread Alvaro Herrera
On 2020-Jul-10, Tomas Vondra wrote: > > postgres(1:12801)=# select * from brin_page_items(get_raw_page('mul', > > 2), 'mul'); > > -[ RECORD 1 > > ]-- > > ---

Re: WIP: BRIN multi-range indexes

2020-07-12 Thread Tomas Vondra
On Sun, Jul 12, 2020 at 07:58:54PM -0400, Alvaro Herrera wrote: On 2020-Jul-10, Tomas Vondra wrote: > postgres(1:12801)=# select * from brin_page_items(get_raw_page('mul', > 2), 'mul'); > -[ RECORD 1 ]-

Re: WIP: BRIN multi-range indexes

2020-07-12 Thread Alvaro Herrera
On 2020-Jul-13, Tomas Vondra wrote: > On Sun, Jul 12, 2020 at 07:58:54PM -0400, Alvaro Herrera wrote: > > Maybe we can try to handle this with some other function that interprets > > the bytea in 'value' and returns a user-readable text. I think it'd > > have to be a superuser-only function, bec

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-12 Thread Bharath Rupireddy
Thanks Thomas for checking this feature. > On Mon, Jul 13, 2020 at 4:06 AM Thomas Munro wrote: > > This error showed up when cfbot tried it: > > COPY BINARY stud_emp FROM > '/home/travis/build/postgresql-cfbot/postgresql/src/test/regress/results/stud_emp.data'; > +ERROR: could not read from COP

Re: Physical slot restart_lsn advances incorrectly requiring restore from archive

2020-07-12 Thread Kyotaro Horiguchi
Hello, John. At Fri, 10 Jul 2020 20:44:30 +, "Hsu, John" wrote in > Hi hackers, > > We believe we’re seeing a problem with how physical slot’s restart_lsn is > advanced leading to the replicas needing to restore from archive in order for > replication to resume. > The logs below are fro

Re: should INSERT SELECT use a BulkInsertState?

2020-07-12 Thread Justin Pryzby
On Thu, Jun 04, 2020 at 10:30:47AM -0700, Andres Freund wrote: > On 2020-05-08 02:25:45 -0500, Justin Pryzby wrote: > > Seems to me it should, at least conditionally. At least if there's a > > function > > scan or a relation or .. > > Well, the problem is that this can cause very very significan

Re: archive status ".ready" files may be created too early

2020-07-12 Thread Kyotaro Horiguchi
Hello. # Sorry, I wrongly thought that I replied to this thread.. At Tue, 7 Jul 2020 09:02:56 +, "matsumura@fujitsu.com" wrote in > At Monday, July 6, 2020 05:13:40 +, "Kyotaro Horiguchi > " wrote in > > > > after WAL buffer is filled up to the requested position. So when it > >

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-12 Thread Amit Langote
On Mon, Jul 13, 2020 at 10:19 AM Bharath Rupireddy wrote: > > On Mon, Jul 13, 2020 at 4:06 AM Thomas Munro wrote: > > > > This error showed up when cfbot tried it: > > > > COPY BINARY stud_emp FROM > > '/home/travis/build/postgresql-cfbot/postgresql/src/test/regress/results/stud_emp.data'; > > +

Re: [PATCH] Performance Improvement For Copy From Binary Files

2020-07-12 Thread Bharath Rupireddy
> > > This is due to the recent commit > > cd22d3cdb9bd9963c694c01a8c0232bbae3ddcfb, in which we restricted the > > raw_buf and line_buf allocations for binary files. Since we are using > > raw_buf for this performance improvement feature, now, it's enough to > > restrict only line_buf for binary f

Re: Creating a function for exposing memory usage of backend process

2020-07-12 Thread Fujii Masao
On 2020/07/10 17:32, torikoshia wrote: On 2020-07-09 02:03, Andres Freund wrote: Hi, I think this is an incredibly useful feature. Thanks for your kind comments and suggestion! On 2020-07-07 22:02:10 +0900, torikoshia wrote: > There can be multiple memory contexts with the same name. S

Re: POC and rebased patch for CSN based snapshots

2020-07-12 Thread Andrey V. Lepikhov
On 7/4/20 7:56 PM, movead...@highgo.ca wrote: As far as I know about Clock-SI, left part of the blue line will setup as a snapshot if master require a snapshot at time t1. But in fact data A should in snapshot but not and data B should out of snapshot but not. If thi

Re: Bug with indexes on whole-row expressions

2020-07-12 Thread Ashutosh Bapat
On Mon, Jun 29, 2020 at 3:43 PM Laurenz Albe wrote: > > CREATE TABLE boom (a integer, b integer); > > -- index on whole-row expression > CREATE UNIQUE INDEX ON boom ((boom)); > > INSERT INTO boom VALUES >(1, 2), >(1, 3); > > ALTER TABLE boom DROP b; > > TABLE boom; > > a > --- > 1 > 1 >

Re: Retry Cached Remote Connections for postgres_fdw in case remote backend gets killed/goes away

2020-07-12 Thread Ashutosh Bapat
On Wed, Jul 8, 2020 at 6:10 PM Bharath Rupireddy wrote: > > I couldn't think of adding a test case to the existing postgres_fdw > regression test suite with an automated scenario of the remote backend > getting killed. You could get a backend's PID using PQbackendPID and then kill it by calling p

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-12 Thread Amit Kapila
On Fri, Jul 10, 2020 at 3:37 PM Dilip Kumar wrote: > > On Sat, Jul 4, 2020 at 11:35 AM Amit Kapila wrote: > > > > > > 8. We can't stream the transaction before we reach the > > SNAPBUILD_CONSISTENT state because some other output plugin can apply > > those changes unlike what we do with pgoutput

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-12 Thread Dilip Kumar
On Mon, Jul 13, 2020 at 10:14 AM Amit Kapila wrote: > > On Fri, Jul 10, 2020 at 3:37 PM Dilip Kumar wrote: > > > > On Sat, Jul 4, 2020 at 11:35 AM Amit Kapila wrote: > > > > > > > > > 8. We can't stream the transaction before we reach the > > > SNAPBUILD_CONSISTENT state because some other outpu

Re: max_slot_wal_keep_size and wal_keep_segments

2020-07-12 Thread Fujii Masao
On 2020/07/09 1:20, Alvaro Herrera wrote: On 2020-Jul-09, Fujii Masao wrote: I attached the patch that renames wal_keep_segments to wal_keep_size. Looks good in a quick once-over. Just two small wording comments: Thanks for review comments! Independently of max_wal_size, -

Re: max_slot_wal_keep_size and wal_keep_segments

2020-07-12 Thread Fujii Masao
On 2020/07/09 13:47, Kyotaro Horiguchi wrote: At Thu, 9 Jul 2020 00:37:57 +0900, Fujii Masao wrote in On 2020/07/02 2:18, David Steele wrote: On 7/1/20 10:54 AM, Alvaro Herrera wrote: On 2020-Jul-01, Fujii Masao wrote: On 2020/07/01 12:26, Alvaro Herrera wrote: On 2020-Jun-30, Fujii

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-12 Thread Dilip Kumar
On Sun, Jul 12, 2020 at 9:56 PM Dilip Kumar wrote: > > On Mon, Jul 6, 2020 at 11:43 AM Dilip Kumar wrote: > > > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila wrote: > > > > > > On Sun, Jul 5, 2020 at 4:47 PM Dilip Kumar wrote: > > > > > > > > On Sat, Jul 4, 2020 at 11:35 AM Amit Kapila > > >

Re: GSSENC'ed connection stalls while reconnection attempts.

2020-07-12 Thread Kyotaro Horiguchi
At Fri, 10 Jul 2020 12:01:10 -0400, Tom Lane wrote in > Kyotaro Horiguchi writes: > > If psql connected using GSSAPI auth and server restarted, reconnection > > sequence stalls and won't return. > > Yeah, reproduced here. (I wonder if there's any reasonable way to > exercise this scenario in s

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-12 Thread Amit Kapila
On Mon, Jul 13, 2020 at 10:40 AM Dilip Kumar wrote: > > On Mon, Jul 13, 2020 at 10:14 AM Amit Kapila wrote: > > > > On Fri, Jul 10, 2020 at 3:37 PM Dilip Kumar wrote: > > > > > > On Sat, Jul 4, 2020 at 11:35 AM Amit Kapila > > > wrote: > > > > > > > > > > > > 8. We can't stream the transaction

Re: partition routing layering in nodeModifyTable.c

2020-07-12 Thread Amit Langote
On Wed, Jul 1, 2020 at 6:56 PM Daniel Gustafsson wrote: > > > On 2 Mar 2020, at 06:08, Amit Langote wrote: > > > > On Mon, Mar 2, 2020 at 4:43 AM Tom Lane wrote: > >> Amit Langote writes: > >>> Rebased again. > >> > >> Seems to need that again, according to cfbot :-( > > > > Thank you, done. >

Re: WIP: BRIN multi-range indexes

2020-07-12 Thread Masahiko Sawada
On Mon, 13 Jul 2020 at 09:33, Alvaro Herrera wrote: > > On 2020-Jul-13, Tomas Vondra wrote: > > > On Sun, Jul 12, 2020 at 07:58:54PM -0400, Alvaro Herrera wrote: > > > > Maybe we can try to handle this with some other function that interprets > > > the bytea in 'value' and returns a user-readable

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-12 Thread Amit Kapila
On Mon, Jul 13, 2020 at 10:47 AM Dilip Kumar wrote: > > On Sun, Jul 12, 2020 at 9:56 PM Dilip Kumar wrote: > > > > On Mon, Jul 6, 2020 at 11:43 AM Dilip Kumar wrote: > > > > > > On Mon, Jul 6, 2020 at 11:31 AM Amit Kapila > > > wrote: > > > > > > > > > > > > > 10. I have got the below failure

Re: Don't choke on files that are removed while pg_rewind runs.

2020-07-12 Thread Justin Pryzby
commit b36805f3c54fe0e50e58bb9e6dad66daca46fbf6 Author: Heikki Linnakangas Date: Sun Jun 28 21:35:51 2015 +0300 ... |@@ -175,22 +175,31 @@ libpqProcessFileList(void) |pg_fatal("unexpected result set while fetching file list\n"); | |/* Read result to local variables */

Re: proposal: possibility to read dumped table's name from file

2020-07-12 Thread Pavel Stehule
ne 12. 7. 2020 v 1:06 odesílatel Justin Pryzby napsal: > On Mon, Jul 06, 2020 at 06:34:15AM +0200, Pavel Stehule wrote: > > >> > Comment support is still missing but easily added :) > > >> Still missing from the latest patch. > > > > > > I can implement a comment support. But I am not sure about

Re: Don't choke on files that are removed while pg_rewind runs.

2020-07-12 Thread Daniel Gustafsson
> On 13 Jul 2020, at 08:10, Justin Pryzby wrote: > Every other access to "res" in this loop is to res(i), which I believe is what > was intended here, too. Currently, it will dumbly loop but skip *every* row > if > the 2nd column (1: size) of the first row (0) is null. Yeah, I agree with that,

Re: POC and rebased patch for CSN based snapshots

2020-07-12 Thread movead...@highgo.ca
>I have doubts that I fully understood your question, but still. >What real problems do you see here? Transaction t1 doesn't get state of >shard2 until time at node with shard2 won't reach start time of t1. >If transaction, that inserted B wants to know about it position in time >relatively to t1