Re: Re: A separate table level option to control compression

2019-04-04 Thread Michael Paquier
On Wed, Apr 03, 2019 at 03:23:33PM +0900, Michael Paquier wrote: > It seems to me that c251336 should have done all those things from the > start... In other terms, isn't that a bug and something that we > should fix and back-patch? I'll begin a new thread about that to > catch more attention, w

Re: speeding up planning with partitions

2019-04-04 Thread Amit Langote
On 2019/04/02 14:50, Amit Langote wrote: > Attached patch is only for HEAD this time. I'll post one for PG 11 (if > you'd like) once we reach consensus on the best thing to do here is. While we're on the topic of the relation between constraint exclusion and partition pruning, I'd like to (re-) p

Re: [HACKERS] Block level parallel vacuum

2019-04-04 Thread Kyotaro HORIGUCHI
Thank you for the rebased version. At Fri, 5 Apr 2019 13:59:36 +0900, Masahiko Sawada wrote in > Thank you for the notice. Rebased. +integer + + + Specifies parallel degree for PARALLEL option. The + value must be at least 1. If the parallel degree + integer is omit

Failure in contrib test _int on loach

2019-04-04 Thread Thomas Munro
Hi, This is a strange failure: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=loach&dt=2019-04-05%2005%3A15%3A00 test _int ... FAILED 649 ms = pgsql.build/contrib/intarray/regression.diffs === diff -U3 /usr/home/pgbf/buildro

Re: Strange coding in _mdfd_openseg()

2019-04-04 Thread Thomas Munro
On Thu, Apr 4, 2019 at 4:16 PM Kyotaro HORIGUCHI wrote: > At Wed, 3 Apr 2019 13:47:46 -0700, Andres Freund wrote > in <20190403204746.2yumq7c2mirmo...@alap3.anarazel.de> > > Yea, I totally agree it's weird. I'm not sure if I'd go for an assertion > > of equality, or just invert the >= (which I a

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-05 08:38:34 +0300, Darafei "Komяpa" Praliaskouski wrote: > On Fri, Apr 5, 2019 at 6:58 AM Tom Lane wrote: > > > Andres Freund writes: > > > I think the right approach would be to do all of this in heap_insert and > > > heap_multi_insert. Whenever starting to work on a page, if IN

Re: GiST VACUUM

2019-04-04 Thread Andrey Borodin
Hi! > 4 апр. 2019 г., в 20:15, Heikki Linnakangas написал(а): > > On 25/03/2019 15:20, Heikki Linnakangas wrote: >> On 24/03/2019 18:50, Andrey Borodin wrote: >>> I was working on new version of gist check in amcheck and understand one >>> more thing: >>> >>> /* Can this page be recycled yet?

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Komяpa
On Fri, Apr 5, 2019 at 6:58 AM Tom Lane wrote: > Andres Freund writes: > > I think the right approach would be to do all of this in heap_insert and > > heap_multi_insert. Whenever starting to work on a page, if INSERT_FROZEN > > is specified, remember whether it is either currently empty, or is

Re: Why does "toast_tuple_target" allow values below TOAST_TUPLE_TARGET?

2019-04-04 Thread David Rowley
On Fri, 5 Apr 2019 at 18:25, Amit Langote wrote: > > On 2019/04/05 14:09, David Rowley wrote: > > Or rather, why does the reloption allow values below the compile-time > > constant? > > Maybe there is already a discussion in progress on the topic? > > * Caveats from reloption toast_tuple_target *

Re: Why does "toast_tuple_target" allow values below TOAST_TUPLE_TARGET?

2019-04-04 Thread Amit Langote
On 2019/04/05 14:09, David Rowley wrote: > Or rather, why does the reloption allow values below the compile-time > constant? Maybe there is already a discussion in progress on the topic? * Caveats from reloption toast_tuple_target * https://www.postgresql.org/message-id/flat/CABOikdMt%3DmOtzW_ax

Re: [GSoC] application ideas

2019-04-04 Thread Andrey Borodin
Hi! We are discussing GSoC details offlist, but I'll put some recommendations on your proposal to the list. > 3 апр. 2019 г., в 2:53, pantilimonov misha написал(а): > > Andrey, thank you for your reply. > >> 24.03.2019, 12:12, "Andrey Borodin" : >> >> I like the idea of more efficient Buffer

Why does "toast_tuple_target" allow values below TOAST_TUPLE_TARGET?

2019-04-04 Thread David Rowley
Or rather, why does the reloption allow values below the compile-time constant? It looks like we currently allow values as low as 128. The problem there is that heap_update() and heap_prepare_insert() both only bother calling toast_insert_or_update() when the tuple's length is above TOAST_TUPLE_T

Re: [HACKERS] Block level parallel vacuum

2019-04-04 Thread Masahiko Sawada
On Fri, Apr 5, 2019 at 4:51 AM Robert Haas wrote: > > On Thu, Apr 4, 2019 at 6:28 AM Masahiko Sawada wrote: > > These patches conflict with the current HEAD. Attached the updated patches. > > They'll need another rebase. > Thank you for the notice. Rebased. Regards, -- Masahiko Sawada NIPPON

RE: Timeout parameters

2019-04-04 Thread Jamison, Kirk
On Thursday, April 4, 2019 5:20PM (GMT+9), Ryohei Nagaura wrote: > > From: Fabien COELHO > > * About socket_timeout v12 patch, I'm not sure there is a consensus. > I think so too. I just made the patch being able to be committed anytime. > > Finally, I rebased all the patches because they have c

Re: Caveats from reloption toast_tuple_target

2019-04-04 Thread Pavan Deolasee
On Thu, Apr 4, 2019 at 11:36 AM Michael Paquier wrote: > > > I mean that toast_tuple_target is broken as-is, because it should be > used on the new tuples of a relation as a threshold to decide if this > tuple should be toasted or not, but we don't actually use the > reloption value for that deci

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-04 23:57:58 -0400, Tom Lane wrote: > Andres Freund writes: > > I think the right approach would be to do all of this in heap_insert and > > heap_multi_insert. Whenever starting to work on a page, if INSERT_FROZEN > > is specified, remember whether it is either currently empty, or i

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-05 09:20:36 +0530, Pavan Deolasee wrote: > On Fri, Apr 5, 2019 at 9:05 AM Andres Freund wrote: > > I think the right approach would be to do all of this in heap_insert and > > heap_multi_insert. Whenever starting to work on a page, if INSERT_FROZEN > > is specified, remember whethe

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Tom Lane
Andres Freund writes: > I think the right approach would be to do all of this in heap_insert and > heap_multi_insert. Whenever starting to work on a page, if INSERT_FROZEN > is specified, remember whether it is either currently empty, or is > already marked as all-visible. If previously empty, mar

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-04 Thread Kyotaro HORIGUCHI
At Thu, 4 Apr 2019 10:52:59 -0400, Robert Haas wrote in > On Wed, Apr 3, 2019 at 10:03 PM Kyotaro HORIGUCHI > wrote: > > > * Insert log record, using delete or insert instead of update log > > > * when only one of the two buffers needs WAL-logging. If this were a > > > * HOT-update, redoing the

Re: fix the spelling mistakes of comments

2019-04-04 Thread Tom Lane
Michael Paquier writes: > On Thu, Apr 04, 2019 at 07:57:02AM -0400, Robert Haas wrote: >> On Wed, Apr 3, 2019 at 6:55 AM Liu, Huailing >> wrote: * This module contains the code for waiting and release★ of backends. >>> I think the word marked★ should be 'releasing'. >> It could be changed

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Pavan Deolasee
On Fri, Apr 5, 2019 at 8:37 AM Andres Freund wrote: > Hi, > > On 2019-04-05 00:06:04 -0300, Alvaro Herrera wrote: > > > > > Hmm, isn't there already a critical section in visibilitymap_set itself? > > There is, but the proposed code sets all visible on the page, and marks > the buffer dirty, befo

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Pavan Deolasee
Hi, On Fri, Apr 5, 2019 at 9:05 AM Andres Freund wrote: > > > I think the right approach would be to do all of this in heap_insert and > heap_multi_insert. Whenever starting to work on a page, if INSERT_FROZEN > is specified, remember whether it is either currently empty, or is > already marked

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-05 00:06:04 -0300, Alvaro Herrera wrote: > On 2019-Apr-04, Andres Freund wrote: > > I still think this is the wrong architecture. > > Hmm. I think the right approach would be to do all of this in heap_insert and heap_multi_insert. Whenever starting to work on a page, if INSERT_FRO

Re: speeding up planning with partitions

2019-04-04 Thread Amit Langote
On 2019/04/05 12:18, David Rowley wrote: > On Fri, 5 Apr 2019 at 16:09, Amit Langote > wrote: >> Although, we still have ways >> to go in terms of scaling generic plan execution to larger partition >> counts, solution(s) for which have been proposed by David but haven't made >> it into master yet

Re: Refactoring the checkpointer's fsync request queue

2019-04-04 Thread Thomas Munro
On Thu, Apr 4, 2019 at 11:47 PM Thomas Munro wrote: > Pushed. Thanks for all the work on this! I managed to break this today while testing with RELSEG_SIZE set to 1 block (= squillions of 8kb files). The problem is incorrect arguments to _mdfd_getseg(), in code added recently by me. Without th

Re: speeding up planning with partitions

2019-04-04 Thread David Rowley
On Fri, 5 Apr 2019 at 16:09, Amit Langote wrote: > Although, we still have ways > to go in terms of scaling generic plan execution to larger partition > counts, solution(s) for which have been proposed by David but haven't made > it into master yet. Is that a reference to the last paragraph in [1

RE: Problem during Windows service start

2019-04-04 Thread Higuchi, Daisuke
Hi, Thank you for picking up this and I'm sorry for delay reply. >> If wait_for_postmaster returns POSTMASTER_STILL_STARTING will it >> be correct to set the status of windows service to SERVICE_START_PENDING ? Yes, I think this is the best. Currently, I do not have good solution to change t

Re: speeding up planning with partitions

2019-04-04 Thread Amit Langote
On 2019/04/05 6:59, David Rowley wrote: > On Fri, 5 Apr 2019 at 07:33, Floris Van Nee wrote: >> I had a question about the performance of pruning of functions like now() >> and current_date. I know these are handled differently, as they cannot be >> excluded during the first phases of planning.

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-05 00:06:04 -0300, Alvaro Herrera wrote: > On 2019-Apr-04, Andres Freund wrote: > > > On 2019-04-04 12:23:08 -0700, Andres Freund wrote: > > > Also, how is this code even close to correct? > > > CheckAndSetPageAllVisible() modifies the buffer in a crucial way, and > > > there's no

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Alvaro Herrera
On 2019-Apr-04, Andres Freund wrote: > On 2019-04-04 12:23:08 -0700, Andres Freund wrote: > > Also, how is this code even close to correct? > > CheckAndSetPageAllVisible() modifies the buffer in a crucial way, and > > there's no WAL logging? Without even a comment arguing why that's OK (I > > don'

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > Kerberos tests are now failing for me (macOS). I'm seeing > > psql: error: could not connect to server: Over-size error packet sent by > the server. > not ok 3 - GSS encryption without auth > > # Failed test 'GSS encryp

RE: libpq debug log

2019-04-04 Thread Iwata, Aya
Hi, > The basic idea being: > > - Each line is a whole message. > - The line begins with <<< for a message received and >>> for a message sent. > - Strings in single quotes are those sent/received as a fixed number of bytes. > - Strings in double quotes are those sent/received as a string. > - 4

RE: Speed up transaction completion faster after many relations are accessed in a transaction

2019-04-04 Thread Imai, Yoshikazu
On Fri, Apr 5, 2019 at 0:05 AM, Tsunakawa, Takayuki wrote: > From: Peter Eisentraut [mailto:peter.eisentr...@2ndquadrant.com] > > I can't detect any performance improvement with the patch applied to > > current master, using the test case from Yoshikazu Imai (2019-03-19). > > That's strange... Pe

RE: Speed up transaction completion faster after many relations are accessed in a transaction

2019-04-04 Thread Tsunakawa, Takayuki
Hi Amit-san, Imai-snan, From: Amit Langote [mailto:langote_amit...@lab.ntt.co.jp] > I was able to detect it as follows. > plan_cache_mode = auto > >HEAD: 1915 tps > Patched: 2394 tps > > plan_cache_mode = custom (non-problematic: generic plan is never created) > >HEAD: 2402 tps > Patche

Re: fix the spelling mistakes of comments

2019-04-04 Thread Michael Paquier
On Thu, Apr 04, 2019 at 07:57:02AM -0400, Robert Haas wrote: > On Wed, Apr 3, 2019 at 6:55 AM Liu, Huailing > wrote: >> * This module contains the code for waiting and release★ of backends. >> * All code in this module executes on the primary. The core streaming >> * replication transport remains

Re: pg_rewind vs superuser

2019-04-04 Thread Michael Paquier
On Thu, Apr 04, 2019 at 01:19:44PM +0200, Magnus Hagander wrote: > All of it, or just the checkpoint part? I assume just the checkpoint part? > AFAIK it does require superuser in those earlier versions? I meant of course the checkpoint part down to 9.5, and the rest down to 11, so done this way. -

RE: Speed up transaction completion faster after many relations are accessed in a transaction

2019-04-04 Thread Imai, Yoshikazu
On Fri, Apr 5, 2019 at 1:31 AM, Amit Langote wrote: > On 2019/04/05 5:42, Peter Eisentraut wrote: > > On 2019-04-04 06:58, Amit Langote wrote: > >> Also, since the "speed up partition planning" patch went in > >> (428b260f8), it might be possible to see the performance boost even > >> with the part

Re: Server Crash due to assertion failure in _bt_check_unique()

2019-04-04 Thread Ashutosh Sharma
On Thu, Apr 4, 2019 at 10:12 PM Peter Geoghegan wrote: > On Thu, Apr 4, 2019 at 4:06 AM Ashutosh Sharma > wrote: > > Attached is the patch with above changes. Please let me know if my > understanding is wrong. Thanks. > > You have it right. This bug slipped in towards the end of development, > w

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-04-04 Thread Amit Langote
On 2019/04/05 5:42, Peter Eisentraut wrote: > On 2019-04-04 06:58, Amit Langote wrote: >> Also, since the "speed up partition planning" patch went in (428b260f8), >> it might be possible to see the performance boost even with the >> partitioning example you cited upthread. > > I can't detect any p

Re: New vacuum option to do only freezing

2019-04-04 Thread Masahiko Sawada
On Fri, Apr 5, 2019 at 4:06 AM Robert Haas wrote: > > On Wed, Apr 3, 2019 at 10:32 PM Masahiko Sawada wrote: > > Attached the updated version patch. > > Committed with a little bit of documentation tweaking. > Thank you for committing them! Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELE

Re: Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-04-04 Thread Masahiko Sawada
On Thu, Apr 4, 2019 at 10:07 PM Julien Rouhaud wrote: > > On Thu, Apr 4, 2019 at 1:23 PM Masahiko Sawada wrote: > > > > On Thu, Apr 4, 2019 at 1:26 PM Tsunakawa, Takayuki > > wrote: > > > > > > From: Fujii Masao [mailto:masao.fu...@gmail.com] > > > > reloption for TOAST is also required? > > > >

RE: Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-04-04 Thread Tsunakawa, Takayuki
From: Masahiko Sawada [mailto:sawada.m...@gmail.com] > "VACUUM" needs or "vacuum" is more appropriate here? Looking at the same file and some other files, "vacuum" looks appropriate because it represents the vacuum action, not the specific VACUUM command. > The format of the documentation of n

Re: Protect syscache from bloating with negative cache entries

2019-04-04 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Thu, 4 Apr 2019 15:44:35 -0400, Robert Haas wrote in > On Thu, Apr 4, 2019 at 8:53 AM Kyotaro HORIGUCHI > wrote: > > So it seems to me that the simplest "Full" version wins. The > > attached is rebsaed version. dlist_move_head(entry) is removed as > > mentioned ab

Re: COPY FROM WHEN condition

2019-04-04 Thread Andres Freund
Hi, On 2019-04-05 12:59:06 +1300, David Rowley wrote: > I read through the final commit to see what had changed and I noticed > that I had forgotten to remove nbuffers from CopyMultiInsertInfo when > changing from a hash table to a List. > > Patch to fix is attached. Pushed, thanks.

RE: Speed up transaction completion faster after many relations are accessed in a transaction

2019-04-04 Thread Tsunakawa, Takayuki
Hi Peter, Imai-san, From: Peter Eisentraut [mailto:peter.eisentr...@2ndquadrant.com] > I can't detect any performance improvement with the patch applied to > current master, using the test case from Yoshikazu Imai (2019-03-19). That's strange... Peter, Imai-san, can you compare your test procedu

Re: COPY FROM WHEN condition

2019-04-04 Thread David Rowley
On Fri, 5 Apr 2019 at 12:32, Andres Freund wrote: > I've pushed this now. Besides those naming changes, I'd to re-add the > zero initialization (I did end up seing compiler warnings when compiling > with optimizations). Also some comment fixes. Thanks for pushing. > I added one more flush locat

Re: Changes to pg_dump/psql following collation "C" in the catalog

2019-04-04 Thread Tom Lane
"Daniel Verite" writes: > I think psql and pg_dump need to adjust, just like the 3rd party tools > will, at least those that support collation-aware search in the catalog. > PFA a patch that implements the slight changes needed. > I'll add an entry for it in the next CF. Hm, if that's as much as

Re: COPY FROM WHEN condition

2019-04-04 Thread Andres Freund
Hi, On 2019-04-04 12:04:28 -0700, Andres Freund wrote: > On 2019-04-03 22:20:00 -0700, Andres Freund wrote: > > On 2019-04-03 20:00:09 +1300, David Rowley wrote: > > > Oops, I forgot about that one. v4 attached. > > > > I'm pretty happy with this. I'm doing some minor changes (e.g. don't > > like

Re: Changes to pg_dump/psql following collation "C" in the catalog

2019-04-04 Thread Chapman Flack
>> "Daniel Verite" writes: >>> One consequence of using the "C" collation in the catalog versus >>> the db collation As an intrigued Person Following At Home, I was happy when I found this little three-message thread had more context in [1] and [2]. :) -Chap [1] https://postgr.es/m/15938.154437

Re: PostgreSQL Buildfarm Client Release 10

2019-04-04 Thread Andrew Dunstan
On Thu, Apr 4, 2019 at 6:18 PM Tom Lane wrote: > > Andrew Dunstan writes: > > The release can be downloaded from > > https://github.com/PGBuildFarm/client-code/archive/REL_10.tar.gz or > > https://buildfarm.postgresql.org/downloads/latest-client.tgz > > I don't actually see it on the buildfarm.po

Re: PostgreSQL Buildfarm Client Release 10

2019-04-04 Thread Tom Lane
Andrew Dunstan writes: > The release can be downloaded from > https://github.com/PGBuildFarm/client-code/archive/REL_10.tar.gz or > https://buildfarm.postgresql.org/downloads/latest-client.tgz I don't actually see it on the buildfarm.postgresql.org server? regards, tom la

Re: Inadequate executor locking of indexes

2019-04-04 Thread David Rowley
On Fri, 5 Apr 2019 at 08:26, Tom Lane wrote: > Pushed with some minor tweaking, mostly comments. Thanks for tweaking and pushing this. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: speeding up planning with partitions

2019-04-04 Thread David Rowley
On Fri, 5 Apr 2019 at 07:33, Floris Van Nee wrote: > I had a question about the performance of pruning of functions like now() and > current_date. I know these are handled differently, as they cannot be > excluded during the first phases of planning. However, curerntly, this new > patch makes t

Re: Refactoring the checkpointer's fsync request queue

2019-04-04 Thread Thomas Munro
On Fri, Apr 5, 2019 at 10:53 AM Thomas Munro wrote: > Ok, here is a patch that adds a one-typedef header and uses > SegmentIndex to replace all cases of BlockNumber and int holding a > segment number (where as an "index" or a "count"). (sorry, I meant "SegmentNumber", not "SegmentIndex") -- Tho

PostgreSQL Buildfarm Client Release 10

2019-04-04 Thread Andrew Dunstan
Announcing Release 10 of the PostgreSQL Buildfarm client Principal feature: support for non-standard repositories: . support multi-element branch names, such as “dev/featurename” or “bug/ticket_number/branchname” . provide a get_branches() method in SCM module . support regular expression branche

Re: Refactoring the checkpointer's fsync request queue

2019-04-04 Thread Thomas Munro
On Fri, Apr 5, 2019 at 2:03 AM Alvaro Herrera wrote: > On 2019-Apr-04, Thomas Munro wrote: > > I don't think it's project policy to put a single typedef into its own > > header like that, and I'm not sure where else to put it. > > shrug. Looks fine to me. I suppose if we don't have it anywhere,

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-04 12:23:08 -0700, Andres Freund wrote: > Also, how is this code even close to correct? > CheckAndSetPageAllVisible() modifies the buffer in a crucial way, and > there's no WAL logging? Without even a comment arguing why that's OK (I > don't think it is)? Peter Geoghegan just remin

Re: New vacuum option to do only freezing

2019-04-04 Thread Bossart, Nathan
On 4/4/19, 12:06 PM, "Robert Haas" wrote: > Committed with a little bit of documentation tweaking. Thanks! I noticed a very small typo in the new documentation. diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml index fdd8151220..c652f8b0bc 100644 --- a/doc/src/sgml/ref/va

Re: [proposal] Add an option for returning SQLSTATE in psql error message

2019-04-04 Thread Tom Lane
didier writes: > [ 0001-Add-sqlstate-output-mode-to-VERBOSITY_v1.patch ] Pushed with some mostly-cosmetic adjustments. The main non-cosmetic thing I did was to adjust the logic so that if no SQLSTATE is available, it acts like TERSE mode. Otherwise, we'd print nothing at all except "ERROR:", wh

Re: query logging of prepared statements

2019-04-04 Thread Justin Pryzby
On Thu, Apr 04, 2019 at 03:07:04PM -0300, Alvaro Herrera wrote: > which does not look good -- the statement is nowhere to be seen. The commit > message quotes this as desirable output: Good catch. Unnamed statements sent behind the scenes by pqExecParams weren't being logged. I specifically han

Re: propagating replica identity to partitions

2019-04-04 Thread Alvaro Herrera
On 2019-Mar-29, Peter Eisentraut wrote: > On 2019-03-28 18:16, Simon Riggs wrote: > > SET TABLESPACE should not recurse because it copies the data, while > > holding long locks. If that was ever fixed so it happened concurrently, > > I would agree this could recurse by default. > > Since a partit

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-04-04 Thread Peter Eisentraut
On 2019-04-04 06:58, Amit Langote wrote: > Also, since the "speed up partition planning" patch went in (428b260f8), > it might be possible to see the performance boost even with the > partitioning example you cited upthread. I can't detect any performance improvement with the patch applied to curr

Re: proposal: pg_restore --convert-to-text

2019-04-04 Thread Alvaro Herrera
I just pushed it with trivial changes: * rebased for cc8d41511721 * changed wording in the error message * added a new test for the condition in t/001_basic.pl * Added the "-" in the --help line of -f. Andrew G. never confirmed that this change is sufficient to appease users being confused by

Re: [HACKERS] Block level parallel vacuum

2019-04-04 Thread Robert Haas
On Thu, Apr 4, 2019 at 6:28 AM Masahiko Sawada wrote: > These patches conflict with the current HEAD. Attached the updated patches. They'll need another rebase. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Robert Haas
On Thu, Apr 4, 2019 at 11:52 AM Antonin Houska wrote: > Robert Haas wrote: > > I'm willing to put some effort into trying to get this into v13 if > > you're willing to keep hacking on it, but there's probably a fair > > amount to do and a year can go by in a hurry. > > That'd be appreciated, espe

Re: Protect syscache from bloating with negative cache entries

2019-04-04 Thread Robert Haas
On Thu, Apr 4, 2019 at 8:53 AM Kyotaro HORIGUCHI wrote: > So it seems to me that the simplest "Full" version wins. The > attached is rebsaed version. dlist_move_head(entry) is removed as > mentioned above in that patch. 1. I really don't think this patch has any business changing the existing log

Re: Inadequate executor locking of indexes

2019-04-04 Thread Tom Lane
David Rowley writes: > Wrong patch. Here's what I meant to send. Pushed with some minor tweaking, mostly comments. Some notes: * We now have a general convention that queries always take the same lock type on indexes as on their parent tables, but that convention is not respected by index DDL.

Re: Changes to pg_dump/psql following collation "C" in the catalog

2019-04-04 Thread Daniel Verite
Tom Lane wrote: > "Daniel Verite" writes: > > One consequence of using the "C" collation in the catalog versus > > the db collation is that pg_dump -t with a regexp may not find > > the same tables as before. It happens when these conditions are > > all met: > > - the collation of the dat

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-04 16:15:54 -0300, Alvaro Herrera wrote: > On 2019-Apr-04, Andres Freund wrote: > > > I'm totally not OK with this from a layering > > POV. CheckAndSetAllVisibleBulkInsertState is entirely heap specific > > (without being named such), whereas all the heap specific bits are > > gett

Re: Row Level Security − leakproof-ness and performance implications

2019-04-04 Thread Peter Eisentraut
On 2019-03-18 20:08, Peter Eisentraut wrote: > I agree that it would be useful to document and discuss better which > built-in operators are leak-proof and which are not. But I don't think > the CREATE POLICY reference page is the place to do it. Note that the > leak-proofness mechanism was origi

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Alvaro Herrera
On 2019-Apr-04, Andres Freund wrote: > I'm totally not OK with this from a layering > POV. CheckAndSetAllVisibleBulkInsertState is entirely heap specific > (without being named such), whereas all the heap specific bits are > getting moved below tableam. This is a fair complaint, but on the other

Re: New vacuum option to do only freezing

2019-04-04 Thread Robert Haas
On Wed, Apr 3, 2019 at 10:32 PM Masahiko Sawada wrote: > Attached the updated version patch. Committed with a little bit of documentation tweaking. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Peter Eisentraut
On 2019-04-04 17:35, Stephen Frost wrote: > Ok, it looks like there's a server-side error happening here, and it > would be good to see what that is, so can you send the server logs? These errors appear several times in the server logs: FATAL: GSSAPI context error DETAIL: Miscellaneous failure

Re: COPY FROM WHEN condition

2019-04-04 Thread Andres Freund
On 2019-04-03 22:20:00 -0700, Andres Freund wrote: > Hi, > > On 2019-04-03 20:00:09 +1300, David Rowley wrote: > > Oops, I forgot about that one. v4 attached. > > I'm pretty happy with this. I'm doing some minor changes (e.g. don't > like the function comment formatting that much, the tableam cal

Re: query logging of prepared statements

2019-04-04 Thread Alvaro Herrera
On 2019-Apr-04, Alvaro Herrera wrote: > I think we could improve on this by setting a "logged" flag in the > portal; if the Parse logs the statement, then don't include the > statement in further lines, otherwise do. Also: I think such a flag could help the case of a query that takes long enough

Re: speeding up planning with partitions

2019-04-04 Thread Floris Van Nee
Hi all, First of all I would like to thank everyone involved in this patch for their hard work on this. This is a big step forward. I've done some performance and functionality testing with the patch that was committed to master and it looks very good. I had a question about the performance of

Re: query logging of prepared statements

2019-04-04 Thread Alvaro Herrera
On 2019-Apr-04, Alvaro Herrera wrote: > However, turning duration logging off and using log_statement=all, this is > what > I get: > > 2019-04-04 14:58:42.564 -03 [31685] LOG: statement: SET search_path = > testlibpq3 > 2019-04-04 14:58:42.565 -03 [31685] LOG: execute > 2019-04-04 14:58:42.5

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread PostgreSQL - Hans-Jürgen Schönig
On 4/4/19 5:52 PM, Antonin Houska wrote: Robert Haas wrote: I'm willing to put some effort into trying to get this into v13 if you're willing to keep hacking on it, but there's probably a fair amount to do and a year can go by in a hurry. That'd be appreciated, especially by my boss. It doesn

Re: query logging of prepared statements

2019-04-04 Thread Alvaro Herrera
On 2019-Mar-04, Justin Pryzby wrote: > Thanks for reviewing. I'm also interested in discussion about whether this > change is undesirable for someone else for some reason ? For me, the existing > output seems duplicative and "denormalized". :) Some digging turned up that the function you're re

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Andres Freund
Hi, On 2019-04-03 10:19:17 +0530, Pavan Deolasee wrote: > diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c > index c1fd7b78ce..09df70a3ac 100644 > --- a/src/backend/commands/copy.c > +++ b/src/backend/commands/copy.c > @@ -2833,6 +2833,15 @@ CopyFrom(CopyState cstate) >

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Robbie Harwood
Tom Lane writes: > I wrote: >> Stephen Frost writes: >>> So I'm a bit surprised that it's taking 4 minutes for you. I wonder if >>> there might be an issue related to the KDC wanting to get some amount of >>> random data and the system you're on isn't producing random bytes very >>> fast..? > >

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Robbie Harwood
Tom Lane writes: > Stephen Frost writes: >> * Tom Lane (t...@sss.pgh.pa.us) wrote: >>> Well, if the caller thinks what is being passed back is an int, >>> it will do a 32-to-64-bit widening, which is almost certainly >>> going to result in a corrupted pointer. > >> Oh, good point. Interesting t

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2019-04-04 Thread Tomas Vondra
Hi, I've been looking at this patch for a while, and it seems pretty much RFC, so barring objections I'll take care of that once I do a bit more testing and review. Unless someone else wants to take care of that. FWIW I wonder if we should add the code for partitioned tables to CopyFrom, conside

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Watching the test logs, I see that essentially all the time on the RHEL6 >> machine is consumed by the two >> # Running: /usr/sbin/kdb5_util create -s -P secret0 >> steps. Is there a case for merging the two scripts so we only have

Re: [GSoC 2019] Proposal: Develop Performance Farm Database and Website

2019-04-04 Thread Mark Wong
Hi Ilaria, Edited for bottom posting. :) On Fri, Mar 29, 2019 at 03:01:05PM +0100, Ilaria wrote: > > Am 29.03.2019 um 13:52 schrieb Peter Eisentraut > > : > > > >> On 2019-03-29 13:04, Robert Haas wrote: > >>> On Tue, Mar 26, 2019 at 9:10 AM Ila B. wrote: > >>> I am Ilaria Battiston, an aspiri

Re: Server Crash due to assertion failure in _bt_check_unique()

2019-04-04 Thread Peter Geoghegan
On Thu, Apr 4, 2019 at 4:06 AM Ashutosh Sharma wrote: > Attached is the patch with above changes. Please let me know if my > understanding is wrong. Thanks. You have it right. This bug slipped in towards the end of development, when the insertstate struct was introduced. I have pushed something

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > I wrote: > > Stephen Frost writes: > >> So I'm a bit surprised that it's taking 4 minutes for you. I wonder if > >> there might be an issue related to the KDC wanting to get some amount of > >> random data and the system you're on isn't produci

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Tom Lane
I wrote: > Stephen Frost writes: >> So I'm a bit surprised that it's taking 4 minutes for you. I wonder if >> there might be an issue related to the KDC wanting to get some amount of >> random data and the system you're on isn't producing random bytes very >> fast..? > Not sure. This is my usua

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Antonin Houska
Robert Haas wrote: > I'm willing to put some effort into trying to get this into v13 if > you're willing to keep hacking on it, but there's probably a fair > amount to do and a year can go by in a hurry. That'd be appreciated, especially by my boss. It doesn't seem likely that in this situation

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > Kerberos tests are now failing for me (macOS). I'm seeing > > psql: error: could not connect to server: Over-size error packet sent by > the server. > not ok 3 - GSS encryption without auth > > # Failed test 'GSS encryp

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> There must be something about the x86_64 ABI that allows this to >> accidentally work -- maybe integers are presumed to be sign-extended >> to 64 bits by callee not caller? I added some logging and verified >> that pgstat.c is seei

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> Well, if the caller thinks what is being passed back is an int, > >> it will do a 32-to-64-bit widening, which is almost certainly > >> going to result in a corrupted pointer.

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Peter Eisentraut
On 2019-04-04 17:16, Tom Lane wrote: > BTW, the kerberos test suite takes nearly 4 minutes for me, is > it supposed to be so slow? I've seen this on some virtualized machines that didn't have a lot of entropy. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 2

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Well, if the caller thinks what is being passed back is an int, >> it will do a 32-to-64-bit widening, which is almost certainly >> going to result in a corrupted pointer. > Oh, good point. Interesting that it still works then. T

Re: GiST VACUUM

2019-04-04 Thread Heikki Linnakangas
On 25/03/2019 15:20, Heikki Linnakangas wrote: On 24/03/2019 18:50, Andrey Borodin wrote: I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) || (Gi

Re: POC: GROUP BY optimization

2019-04-04 Thread Dmitry Dolgov
> On Thu, Jan 31, 2019 at 12:24 PM Andres Freund wrote: > > As nothing has happened since, I'm marking this as returned with > feedback. This patch was on my radar for some time in the past and we've seen use cases where it could be pretty useful (probably even without the incremental sort patch)

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> I'm not very sure why the integer/pointer confusion in pgstat_bestart > >> doesn't cause hard crashes when using gss auth --- or does > >> this suite not actually test that? >

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Robert Haas
On Thu, Apr 4, 2019 at 9:57 AM Antonin Houska wrote: > I think I finally understand. Originally I thought the question is how to > compute correct page checksum while the hint bits can be changed w/o exclusive > lock on the buffer. Now I realize that it's more about *recovery*: if the hint > bit c

Re: [PATCH v20] GSSAPI encryption support

2019-04-04 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> I'm not very sure why the integer/pointer confusion in pgstat_bestart >> doesn't cause hard crashes when using gss auth --- or does >> this suite not actually test that? > Isn't it just saying that because of the implicit declarati

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-04-04 Thread Robert Haas
On Wed, Apr 3, 2019 at 10:03 PM Kyotaro HORIGUCHI wrote: > > * Insert log record, using delete or insert instead of update log > > * when only one of the two buffers needs WAL-logging. If this were a > > * HOT-update, redoing the WAL record would result in a broken > > * hot-chain. However, that n

  1   2   >