Re: pgbench - extend initialization phase control

2019-07-15 Thread Ibrar Ahmed
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested Other than that, the patch looks good to me. The new status of t

Re: PG 12 draft release notes

2019-07-15 Thread Michael Paquier
On Mon, Jul 15, 2019 at 08:51:34PM +0200, Laurenz Albe wrote: > I wonder if commits 0ba06e0bf and 40cfe8606 are worth mentioning > in the release notes. They make "pg_test_fsync" work correctly > on Windows for the first time. I don't know about this point specifically. Improving support for pg_

Re: Built-in connection pooler

2019-07-15 Thread Konstantin Knizhnik
On 15.07.2019 17:04, Konstantin Knizhnik wrote: On 14.07.2019 8:03, Thomas Munro wrote: On my FreeBSD box (which doesn't have epoll(), so it's latch.c's old school poll() for now), I see the connection proxy process eating a lot of CPU and the temperature rising.  I see with truss that it's

Re: pg_receivewal documentation

2019-07-15 Thread Michael Paquier
On Wed, Jul 10, 2019 at 11:26:04AM -0400, Jesper Pedersen wrote: > On 7/10/19 10:24 AM, Alvaro Herrera wrote: > > +1 to document this caveat. >> >> How about >> Note that while WAL will be flushed with this setting, >> pg_receivewal never applies it, so >> must not be s

Re: POC: Cleaning up orphaned files using undo logs

2019-07-15 Thread Amit Kapila
On Mon, Jul 15, 2019 at 9:56 PM Robert Haas wrote: > > On Sat, Jul 13, 2019 at 6:26 AM Amit Kapila wrote: > > I think then I am missing something because what I am talking about is > > below code in rbt_insert: > > What you're saying here is that, with an rbtree, an exact match will > result in a

Re: make \d pg_toast.foo show its indices ; and, \d toast show its main table ; and \d relkind=I show its partitions (and tablespace)

2019-07-15 Thread Justin Pryzby
Sorry, I missed this until now. On Sun, Jun 30, 2019 at 10:26:28AM +0200, Fabien COELHO wrote: > *** About toast table v3: > > Patch applies cleanly, compiles, works for me. > > ISTM that the he query should be unambiguous: pg_catalog.pg_class instead of > pg_class, add an alias (eg c), use c.FI

Re: pgbench - add pseudo-random permutation function

2019-07-15 Thread Thomas Munro
On Fri, May 24, 2019 at 2:46 AM Fabien COELHO wrote: > Here is a v15 which is a rebase, plus a large simplification of the modmul > function if an int128 type is available, which is probably always… > > Function nbits(), which was previously discussed, has been simplified by > > using the functio

Re: POC: Cleaning up orphaned files using undo logs

2019-07-15 Thread Amit Kapila
On Tue, Jul 16, 2019 at 2:09 AM Robert Haas wrote: > > PerformUndoActions() also thinks that there is a possibility of > failing to insert a failed request into the error queue, and makes > reference to such requests being rediscovered by the discard worker, > but I thought (as I said in my previo

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Masahiko Sawada
On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > On Fri, Jul 12, 2019 at 02:15:02PM +0900, Masahiko Sawada wrote: > > > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL. > > > 8k pages will use the LSN as a nonce, which will be encrypted to > > > generate the initial

Re: Fix typos and inconsistencies for HEAD (take 6)

2019-07-15 Thread Michael Paquier
On Sun, Jul 14, 2019 at 08:24:01AM +0300, Alexander Lakhin wrote: > 6.10. fildes -> fd Not sure that this one was worth bothering. And the rest looks correct after review, so applied! Thanks! > As a side note, while looking at dt_common.c (fixing 6.47), I've got a > feeling that the datetktbl i

Re: doc: mention pg_reload_conf() in pg_hba.conf documentation

2019-07-15 Thread Ian Barwick
On 7/16/19 10:09 AM, Bruce Momjian wrote: On Mon, Jul 15, 2019 at 12:47:18PM +0900, Ian Barwick wrote: Hi I noticed the documentation for pg_hba.conf: https://www.postgresql.org/docs/current/auth-pg-hba-conf.html says: you will need to signal the postmaster (using pg_ctl reload or ki

Re: POC: converting Lists into arrays

2019-07-15 Thread David Steele
On 7/15/19 11:07 PM, Tom Lane wrote: David Rowley writes: The only thoughts I have so far here are that it's a shame that the function got called list_qsort() and not just list_sort(). I don't see why callers need to know anything about the sort algorithm that's being used. Meh. list_qsort(

Re: POC: converting Lists into arrays

2019-07-15 Thread Tom Lane
David Rowley writes: > The only thoughts I have so far here are that it's a shame that the > function got called list_qsort() and not just list_sort(). I don't > see why callers need to know anything about the sort algorithm that's > being used. Meh. list_qsort() is quicksort only to the extent

Re: POC: converting Lists into arrays

2019-07-15 Thread David Rowley
On Tue, 16 Jul 2019 at 07:49, Tom Lane wrote: > A possibly controversial point is that I made list_qsort() sort the > given list in-place, rather than building a new list as it has > historically. For every single one of the existing and new callers, > copying the input list is wasteful, because

Re: Change ereport level for QueuePartitionConstraintValidation

2019-07-15 Thread Amit Langote
On Tue, Jul 16, 2019 at 10:15 AM David Rowley wrote: > On Tue, 16 Jul 2019 at 03:13, Robert Haas wrote: > > I vote for changing it to NOTICE instead of DEBUG1. > > Well, there are certainly other DDL commands that spit out NOTICES. > > postgres=# create table z (a int); > CREATE TABLE > postgres=

Re: Change ereport level for QueuePartitionConstraintValidation

2019-07-15 Thread Tom Lane
David Rowley writes: > Would anyone complain if we made them all INFO? That would be remarkably horrid, because that makes them unsuppressable. I'm generally for having these be less in-your-face, not more so. regards, tom lane

Re: PG 12 draft release notes

2019-07-15 Thread Peter Geoghegan
On Wed, Jun 12, 2019 at 7:48 PM Bruce Momjian wrote: > Great, patch applied. I think that it would make sense to have a v12 release note item for amcheck's new "rootdescend" verification option: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=c1afd175b5b2e5c44f6da34988342e00ecdfb5

Re: SegFault on 9.6.14

2019-07-15 Thread Jerry Sievers
Tomas Vondra writes: > On Mon, Jul 15, 2019 at 07:22:55PM -0500, Jerry Sievers wrote: > >>Tomas Vondra writes: >> >>> On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote: >>> Greetings Hackers. We have a reproduceable case of $subject that issues a backtrace such as se

Re: PG 12 draft release notes

2019-07-15 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 10:18:07PM +0700, John Naylor wrote: > I noticed a small typo in the release notes in the list of languages > with new stemmers (see attached) Sorry, fixed, thanks. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprise

Re: Change ereport level for QueuePartitionConstraintValidation

2019-07-15 Thread David Rowley
On Tue, 16 Jul 2019 at 03:13, Robert Haas wrote: > I vote for changing it to NOTICE instead of DEBUG1. Well, there are certainly other DDL commands that spit out NOTICES. postgres=# create table z (a int); CREATE TABLE postgres=# create table x (a int) inherits(z); NOTICE: merging column "a" wi

Re: Parallel Append subplan order instability on aye-aye

2019-07-15 Thread Tom Lane
I wrote: > So that data-collection patch has been in place for nearly 2 months > (since 2019-05-21), and in that time we've seen a grand total of > no repeats of the original problem, as far as I've seen. Oh ... wait a minute. I decided to go scrape the buildfarm logs to confirm my impression tha

Re: doc: mention pg_reload_conf() in pg_hba.conf documentation

2019-07-15 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 12:47:18PM +0900, Ian Barwick wrote: > Hi > > I noticed the documentation for pg_hba.conf: > > https://www.postgresql.org/docs/current/auth-pg-hba-conf.html > > says: > > you will need to signal the postmaster (using pg_ctl reload or kill -HUP) > to > make it

doesn't execSRF.c do that already?

2019-07-15 Thread Chapman Flack
I see some code of long standing in PL/Java where its handler for a set-returning function creates a new context, "PL/Java row context", during first-call init, as a child of context->multi_call_memory_ctx, diligently resets it with every percall-setup and before calling the user function, and dele

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 06:08:28PM -0400, Sehrope Sarkuni wrote: > Hi, > > Some more thoughts on CBC vs CTR modes. There are a number of > advantages to using CTR mode for page encryption. > > CTR encryption modes can be fully parallelized, whereas CBC can only > parallelized for decryption. Whil

Re: SegFault on 9.6.14

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 07:22:55PM -0500, Jerry Sievers wrote: Tomas Vondra writes: On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote: Greetings Hackers. We have a reproduceable case of $subject that issues a backtrace such as seen below. The query that I'd prefer to sanitize b

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Tue, Jul 16, 2019 at 02:04:58AM +0200, Tomas Vondra wrote: > On Mon, Jul 15, 2019 at 06:05:37PM -0400, Bruce Momjian wrote: > > On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote: > > > On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote: > > > > The crazy seems more sane now

Re: SegFault on 9.6.14

2019-07-15 Thread Jerry Sievers
Tomas Vondra writes: > On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote: > >>Greetings Hackers. >> >>We have a reproduceable case of $subject that issues a backtrace such as >>seen below. >> >>The query that I'd prefer to sanitize before sending is <30 lines of at >>a glance, not ter

Re: Parallel Append subplan order instability on aye-aye

2019-07-15 Thread Tom Lane
[ reviving a thread that's been idle for awhile ] I wrote: > Thomas Munro writes: >> Huh, idiacanthus failed showing vacuum_count 0, in select_parallel. >> So ... the VACUUM command somehow skipped those tables? > No, because the reltuples counts are correct. I think what we're > looking at the

Re: SegFault on 9.6.14

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote: Greetings Hackers. We have a reproduceable case of $subject that issues a backtrace such as seen below. The query that I'd prefer to sanitize before sending is <30 lines of at a glance, not terribly complex logic. It nonetheless di

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 06:05:37PM -0400, Bruce Momjian wrote: On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote: On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote: > The crazy seems more sane now --- "encrypt the page with CRC contents as > zero" (which we probably alread

Re: pgbench - add option to show actual builtin script code

2019-07-15 Thread Thomas Munro
On Fri, Jul 12, 2019 at 4:20 AM Fabien COELHO wrote: > >>> Now the patch is good now. > >>> > >>> The new status of this patch is: Ready for Committer > > > > Why aren't we instead putting the exact scripts in the documentation? > > Having to call pgbench with a special flag to get the script text

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote: On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote: On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > > One extra thing we should consider is au

Re: Using unique btree indexes for pathkeys with one extra column

2019-07-15 Thread Tom Lane
Thomas Munro writes: > In the category "doing more tricks with our existing btrees", which > includes all that difficult stuff like skip scans and incremental > sort, here's an easier planner-only one: if you have a unique index > on (a) possibly "including" (b) and you have a pathkey (a, b), you

SegFault on 9.6.14

2019-07-15 Thread Jerry Sievers
Greetings Hackers. We have a reproduceable case of $subject that issues a backtrace such as seen below. The query that I'd prefer to sanitize before sending is <30 lines of at a glance, not terribly complex logic. It nonetheless dies hard after a few seconds of running and as expected, results i

Re: pgbench - add minimal stats on initialization

2019-07-15 Thread Thomas Munro
On Fri, Apr 12, 2019 at 12:44 AM Ibrar Ahmed wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed > Implements feature: tested, passed > Spec compliant: tested, passed > Documentation:not tested > >

Re: refactoring - share str2*int64 functions

2019-07-15 Thread Thomas Munro
On Mon, Jul 15, 2019 at 5:08 PM Fabien COELHO wrote: > The compromise I can offer is to change the name of the first one, say to > "pg_scanint8" to reflect its former backend name. Attached a v4 which does > a renaming so as to avoid the name similarity but signature difference. I > also made both

Removing unneeded downlink field from nbtree stack struct

2019-07-15 Thread Peter Geoghegan
Attached patch slightly simplifies _bt_getstackbuf() by making it accept a child BlockNumber argument, rather than requiring that callers store the child block number in the parent stack item's bts_btentry field. We can remove the bts_btentry field from the BTStackData struct, because we know where

Re: Patch to document base64 encoding

2019-07-15 Thread Karl O. Pinc
On Mon, 15 Jul 2019 23:00:55 +0200 (CEST) Fabien COELHO wrote: > The patch clarifies the documentation about encode/decode and other > text/binary string conversion functions. Other notable changes: Corrects categorization of functions as string or binary. Reorders functions alphabeticall

Re: Adding SMGR discriminator to buffer tags

2019-07-15 Thread Thomas Munro
On Tue, Jul 16, 2019 at 1:49 AM Robert Haas wrote: > [long form -1] > > But how about just using a magic database OID? This patch was just an experiment based on discussion here: https://www.postgresql.org/message-id/flat/CA%2BhUKG%2BDE0mmiBZMtZyvwWtgv1sZCniSVhXYsXkvJ_Wo%2B83vvw%40mail.gmail.com

Use PageIndexTupleOverwrite() within nbtsort.c

2019-07-15 Thread Peter Geoghegan
Attached patch slightly simplifies nbtsort.c by making it use PageIndexTupleOverwrite() to overwrite the last right non-pivot tuple with the new high key (pivot tuple). PageIndexTupleOverwrite() is designed so that code like this doesn't need to delete and re-insert to replace an existing tuple. T

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote: > On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > > > One extra thing we should consider is authenticated encryption. We can't > > > just encrypt the page

Re: POC: converting Lists into arrays

2019-07-15 Thread Tom Lane
I wrote: > [ list_qsort-API-change.patch ] Also, here's a follow-on patch that cleans up some crufty code in heap.c and relcache.c to use list_qsort, rather than manually doing insertions into a list that's kept ordered. The existing comments argue that that's faster than qsort for small N, but I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Sehrope Sarkuni
Hi, Some more thoughts on CBC vs CTR modes. There are a number of advantages to using CTR mode for page encryption. CTR encryption modes can be fully parallelized, whereas CBC can only parallelized for decryption. While both can use AES specific hardware such as AES-NI, CTR modes can go a step fu

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote: > On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote: > > The crazy seems more sane now --- "encrypt the page with CRC contents as > > zero" (which we probably already do to compute the CRC), then compute > > the CRC, and modi

Re: Creating partitions automatically at least on HASH?

2019-07-15 Thread Fabien COELHO
Hello Robert and Robert, CREATE TABLE foo(...) PARTITION BY HASH AUTOMATIC (MODULUS 10); -- or some other syntax This would be a relief on the longer path of dynamically creating partitions, but with lower costs than a dynamic approach. Yeah, I think something like this would be reaso

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: One extra thing we should consider is authenticated encryption. We can't just encrypt the pages (no matter which AES mode is used - XTS/CBC/...), as that does not provide

RE: minimizing pg_stat_statements performance overhead

2019-07-15 Thread Fabien COELHO
Hello Raymond, Sure, it definitely makes sense to reduce the overhead when the extension is disabled. I wanted to understand the source of performance issue, and your explanations where not enough for reproducing it. Thanks again Fabien. I am attaching the patch to this email in the hope of

Re: Patch to document base64 encoding

2019-07-15 Thread Fabien COELHO
Hello Karl, Attached is doc_base64_v10.patch Patch applies cleanly. Doc gen ok. The patch clarifies the documentation about encode/decode and other text/binary string conversion functions. No further comments beyond the title thing (Function x) already discussed, which is not a stopper.

Re: Allow an alias to be attached directly to a JOIN ... USING

2019-07-15 Thread Fabien COELHO
Hello Peter, A small new feature in SQL:2016 allows attaching a table alias to a JOIN/USING construct: ::= USING [ AS ] (The part in brackets is new.) This seems quite useful, and it seems the code would already support this if we allow the grammar to accept this syntax.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote: On Mon, Jul 15, 2019 at 03:47:59AM +0200, Tomas Vondra wrote: On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote: > We could check the CRC prior to encryption and throw an ERROR if it is > not correct. After decryption we can

Re: POC: Cleaning up orphaned files using undo logs

2019-07-15 Thread Robert Haas
On Mon, Jul 1, 2019 at 3:54 AM Thomas Munro wrote: > 2. Introduced a new RMGR callback rm_undo_status. It is used to > decide when record sets in the UNDO_SHARED category should be > discarded (instead of the usual single xid-based rules). The possible > answers are "discard me now!", "ask me a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Sat, Jul 13, 2019 at 07:34:22AM -0400, Joe Conway wrote: > I stand by my position. At a minimum, we need a choice of AES128 and AES256. These are compelling arguments. Agreed. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 03:47:59AM +0200, Tomas Vondra wrote: > On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote: > > We could check the CRC prior to encryption and throw an ERROR if it is > > not correct. After decryption we can check it again -- if it no longer > > matches we would know

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote: > On 7/13/19 5:58 PM, Tomas Vondra wrote: > In my email I linked the wrong page for [2]. The correct one is here: > [2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html > > Following that, I think we could end up with three

Re: POC: converting Lists into arrays

2019-07-15 Thread Tom Lane
I wrote: > BTW, further on the subject of performance --- I'm aware of at least > these topics for follow-on patches: > ... > * Adjust API for list_qsort(), as discussed, to save indirections and > avoid constructing an intermediate pointer array. I also seem to recall > one place in the planner t

Re: Creating partitions automatically at least on HASH?

2019-07-15 Thread Robert Eckhardt
On Mon, Jul 15, 2019 at 10:54 AM Robert Haas wrote: > > On Mon, Jul 15, 2019 at 1:29 AM Fabien COELHO wrote: > > Hello pgdevs, > > > > sorry if this has been already discussed, but G did not yield anything > > convincing about that. > > > > While looking at HASH partitioning and creating a few on

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > One extra thing we should consider is authenticated encryption. We can't > just encrypt the pages (no matter which AES mode is used - XTS/CBC/...), > as that does not provide integrity protection (i.e. can't detect when > the ciphertex

Re: Psql patch to show access methods info

2019-07-15 Thread Nikita Glukhov
On 01.07.2019 14:06, Thomas Munro wrote: On Sun, Mar 31, 2019 at 2:13 PM wrote: Thanks for review. Hi Sergey, A new Commitfest is here and this doesn't apply -- could you please post a rebase? Thanks, Attached 7th version of the patches rebased onto current master. -- Nikita Glukhov Post

Re: PG 12 draft release notes

2019-07-15 Thread Laurenz Albe
On Sat, 2019-05-11 at 16:33 -0400, Bruce Momjian wrote: > I have posted a draft copy of the PG 12 release notes here: I wonder if commits 0ba06e0bf and 40cfe8606 are worth mentioning in the release notes. They make "pg_test_fsync" work correctly on Windows for the first time. Yours, Laurenz Albe

Re: POC: converting Lists into arrays

2019-07-15 Thread Tom Lane
Daniel Gustafsson writes: >> On 13 Jul 2019, at 18:32, Tom Lane wrote: >> I see from the cfbot that v8 is already broken (new call of lnext >> to be fixed). Don't really want to keep chasing a moving target, >> so unless I hear objections I'm going to adjust the additional >> spot(s) and commit

Re: Insecure initialization of required_relids field

2019-07-15 Thread Andrey Lepikhov
On 15/07/2019 18:48, Tom Lane wrote: Andrey Lepikhov writes: commit a31ad27fc5d introduced required_relids field. By default, it links to the clause_relids. It works good while we do not modify clause_relids or required_relids. But in the case of modification such initialization demands us t

Re: How to reclaim the space of dropped columns of a table?

2019-07-15 Thread Tom Lane
"David G. Johnston" writes: > On Mon, Jul 15, 2019 at 8:42 AM Paul Guo wrote: >> This seems to a bit vague for users (how to rewrite but keep the table >> definition) and it seems to still keep the dropped columns (though with >> null). Isn't it better to leave the functionality to command like '

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote: > On 7/13/19 9:38 AM, Joe Conway wrote: > --- > [1] and [2] show that at least some file system encryption uses a > different key per file. Yes, I see later they did that for per-file keys, but I think with WAL and crash recovery we decid

Re: POC: Cleaning up orphaned files using undo logs

2019-07-15 Thread Robert Haas
On Sat, Jul 13, 2019 at 6:26 AM Amit Kapila wrote: > I think then I am missing something because what I am talking about is > below code in rbt_insert: What you're saying here is that, with an rbtree, an exact match will result in a merging of requests which we don't want, so we have to make them

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Bruce Momjian
On Sat, Jul 13, 2019 at 09:38:06AM -0400, Joe Conway wrote: > On 7/11/19 9:05 PM, Bruce Momjian wrote: > >> Regardless of the method to split the heap into different keys, I think > >> there should be an option for some tables to not be encrypted. If we > >> decide it must be all or nothing for the

Re: How to reclaim the space of dropped columns of a table?

2019-07-15 Thread David G. Johnston
On Mon, Jul 15, 2019 at 8:42 AM Paul Guo wrote: > This seems to a bit vague for users (how to rewrite but keep the table > definition) and it seems to still keep the dropped columns (though with > null). Isn't it better to leave the functionality to command like 'vacuum > full' to completely remo

How to reclaim the space of dropped columns of a table?

2019-07-15 Thread Paul Guo
Hello hackers, I have been having a question about this with no answer from various sources . As known after dropping a column using 'alter table', table is not rewritten and vacuum full does not remove them also (still see the dropped column in pg_attribute). PG document says: https://www.postg

Re: PG 12 draft release notes

2019-07-15 Thread John Naylor
I noticed a small typo in the release notes in the list of languages with new stemmers (see attached) -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services stemming-Tamil.patch Description: Binary data

Re: Change ereport level for QueuePartitionConstraintValidation

2019-07-15 Thread Robert Haas
On Sun, Jul 14, 2019 at 7:46 PM Thomas Munro wrote: > ... and retreating to a safe distance. Is that measure in, like, light-years? I vote for changing it to NOTICE instead of DEBUG1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Creating partitions automatically at least on HASH?

2019-07-15 Thread Robert Haas
On Mon, Jul 15, 2019 at 1:29 AM Fabien COELHO wrote: > Hello pgdevs, > > sorry if this has been already discussed, but G did not yield anything > convincing about that. > > While looking at HASH partitioning and creating a few ones, it occured to > me that while RANGE and LIST partitions cannot be

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-15 Thread Tomas Vondra
On Mon, Jul 15, 2019 at 09:25:32AM -0400, James Coleman wrote: On Sun, Jul 14, 2019 at 10:16 PM Tomas Vondra wrote: >> >> Attached is a slightly modified patch series: >> >> >> >> 1) 0001 considers incremental sort paths in various places (adds the new >> >> generate_useful_gather_paths and mod

Re: Check-out mutable functions in check constraints

2019-07-15 Thread Robert Haas
On Sat, Jul 13, 2019 at 11:17 AM Tom Lane wrote: > If the spec is allowing this because it can be seen > to be safe, why should we not allow other cases that the user has > taken the trouble to prove to themselves are safe? (If their proof is > wrong, well, it wouldn't be the first bug in anyone'

Re: SHOW CREATE

2019-07-15 Thread Robert Haas
On Fri, Jul 5, 2019 at 1:14 PM Corey Huinker wrote: > I'd like to hear what others have to say, and incorporate that feedback into > a follow up proposal. I am unclear how this could be implemented without ending up with a ton of extra code that has to be maintained. pg_dump is a client-side to

Re: Built-in connection pooler

2019-07-15 Thread Konstantin Knizhnik
On 14.07.2019 8:03, Thomas Munro wrote: On my FreeBSD box (which doesn't have epoll(), so it's latch.c's old school poll() for now), I see the connection proxy process eating a lot of CPU and the temperature rising. I see with truss that it's doing this as fast as it can: poll({ 13/POLLIN 17

Re: Adding SMGR discriminator to buffer tags

2019-07-15 Thread Robert Haas
On Mon, Jul 15, 2019 at 6:59 AM Thomas Munro wrote: > That's an enum, so it works out to a word per record. The obvious way > to avoid increasing the size is shove the SMGR ID into the same space > that holds the forknum. Unlike BufferTag, where forknum currently > swims in 32 bits which this pa

Re: Insecure initialization of required_relids field

2019-07-15 Thread Tom Lane
Andrey Lepikhov writes: > commit a31ad27fc5d introduced required_relids field. By default, it > links to the clause_relids. > It works good while we do not modify clause_relids or required_relids. > But in the case of modification such initialization demands us to > remember, that this field is

Re: Support for jsonpath .datetime() method

2019-07-15 Thread Anastasia Lubennikova
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested Hi, In general, the feature looks good. It is consistent with th

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-15 Thread James Coleman
On Sun, Jul 14, 2019 at 10:16 PM Tomas Vondra wrote: > >> >> Attached is a slightly modified patch series: > >> >> > >> >> 1) 0001 considers incremental sort paths in various places (adds the new > >> >> generate_useful_gather_paths and modifies places calling > >> >> create_sort_path) > >> > > >

Re: Introduce timeout capability for ConditionVariableSleep

2019-07-15 Thread Robert Haas
On Fri, Jul 12, 2019 at 11:03 PM Thomas Munro wrote: > I pushed this too. It's a separate commit, because I think there is > at least a theoretical argument that it should be back-patched. I'm > not going to do that today though, because I doubt anyone is relying > on ConditionVariableSignal() w

Re: Option to dump foreign data in pg_dump

2019-07-15 Thread Luis Carril
On 15.07.19 12:06, Daniel Gustafsson wrote: On 12 Jul 2019, at 16:08, Luis Carril wrote: On 28 Jun 2019, at 19:55, Luis Carril wrote: What about providing a list of FDW servers instead of an all or nothing option? In that way the user really has to do a conscious decision to dump the content

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Antonin Houska
Masahiko Sawada wrote: > On Mon, Jun 17, 2019 at 11:02 PM Tomas Vondra > wrote: > > > > On Mon, Jun 17, 2019 at 08:39:27AM -0400, Joe Conway wrote: > > >On 6/17/19 8:29 AM, Masahiko Sawada wrote: > > >> From perspective of cryptographic, I think the fine grained TDE would > > >> be better solut

Re: [proposal] de-TOAST'ing using a iterator

2019-07-15 Thread John Naylor
On Wed, Jun 19, 2019 at 8:51 PM Binguo Bao wrote: > [v4 patch] Hi Binguo, I can verify I get no warnings with the v4 patch. I've done some additional performance testing. First, to sum up your results: > insert into detoast_c (a) select > repeat('1234567890-=abcdefghijklmnopqrstuvwxyz', 10

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-15 Thread Antonin Houska
Tomas Vondra wrote: > On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote: > >On 7/13/19 9:38 AM, Joe Conway wrote: > >[5] has this to say which seems independent of mode: > > > >"When encrypting data with a symmetric block cipher, which uses blocks > >of n bits, some security concerns be

Documentation fix for adding a column with a default value

2019-07-15 Thread Daniel Westermann (DWE)
Hi, the tip in the "Adding a column" section is not true anymore since PostgreSQL 11: https://www.postgresql.org/docs/current/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN Attached a patch proposal for this. Regards Daniel diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml ind

Re: Adding SMGR discriminator to buffer tags

2019-07-15 Thread Thomas Munro
On Fri, Jul 12, 2019 at 11:19 AM Shawn Debnath wrote: > On Fri, Jul 12, 2019 at 10:16:21AM +1200, Thomas Munro wrote: > + > + /* TODO: How should we handle other smgr IDs? */ > + if (smgrid != SMGR_MD) > continue; > > All files are copied verbati

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2019-07-15 Thread Paul Guo
On Mon, Jul 8, 2019 at 11:16 AM Thomas Munro wrote: > On Wed, Jun 19, 2019 at 7:22 PM Paul Guo wrote: > > I updated the patch to v3. In this version, we skip the error if copydir > fails due to missing src/dst directory, > > but to make sure the ignoring is legal, I add a simple log/forget > mec

Re: [HACKERS] WIP: Aggregation push-down

2019-07-15 Thread Richard Guo
On Fri, Jul 12, 2019 at 4:42 PM Antonin Houska wrote: > Richard Guo wrote: > > > I didn't fully follow the whole thread and mainly looked into the > > latest > > patch set. So what are the considerations for abandoning the > > aggmultifn > > concept? > > Originally the function was there to supp

Re: POC: converting Lists into arrays

2019-07-15 Thread Daniel Gustafsson
> On 13 Jul 2019, at 18:32, Tom Lane wrote: > I see from the cfbot that v8 is already broken (new call of lnext > to be fixed). Don't really want to keep chasing a moving target, > so unless I hear objections I'm going to adjust the additional > spot(s) and commit this pretty soon, like tomorrow

Re: Option to dump foreign data in pg_dump

2019-07-15 Thread Daniel Gustafsson
> On 12 Jul 2019, at 16:08, Luis Carril wrote: > > > > On 28 Jun 2019, at 19:55, Luis Carril wrote: > > > What about providing a list of FDW servers instead of an all or nothing > > > option? In that way the user really has to do a conscious decision to > > > dump the content of the foreign ta

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-15 Thread Paul Guo
On Wed, Jul 10, 2019 at 3:28 PM Michael Paquier wrote: > On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote: > > Yes, the patches changed Makefile so that pg_rewind and pg_basebackup > could > > use some common code, but for Windows build, I'm not sure where are those > > window build files

Re: Built-in connection pooler

2019-07-15 Thread Konstantin Knizhnik
On 15.07.2019 1:48, Thomas Munro wrote: On Mon, Jul 15, 2019 at 7:56 AM Konstantin Knizhnik wrote: Can you please explain me more precisely how to reproduce the problem (how to configure postgres and how to connect to it)? Maybe it's just that postmaster.c's ConnCreate() always allocates a