Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Masahiko Sawada
On Wed, Jul 10, 2019 at 11:06 AM Stephen Frost wrote: > > Greetings, > > * Ryan Lambert (r...@rustprooflabs.com) wrote: > > > What I think Tomas is getting at here is that we don't write a page only > > > once. > > > > > A nonce of tableoid+pagenum will only be unique the first time we write > > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Masahiko Sawada
On Tue, Jul 9, 2019 at 10:16 PM Joe Conway wrote: > > On 7/9/19 8:39 AM, Ryan Lambert wrote: > > Hi Thomas, > > > >> CBC mode does require > >> random nonces, other modes may be fine with even sequences as long as > >> the values are not reused. > > > > I disagree that CBC mode requires random non

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Masahiko Sawada
On Tue, Jul 9, 2019 at 9:01 PM Joe Conway wrote: > > On 7/9/19 6:07 AM, Peter Eisentraut wrote: > > On 2019-07-08 18:09, Joe Conway wrote: > >> In my mind, and in practice to a > >> large extent, a postgres tablespace == a unique mount point. > > > > But a critical difference is that in file syste

Re: pgbench - add \aset to store results of a combined query

2019-07-09 Thread Fabien COELHO
Hello Ibrar, SELECT 1 AS one \; SELECT 2 AS two UNION SELECT 2 \; SELECT 3 AS three \aset will set both "one" and "three", while "two" is not set because there were two rows. It is a kind of more permissive \gset. Are you sure two is not set :)? SELECT 2 AS two UNION SELECT 2; -- only

Re: POC: Cleaning up orphaned files using undo logs

2019-07-09 Thread Amit Kapila
On Sat, Jul 6, 2019 at 1:47 AM Robert Haas wrote: > > In fact, it seems to me that we shouldn't have any such thing as > "queue entries" at all. The queues should just be pointing to > RollbackHashEntry *, and we should add all the fields there that are > present in any of the "queue entry" struc

Re: range_agg

2019-07-09 Thread David Fetter
On Tue, Jul 09, 2019 at 09:40:59AM -0700, Paul A Jungwirth wrote: > On Tue, Jul 9, 2019 at 8:51 AM David Fetter wrote: > > > - A multirange type is an extra thing you get when you define a range > > > (just like how you get a tstzrange[]). Therefore > > > - I don't need separate commands to ad

Re: Copy data to DSA area

2019-07-09 Thread Thomas Munro
On Tue, Jun 25, 2019 at 12:53 AM Ideriha, Takeshi wrote: > I've rebased the patch to fit the core code rather than extension. > Regarding shared memory context (ShmContext), I added three > APIs: > - CreatePermShmContext > create "permanent" context located in shared memory > - CreateTempShmCont

Re: FETCH FIRST clause PERCENT option

2019-07-09 Thread Kyotaro Horiguchi
Hello. At Tue, 9 Jul 2019 21:56:32 -0600, Ryan Lambert wrote in > I did some more testing. I initialized a database with 1 million rows with > indexes and joins to test against and ran pgbench with a few different > settings for % to return. I started with a base query not utilizing the > new

make clean removes excesively

2019-07-09 Thread Kyotaro Horiguchi
I found the following make's behavior is annoying (at dab81b9953). make distclean ./configure make all make -j4 clean all relpath.c:21:10: fatal error: catalog/pg_tablespace_d.h: No such file or directory #include "catalog/pg_tablespace_d.h" (-j is needed, this happnes for me by -j2) Jus

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Dilip Kumar
On Wed, Jul 10, 2019 at 10:15 AM Amit Langote wrote: > > Hi Dilip, > > On Wed, Jul 10, 2019 at 1:29 PM Dilip Kumar wrote: > > On Wed, Jul 10, 2019 at 9:44 AM Dilip Kumar wrote: > > > On Fri, Nov 2, 2018 at 1:34 PM Amit Langote wrote: > > > > Okay, here are two patches: > > > > > > > > 0001 adds

doc: minor update for description of "pg_roles" view

2019-07-09 Thread Ian Barwick
Hi Here: https://www.postgresql.org/docs/12/view-pg-roles.html we state: "This view explicitly exposes the OID column of the underlying table, since that is needed to do joins to other catalogs." I think it's superfluous to mention this now OIDs are exposed by default; attached patch (

Re: Optimize partial TOAST decompression

2019-07-09 Thread Binguo Bao
Tomas Vondra 于2019年7月10日周三 上午5:12写道: > On Sat, Jul 06, 2019 at 05:23:37PM +0200, Tomas Vondra wrote: > >On Sat, Jul 06, 2019 at 02:27:56AM +0800, Binguo Bao wrote: > >>Hi, Tomas! > >>Thanks for your testing and the suggestion. > >> > >>That's quite bizarre behavior - it does work with a prefix, b

Re: Postgres 11: Table Partitioning and Primary Keys

2019-07-09 Thread Amit Langote
On Wed, Jul 10, 2019 at 7:53 AM Alvaro Herrera wrote: > On 2019-Jul-09, Amit Langote wrote: > > "While defining a primary key and unique constraints on partitioned > > tables is supported, the set of columns being constrained must include > > all of the partition key columns. This limitation exis

Re: SQL/JSON path issues/questions

2019-07-09 Thread Alexander Korotkov
On Mon, Jul 8, 2019 at 12:30 AM Alexander Korotkov wrote: > On Thu, Jul 4, 2019 at 4:38 PM Liudmila Mantrova > wrote: > > Thank you! > > > > I think we can make this sentence even shorter, the fix is attached: > > > > "To refer to a JSON element stored at a lower nesting level, add one or > > mo

Re: range_agg

2019-07-09 Thread Pavel Stehule
st 10. 7. 2019 v 6:26 odesílatel Paul A Jungwirth < p...@illuminatedcomputing.com> napsal: > On Tue, Jul 9, 2019 at 12:24 PM Pavel Stehule > wrote: > > út 9. 7. 2019 v 21:10 odesílatel Pavel Stehule > napsal: > >> I afraid so with generic multiragetype there lot of array > infrastructure will be

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-09 Thread Michael Paquier
On Tue, Jul 09, 2019 at 01:09:38PM +0200, Peter Eisentraut wrote: > You can already do that: Run a query through psql to get a list of > affected tables or indexes and feed those to reindexdb using -i or -t > options. Sure, but that's limited if one can only afford a limited amount of downtime for

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Amit Langote
Hi Dilip, On Wed, Jul 10, 2019 at 1:29 PM Dilip Kumar wrote: > On Wed, Jul 10, 2019 at 9:44 AM Dilip Kumar wrote: > > On Fri, Nov 2, 2018 at 1:34 PM Amit Langote wrote: > > > Okay, here are two patches: > > > > > > 0001 adds a new RelOptInfo member inh_root_parent that's set for > > > inheritanc

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-09 Thread Michael Paquier
On Tue, Jul 09, 2019 at 11:54:29AM -0700, Melanie Plageman wrote: > It might be worth post-processing results files to ignore row ordering > in some cases to allow for easier comparison. Has this been proposed > in the past? Not that I recall. -- Michael signature.asc Description: PGP signature

Re: shared-memory based stats collector

2019-07-09 Thread Kyotaro Horiguchi
At Thu, 04 Jul 2019 19:27:54 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190704.192754.27063464.horikyota@gmail.com> > #db #tbl #clients #iter #xactlen #referers > A1: 1 1 1 2100 > A2: 1 1 1 2101 > B

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Dilip Kumar
On Wed, Jul 10, 2019 at 9:44 AM Dilip Kumar wrote: > > On Fri, Nov 2, 2018 at 1:34 PM Amit Langote > wrote: > > > > On 2018/11/01 20:34, Dilip Kumar wrote: > > > On Mon, Oct 29, 2018 at 2:53 PM Amit Langote wrote: > > >> Anyway, why don't we just use the child table's AppendRelInfo to get the > >

Re: range_agg

2019-07-09 Thread Paul A Jungwirth
On Tue, Jul 9, 2019 at 12:24 PM Pavel Stehule wrote: > út 9. 7. 2019 v 21:10 odesílatel Pavel Stehule > napsal: >> I afraid so with generic multiragetype there lot of array infrastructure >> will be duplicated > > on second hand - it is true so classic array concat is not optimal for set of >

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-07-09 Thread Kyotaro Horiguchi
Hello. Rebased the patch to master(bd56cd75d2). regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From ac52e2c1c56a96c1745149ff4220a3a116d6c811 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Thu, 11 Oct 2018 10:03:21 +0900 Subject: [PATCH 1/3] TAP test for copy-truncation o

Re: range_agg

2019-07-09 Thread Paul A Jungwirth
On Tue, Jul 9, 2019 at 12:02 PM Jeff Davis wrote: > > - Multirange in/out work just like arrays, e.g. '{"[1,3)", "[5,6)"}' > > It would be cool to have a better text representation. We could go > simple like: > >'[1,3) [5,6)' Will that work with all ranges, even user-defined ones? With a tstz

Re: Remove page-read callback from XLogReaderState.

2019-07-09 Thread Kyotaro Horiguchi
Hello. The patch gets disliked by my tool chain. Fixed the usage of PG_USED_FOR_ASSERTS_ONLY and rebased to bd56cd75d2. -- Kyotaro Horiguchi NTT Open Source Software Center >From 1dabdce6993b73408b950cb8c348c4999178b9a0 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Thu, 18 Apr 2019 10:2

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Dilip Kumar
On Fri, Nov 2, 2018 at 1:34 PM Amit Langote wrote: > > On 2018/11/01 20:34, Dilip Kumar wrote: > > On Mon, Oct 29, 2018 at 2:53 PM Amit Langote wrote: > >> Anyway, why don't we just use the child table's AppendRelInfo to get the > >> parent's version of varattno instead of creating a new function?

Re: FETCH FIRST clause PERCENT option

2019-07-09 Thread Ryan Lambert
I did some more testing. I initialized a database with 1 million rows with indexes and joins to test against and ran pgbench with a few different settings for % to return. I started with a base query not utilizing the new functionality. The queries used are similar to my prior examples, code at [

Re: Minimal logical decoding on standbys

2019-07-09 Thread Robert Haas
On Tue, Jul 9, 2019 at 11:14 PM Andres Freund wrote: > > + if (xlrec->wal_level < WAL_LEVEL_LOGICAL) > > + ereport(ERROR, > > + > > (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > > +

Re: Minimal logical decoding on standbys

2019-07-09 Thread Andres Freund
Hi, Thanks for the new version! Looks like we're making progress towards something committable here. I think it'd be good to split the patch into a few pieces. I'd maybe do that like: 1) WAL format changes (plus required other changes) 2) Recovery conflicts with slots 3) logical decoding on stand

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Stephen Frost
Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > If a random number were generated instead its result would need to be > stored somewhere too, correct? Yes. Thanks, Stephen signature.asc Description: PGP signature

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Ryan Lambert
If a random number were generated instead its result would need to be stored somewhere too, correct? > That might also allow things like backup software to work > on these encrypted data files for page-level backups without needing > access to the key and that'd be pretty neat. +1 Ryan

Re: Implementing Incremental View Maintenance

2019-07-09 Thread Takuma Hoshiai
On Wed, 10 Jul 2019 11:07:15 +0900 Takuma Hoshiai wrote: > Hi Thomas, > > 2019年7月8日(月) 15:32 Thomas Munro : > > > On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata wrote: > > > Attached is a WIP patch of IVM which supports some aggregate functions. > > > > Hi Nagata-san and Hoshiai-san, > > > > Tha

Re: Index Skip Scan

2019-07-09 Thread Thomas Munro
On Wed, Jul 10, 2019 at 1:32 AM Jesper Pedersen wrote: > > While updating the Loose Index Scan wiki page with links to other > > products' terminology on this subject, I noticed that MySQL can > > skip-scan MIN() and MAX() in the same query. Hmm. That seems quite > > desirable. I think it requi

Re: coypu: "FATAL: sorry, too many clients already"

2019-07-09 Thread Tom Lane
Thomas Munro writes: > Several times on master[1] beginning with an initial occurrence 36 > days ago, and every time on REL_12_STABLE[2], but not on older > branches, build farm animal coypu has failed in the regression tests > with the error given in the subject. How can there be too many if > t

Re: Implementing Incremental View Maintenance

2019-07-09 Thread Takuma Hoshiai
Hi Thomas, 2019年7月8日(月) 15:32 Thomas Munro : > On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata wrote: > > Attached is a WIP patch of IVM which supports some aggregate functions. > > Hi Nagata-san and Hoshiai-san, > > Thank you for working on this. I enjoyed your talk at PGCon. I've > added Kevin

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Stephen Frost
Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > > What I think Tomas is getting at here is that we don't write a page only > > once. > > > A nonce of tableoid+pagenum will only be unique the first time we write > > out that page. Seems unlikely that we're only going to be writing the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Ryan Lambert
> What I think Tomas is getting at here is that we don't write a page only > once. > A nonce of tableoid+pagenum will only be unique the first time we write > out that page. Seems unlikely that we're only going to be writing these > pages once though- what we need is a nonce that's unique for *ev

Re: Extending PostgreSQL with a Domain-Specific Language (DSL) - Development

2019-07-09 Thread David G. Johnston
On Tue, Jul 9, 2019 at 5:43 PM Tom Mercha wrote: > I am still a bit of a novice with PostgreSQL internals. Could you please > provide some more detail on your comment regarding affecting permanent > session state? I was not referring to internals. BEGIN; CREATE TEMP TABLE tempdo (id int); DO $

coypu: "FATAL: sorry, too many clients already"

2019-07-09 Thread Thomas Munro
Hello, Several times on master[1] beginning with an initial occurrence 36 days ago, and every time on REL_12_STABLE[2], but not on older branches, build farm animal coypu has failed in the regression tests with the error given in the subject. How can there be too many if there are only 20 in a pa

Re: warning to publication created and wal_level is not set to logical

2019-07-09 Thread Tom Lane
Thomas Munro writes: > New status: Ready for Committer. If nobody wants to bikeshed the > wording or other details, I will commit this tomorrow. Hm, so: 1. + errmsg("insufficient wal_level to publish logical changes"), Might read better as "wal_level is insufficient to

Re: Extending PostgreSQL with a Domain-Specific Language (DSL) - Development

2019-07-09 Thread Tom Mercha
On 10/07/2019 02:31, David G. Johnston wrote: > On Tue, Jul 9, 2019 at 5:23 PM Tom Mercha wrote: > >> >> I understand that you never wrote any PL handler but was just thinking >> about this functionality as a follow-up to our conversation. I was just >> wondering whether anonymous DO blocks *must

Re: Extending PostgreSQL with a Domain-Specific Language (DSL) - Development

2019-07-09 Thread David G. Johnston
On Tue, Jul 9, 2019 at 5:23 PM Tom Mercha wrote: > > I understand that you never wrote any PL handler but was just thinking > about this functionality as a follow-up to our conversation. I was just > wondering whether anonymous DO blocks *must* return void or not? > > The docs for DO say it is a

Re: Extending PostgreSQL with a Domain-Specific Language (DSL) - Development

2019-07-09 Thread Tom Mercha
On 09/07/2019 23:22, Tomas Vondra wrote: > On Sun, Jul 07, 2019 at 11:06:38PM +, Tom Mercha wrote: >> On 06/07/2019 00:06, Tomas Vondra wrote: >>> First of all, it's pretty difficult to follow the discussion when it's >>> not clear what's the original message and what's the response. E-mail >>>

Re: warning to publication created and wal_level is not set to logical

2019-07-09 Thread Thomas Munro
On Tue, Jul 9, 2019 at 5:40 PM Lucas Viecelli wrote: > Follow the correct file, I added the wrong patch in the previous email New status: Ready for Committer. If nobody wants to bikeshed the wording or other details, I will commit this tomorrow. -- Thomas Munro https://enterprisedb.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Stephen Frost
Greetings, * Joe Conway (m...@joeconway.com) wrote: > On 7/9/19 5:42 PM, Tomas Vondra wrote: > > There are two basic ways to construct nonces - CSPRNG and sequences, and > > then a combination of both, i.e. one part is generated from a sequence > > and one randomly. > > > > FWIW not sure using OI

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Joe Conway
On 7/9/19 5:42 PM, Tomas Vondra wrote: > There are two basic ways to construct nonces - CSPRNG and sequences, and > then a combination of both, i.e. one part is generated from a sequence > and one randomly. > > FWIW not sure using OIDs as nonces directly is a good idea, as those are > inherently l

Re: Postgres 11: Table Partitioning and Primary Keys

2019-07-09 Thread Tom Lane
Alvaro Herrera writes: > That's a mild personal preference only though. Anyway, based on your > proposed wording, I wrote this: > > >Unique constraints on partitioned tables (as well as primary keys) >must constrain all the partition key columns. This limitation exi

Re: Postgres 11: Table Partitioning and Primary Keys

2019-07-09 Thread Alvaro Herrera
On 2019-Jul-09, Amit Langote wrote: > As mentioned in the docs, defining exclusion constraints on > partitioned tables is not supported. Right. > "While defining a primary key and unique constraints on partitioned > tables is supported, the set of columns being constrained must include > all of

Re: pg_receivewal documentation

2019-07-09 Thread Laurenz Albe
Jesper Pedersen wrote: > Thanks for the review, and the changes. > > However, I think it belongs in the --synchronous section, so what about > moving it there as attached ? Works for me. Marked as "ready for committer". Yours, Laurenz Albe

Re: progress report for ANALYZE

2019-07-09 Thread Alvaro Herrera
On 2019-Jul-08, Robert Haas wrote: > On Mon, Jul 8, 2019 at 2:18 PM Alvaro Herrera > wrote: > > Yeah, I got the impression that that was determined to be the desirable > > behavior, so I made it do that, but I'm not really happy about it > > either. We're not too late to change the CREATE INDEX

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 05:31:49PM -0400, Alvaro Herrera wrote: On 2019-Jul-09, Tomas Vondra wrote: On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote: > On 2019-Jul-09, Joe Conway wrote: > > > > Ot you could just encrypt them with a different key, and you would not > > > need to ma

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote: On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote: On 7/9/19 11:11 AM, Bruce Momjian wrote: > Good point about nonce and IV. I wonder if running the nonce > through the cipher with the key makes it random enough to use as an

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote: > >On 2019-Jul-09, Joe Conway wrote: > > > >>> Ot you could just encrypt them with a different key, and you would not > >>> need to make database OID part of the nonce.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Alvaro Herrera
On 2019-Jul-09, Tomas Vondra wrote: > On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote: > > On 2019-Jul-09, Joe Conway wrote: > > > > > > Ot you could just encrypt them with a different key, and you would not > > > > need to make database OID part of the nonce. > > > > > > Yeah tha

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote: On 2019-Jul-09, Joe Conway wrote: > Ot you could just encrypt them with a different key, and you would not > need to make database OID part of the nonce. Yeah that was pretty much exactly what I was trying to say above ;-) So yo

Re: Extending PostgreSQL with a Domain-Specific Language (DSL) - Development

2019-07-09 Thread Tomas Vondra
On Sun, Jul 07, 2019 at 11:06:38PM +, Tom Mercha wrote: On 06/07/2019 00:06, Tomas Vondra wrote: First of all, it's pretty difficult to follow the discussion when it's not clear what's the original message and what's the response. E-mail clients generally indent the original message with '>'

Re: Optimize partial TOAST decompression

2019-07-09 Thread Tomas Vondra
On Sat, Jul 06, 2019 at 05:23:37PM +0200, Tomas Vondra wrote: On Sat, Jul 06, 2019 at 02:27:56AM +0800, Binguo Bao wrote: Hi, Tomas! Thanks for your testing and the suggestion. That's quite bizarre behavior - it does work with a prefix, but not with suffix. And the exact ERROR changes after th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Alvaro Herrera
On 2019-Jul-09, Joe Conway wrote: > > Ot you could just encrypt them with a different key, and you would not > > need to make database OID part of the nonce. > > Yeah that was pretty much exactly what I was trying to say above ;-) So you need to decrypt each file and encrypt again when doing CRE

Re: [HACKERS] Cached plans and statement generalization

2019-07-09 Thread Konstantin Knizhnik
On 09.07.2019 15:16, Thomas Munro wrote: On Tue, Jul 9, 2019 at 7:32 AM Konstantin Knizhnik wrote: Sorry, are you tests autoprepare-16.patch I have sent in the last e-mail? I can not reproduce the problem with building documentation: + autoprepare_threshold (integer/type>) The problem

Re: [PATCH] Fix trigger argument propagation to child partitions

2019-07-09 Thread Alvaro Herrera
On 2019-Jul-09, Tomas Vondra wrote: > On Tue, Jul 09, 2019 at 03:00:27PM +0200, Patrick McHardy wrote: > > The following patch fixes propagation of arguments to the trigger > > function to child partitions both when initially creating the trigger > > and when adding new partitions to a partitioned

Re: Broken defenses against dropping a partitioning column

2019-07-09 Thread Tom Lane
Tomas Vondra writes: > On Mon, Jul 08, 2019 at 10:58:56AM -0400, Tom Lane wrote: >> So I think we're probably stuck with the approach of adding new internal >> dependencies. If we go that route, then our options for the released >> branches are (1) do nothing, or (2) back-patch the code that adds

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Joe Conway
On 7/9/19 4:12 PM, Tomas Vondra wrote: > On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote: >>On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote: >>> the input nonce used to generate the IV could be something like >>> pg_class.oid and blocknum concatenated together with some del

Re: benchmarking Flex practices

2019-07-09 Thread Tom Lane
John Naylor writes: > [ v4 patches for trimming lexer table size ] I reviewed this and it looks pretty solid. One gripe I have is that I think it's best to limit backup-prevention tokens such as quotecontinuefail so that they match only exact prefixes of their "success" tokens. This seems clear

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote: On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote: On 7/9/19 11:11 AM, Bruce Momjian wrote: > Good point about nonce and IV. I wonder if running the nonce > through the cipher with the key makes it random enough to use as an

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Joe Conway
On 7/9/19 3:50 PM, Bruce Momjian wrote: > On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote: >> On 7/9/19 11:11 AM, Bruce Momjian wrote: >> > Good point about nonce and IV. I wonder if running the nonce >> > through the cipher with the key makes it random enough to use as an >> > IV. >> >>

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote: > On 7/9/19 11:11 AM, Bruce Momjian wrote: > > Good point about nonce and IV. I wonder if running the nonce > > through the cipher with the key makes it random enough to use as an > > IV. > > Based on that NIST document it seems so. > > Th

Re: Ltree syntax improvement

2019-07-09 Thread Dmitry Belyavsky
On Mon, Jul 8, 2019 at 11:33 PM Alvaro Herrera wrote: > On 2019-Jul-08, Dmitry Belyavsky wrote: > > > I did not introduce any functions. I've just changed the parser. > > I mean the C-level functions -- count_parts_ors() and so on. > > Added a comment to count_parts_ors() The other functions in

Re: range_agg

2019-07-09 Thread Pavel Stehule
út 9. 7. 2019 v 21:10 odesílatel Pavel Stehule napsal: > > > út 9. 7. 2019 v 20:25 odesílatel Jeff Davis napsal: > >> On Tue, 2019-07-09 at 07:08 +0200, Pavel Stehule wrote: >> > >> > I am not against a multirange type, but I miss a explanation why you >> > introduce new kind of types and don't

Re: Broken defenses against dropping a partitioning column

2019-07-09 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 10:58:56AM -0400, Tom Lane wrote: Alvaro Herrera writes: That said, I'm not sure I see the use case for an ALTER TABLE .. DROP COLUMN command that turns a partitioned table (with existing partitions containing data) into one non-partitioned table with all data minus the

Re: range_agg

2019-07-09 Thread Pavel Stehule
út 9. 7. 2019 v 20:25 odesílatel Jeff Davis napsal: > On Tue, 2019-07-09 at 07:08 +0200, Pavel Stehule wrote: > > > > I am not against a multirange type, but I miss a explanation why you > > introduce new kind of types and don't use just array of ranges. > > > > Introduction of new kind of types

Re: range_agg

2019-07-09 Thread Paul Jungwirth
On 7/9/19 12:01 PM, Alvaro Herrera wrote: On 2019-Jul-08, Paul A Jungwirth wrote: - You can subscript a multirange like you do an array (? This could be a function instead.) Note that we already have a patch in the pipe to make subscripting an extensible operation, which would fit pretty well

Re: range_agg

2019-07-09 Thread Jeff Davis
On Mon, 2019-07-08 at 09:46 -0700, Paul A Jungwirth wrote: > - A multirange type is an extra thing you get when you define a range > (just like how you get a tstzrange[]). Therefore Agreed. > - I'm adding a new typtype for multiranges. ('m' in pg_type). Sounds reasonable. > - I'm just addin

Re: range_agg

2019-07-09 Thread Alvaro Herrera
On 2019-Jul-08, Paul A Jungwirth wrote: > - You can subscript a multirange like you do an array (? This could be > a function instead.) Note that we already have a patch in the pipe to make subscripting an extensible operation, which would fit pretty well here, I think. Also, I suppose you would

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-09 Thread Melanie Plageman
On Mon, Jul 8, 2019 at 12:21 PM Tom Lane wrote: > The point of regressplans.sh is to see if anything goes seriously > wrong when forcing non-default plan choices --- seriously wrong being > defined as crashes or semantically wrong answers. It's not expected > that the regression tests will autom

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-09 Thread Dave Cramer
On Mon, 22 Jan 2018 at 07:39, Robert Haas wrote: > On Sun, Jan 21, 2018 at 5:41 PM, Craig Ringer > wrote: > > If we'd done server_version_num in 9.5, for example, less stuff would've > > broken with pg10. > > Yeah, and if Tom hadn't forced it to be reverted from *8.2*, then > every version anyon

Re: range_agg

2019-07-09 Thread Jeff Davis
On Tue, 2019-07-09 at 07:08 +0200, Pavel Stehule wrote: > > I am not against a multirange type, but I miss a explanation why you > introduce new kind of types and don't use just array of ranges. > > Introduction of new kind of types is not like introduction new type. The biggest benefit, in my o

Re: Proposal to add GUC_REPORT to lc_monetary, lc_numeric and search_path

2019-07-09 Thread Dave Cramer
See attached patch. I think adding GUC_REPORT to search_path is probably the most important as this is potentially a security issue. See joe conway's blog on security and search path here https://info.crunchydata.com/blog/postgresql-defaults-and-impact-on-security-part-2 I also see there was a pr

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Joe Conway
On 7/9/19 11:11 AM, Bruce Momjian wrote: > On Tue, Jul 9, 2019 at 09:16:17AM -0400, Joe Conway wrote: >> On 7/9/19 8:39 AM, Ryan Lambert wrote: >> > Hi Thomas, >> > >> >> CBC mode does require >> >> random nonces, other modes may be fine with even sequences as long as >> >> the values are not reu

Re: Increasing default value for effective_io_concurrency?

2019-07-09 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 08:11:55PM -0400, Bruce Momjian wrote: On Wed, Jul 3, 2019 at 11:42:49AM -0400, Robert Haas wrote: On Wed, Jul 3, 2019 at 11:24 AM Tomas Vondra wrote: > Maybe. And it would probably work for the systems I used for benchmarks. > > It however assumes two things: (a) the s

Re: pg_receivewal documentation

2019-07-09 Thread Jesper Pedersen
Hi Laurenz, On 7/9/19 5:16 AM, Laurenz Albe wrote: On Thu, 2019-06-27 at 10:06 -0400, Jesper Pedersen wrote: Here is a patch for the pg_receivewal documentation to highlight that WAL isn't acknowledged to be applied. I think it is a good idea to document this, but I have a few quibbles with t

Re: [PATCH] Fix trigger argument propagation to child partitions

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 03:00:27PM +0200, Patrick McHardy wrote: The following patch fixes propagation of arguments to the trigger function to child partitions both when initially creating the trigger and when adding new partitions to a partitioned table. Thanks for the report and bugfix. It s

Re: range_agg

2019-07-09 Thread Paul A Jungwirth
On Tue, Jul 9, 2019 at 8:51 AM David Fetter wrote: > > - A multirange type is an extra thing you get when you define a range > > (just like how you get a tstzrange[]). Therefore > > - I don't need separate commands to add/drop multirange types. You get > > one when you define a range type, and

Development Environment

2019-07-09 Thread Igal @ Lucee.org
I have been wanting to contribute to the Postgres project for a while now, and I wanted to get some suggestions about the IDE and other tools that others are using (preferably, somewhat modern tools). Can anyone share what IDE they are using and if they have any other tips on setting up a deve

Re: range_agg

2019-07-09 Thread Paul A Jungwirth
On Mon, Jul 8, 2019 at 10:09 PM Pavel Stehule wrote: > po 8. 7. 2019 v 18:47 odesílatel Paul A Jungwirth > napsal: > I am not against a multirange type, but I miss a explanation why you > introduce new kind of types and don't use just array of ranges. Hi Pavel, I'm sorry, and thanks for your f

Re: clean up docs for v12

2019-07-09 Thread Justin Pryzby
I made bunch of changes based on Andres' review and I split some more indisputable 1 line changes from the large commit, hoping it will be easier to review both. Several bits and pieces of the patch have been applied piecemeal, but I was hoping to avoid continuing to do that. I think at least the

Re: range_agg

2019-07-09 Thread David Fetter
On Mon, Jul 08, 2019 at 09:46:44AM -0700, Paul A Jungwirth wrote: > On Sat, Jul 6, 2019 at 12:13 PM Jeff Davis wrote: > > > > On Fri, 2019-07-05 at 09:58 -0700, Paul A Jungwirth wrote: > > > user-defined range types. So how about I start on it and see how it > > > goes? I expect I can follow the e

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > I agree that all of that isn't necessary for an initial implementation, > > I was rather trying to lay out how we could improve on

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 09:16:17AM -0400, Joe Conway wrote: > On 7/9/19 8:39 AM, Ryan Lambert wrote: > > Hi Thomas, > > > >> CBC mode does require > >> random nonces, other modes may be fine with even sequences as long as > >> the values are not reused.    > > > > I disagree that CBC mode require

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > I agree that all of that isn't necessary for an initial implementation, > I was rather trying to lay out how we could improve on this in the > future and why having the keying done at a tabl

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jul 9, 2019 at 08:01:35AM -0400, Joe Conway wrote: > > On 7/9/19 6:07 AM, Peter Eisentraut wrote: > > > On 2019-07-08 18:09, Joe Conway wrote: > > >> In my mind, and in practice to a > > >> large extent, a postgres tablespace == a uni

Re: fix for BUG #3720: wrong results at using ltree

2019-07-09 Thread Oleg Bartunov
On Mon, Jul 8, 2019 at 7:22 AM Thomas Munro wrote: > > On Sun, Apr 7, 2019 at 3:46 AM Tom Lane wrote: > > =?UTF-8?Q?Filip_Rembia=C5=82kowski?= writes: > > > Here is my attempt to fix a 12-years old ltree bug (which is a todo item). > > > I see it's not backward-compatible, but in my understandin

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-09 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 09:28:42AM -0400, James Coleman wrote: On Mon, Jul 8, 2019 at 9:37 PM Tomas Vondra wrote: On Mon, Jul 08, 2019 at 12:07:06PM -0400, James Coleman wrote: > ... > >I guess I'm still not following. If (2) is responsible (currently) for >adding an explicit sort, why wouldn'

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 10:34:06AM +0200, Tomas Vondra wrote: > > I think the issues is that we can't use a _counter_ for the nonce since > > each page-0 of each table would use the same nonce, and each page-1, > > etc. I assume we would use the table oid and page number as the nonce. > > We can't

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 09:57:57PM -0600, Ryan Lambert wrote: > Hey everyone, > > Here is my input regarding nonces and randomness. > > > As I understand it, the NIST recommendation is a 96-bit *random* nonce, > > I could not find that exact requirement in the NIST documents, though given > the

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-09 Thread Paul Guo
Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could use some common code, but for Windows build, I'm not sure where are those window build files. Does anyone know about that? Thanks. On Tue, Jul 9, 2019 at 6:55 AM Thomas Munro wrote: > On Tue, Jul 2, 2019 at 5:46 PM Paul

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 06:45:50PM -0400, Bruce Momjian wrote: > On Mon, Jul 8, 2019 at 06:23:13PM -0400, Bruce Momjian wrote: > > Yes, 'postgres' can be used to create a nice md5 rainbow table that > > works on many servers --- good point. Are rainbow tables possible with > > something like AES?

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-09 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 08:01:35AM -0400, Joe Conway wrote: > On 7/9/19 6:07 AM, Peter Eisentraut wrote: > > On 2019-07-08 18:09, Joe Conway wrote: > >> In my mind, and in practice to a > >> large extent, a postgres tablespace == a unique mount point. > > > > But a critical difference is that in f

Re: Feature: Add Greek language fulltext search

2019-07-09 Thread Panagiotis Mavrogiorgos
On Thu, Jul 4, 2019 at 1:39 PM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2019-03-25 12:04, Panagiotis Mavrogiorgos wrote: > > Last November snowball added support for Greek language [1]. Following > > the instructions [2], I wrote a patch that adds fulltext search for > > Gr

Re: [HACKERS] WIP: Aggregation push-down

2019-07-09 Thread Antonin Houska
Richard Guo wrote: > Another rebase is needed for the patches. Done. -- Antonin Houska Web: https://www.cybertec-postgresql.com >From f656bd8d46afb9cb0a331cf3d34b9eed39f5e360 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Tue, 9 Jul 2019 15:30:13 +0200 Subject: [PATCH 1/3] Introduce Rel

Re: Index Skip Scan

2019-07-09 Thread Jesper Pedersen
Hi, On 7/4/19 6:59 AM, Thomas Munro wrote: For the MIN query you just need a path with Pathkeys: { i ASC, j ASC }, UniqueKeys: { i, j }, doing the MAX query you just need j DESC. David, are you thinking about something like the attached ? Some questions. * Do you see UniqueKey as a "comple

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-09 Thread James Coleman
On Mon, Jul 8, 2019 at 6:37 PM Alexander Korotkov wrote: > > On Thu, Jul 4, 2019 at 4:25 PM James Coleman wrote: > > Process questions: > > - Do I need to explicitly move the patch somehow to the next CF? > > We didn't manage to register it on current (July) commitfest. So, > please, register it

  1   2   >