Value of Transparent Data Encryption (TDE)

2019-09-30 Thread Bruce Momjian
* encrypts storage * perhaps easier to implement than file system encryption Is that accurate? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-01 Thread Bruce Momjian
.g. > keys into the server log. Even if proper key management prevents leaking > keys, there are still user data - say, credit card numbers, and such. Fortunately, the full-cluster encryption keys are stored encrypted in pg_control and are never accessible unencrypted at the SQL le

Re: Value of Transparent Data Encryption (TDE)

2019-10-01 Thread Bruce Momjian
On Tue, Oct 1, 2019 at 03:43:05PM +0200, Tomas Vondra wrote: > On Mon, Sep 30, 2019 at 05:40:52PM -0400, Bruce Momjian wrote: > Maybe. I think this is approaching the problem from the wrong angle. > Encryption is more a means of achieving something. OK, for compliance > purposes it&#x

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-01 Thread Bruce Momjian
s patch/proposal just propagates the same idea to all other > >> similar places I could find. > >> > >> ~ > >> > >> Result: > >> Less code. Removes ~200 unnecessary memsets. > >> More consistent initialisation. &g

Re: Value of Transparent Data Encryption (TDE)

2019-10-01 Thread Bruce Momjian
On Tue, Oct 1, 2019 at 11:54:26AM -0400, Bruce Momjian wrote: > On Tue, Oct 1, 2019 at 03:43:05PM +0200, Tomas Vondra wrote: > > Plus it allows features you can't easily achieve with fs encryption, > > because the filesystem only sees opaque data files. So having keys p

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-01 Thread Bruce Momjian
On Mon, Sep 30, 2019 at 05:26:33PM -0400, Bruce Momjian wrote: > For full-cluster Transparent Data Encryption (TDE), the current plan is > to encrypt all heap and index files, WAL, and all pgsql_tmp (work_mem > overflow). The plan is: > > > https://wiki.p

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Bruce Momjian
en issue a server log message if encryption is enabled and syslog is not being used. (I don't know how to test if syslog is being shipped to a remote server.) -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-04 Thread Bruce Momjian
to avoid giving any wrong impressions, > even fleeting ones. Yeah, it is certainly weird that you have to assign the first array element to get the rest to be zeros. By using a macro, we can document this behavior in one place. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Bruce Momjian
> * No support from hardware crypto acceleration. Aren't all modern systems going to have hardware crypto acceleration, i.e., AES-NI CPU extensions. Does that mean there is no value of partial encryption on such systems? Looking at the overhead numbers I have seen for AES-NI-enabled

Re: format of pg_upgrade loadable_libraries warning

2019-10-04 Thread Bruce Momjian
with them. The attached patch fixes the output text to say "in database", e.g.: Could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory in database: postgres in database: too Would intenting help to

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Bruce Momjian
On Fri, Oct 4, 2019 at 11:31:00PM +0200, Tomas Vondra wrote: > On Fri, Oct 04, 2019 at 03:57:32PM -0400, Bruce Momjian wrote: > > The current approach is to encrypt anything that contains user data, > > which includes heap, index, and WAL files. I think replication slots

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Bruce Momjian
On Fri, Oct 4, 2019 at 11:48:19PM +0200, Tomas Vondra wrote: > On Fri, Oct 04, 2019 at 04:58:14PM -0400, Bruce Momjian wrote: > > On Fri, Oct 4, 2019 at 10:46:57PM +0200, Tomas Vondra wrote: > > > Oracle also has a handy "TDE best practices" document [2], which s

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-04 Thread Bruce Momjian
On Sat, Oct 5, 2019 at 12:54:35AM +0200, Tomas Vondra wrote: > On Fri, Oct 04, 2019 at 06:06:10PM -0400, Bruce Momjian wrote: > > For full-cluster TDE with AES-NI-enabled, the performance impact is > > usually ~4%, so doing anything more granular doesn't seem useful.

Re: format of pg_upgrade loadable_libraries warning

2019-10-04 Thread Bruce Momjian
On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote: > On Fri, Oct 04, 2019 at 05:37:46PM -0400, Bruce Momjian wrote: > > On Wed, Oct 2, 2019 at 12:23:37PM -0500, Justin Pryzby wrote: > > > Regarding the previous thread and commit here: > > > https://www.pos

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Bruce Momjian
On Sat, Oct 5, 2019 at 09:13:59PM +0200, Tomas Vondra wrote: > On Fri, Oct 04, 2019 at 08:14:44PM -0400, Bruce Momjian wrote: > > On Sat, Oct 5, 2019 at 12:54:35AM +0200, Tomas Vondra wrote: > > > On Fri, Oct 04, 2019 at 06:06:10PM -0400, Bruce Momjian wrote: > > >

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Bruce Momjian
On Mon, Oct 7, 2019 at 09:44:30AM -0400, Robert Haas wrote: > On Fri, Oct 4, 2019 at 5:49 PM Bruce Momjian wrote: > > We spend a lot of time figuring out exactly how to safely encrypt WAL, > > heap, index, and pgsql_tmp files. The idea of doing this for another > > 20

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Bruce Momjian
On Mon, Oct 7, 2019 at 11:26:24AM -0400, Robert Haas wrote: > On Mon, Oct 7, 2019 at 11:02 AM Bruce Momjian wrote: > > For clog, it is not append-only, and bytes are rewritten (from zero to > > non-zero), so there would have to be a new nonce for every clog file > > write to

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Bruce Momjian
On Mon, Oct 7, 2019 at 12:30:37PM -0400, Robert Haas wrote: > On Mon, Oct 7, 2019 at 11:48 AM Bruce Momjian wrote: > > Well, I am starting with the things I _know_ need encrypting, and am > > then waiting for others to tell me what to add. Cybertec has not > > provided a

Re: format of pg_upgrade loadable_libraries warning

2019-10-07 Thread Bruce Momjian
On Fri, Oct 4, 2019 at 11:55:21PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote: > >> I would argue to include in 12.1, since 12 is what most everyone will use > >> for > >> upgrades, and pat

Re: PATCH: Add uri percent-encoding for binary data

2019-10-07 Thread Bruce Momjian
#x27;uri'); > + encode > +--- > + en%C0%DEd > +(1 row) > + > +SELECT decode('%De%c0%DEd', 'uri'); > + decode > + > + \xdec0de64 > +(1 row) > + > +SELECT decode('error%Ex', 'uri'); > +ERR

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-07 Thread Bruce Momjian
On Mon, Oct 7, 2019 at 09:40:22PM +0200, Tomas Vondra wrote: > On Mon, Oct 07, 2019 at 10:22:22AM -0400, Bruce Momjian wrote: > > > So essentially the argument is - if you have hw crypto acceleration (aka > > > AES-NI), then the overhead of all-cluster encryption is so low i

Re: format of pg_upgrade loadable_libraries warning

2019-10-08 Thread Bruce Momjian
On Mon, Oct 7, 2019 at 01:47:57PM -0400, Bruce Momjian wrote: > On Fri, Oct 4, 2019 at 11:55:21PM -0400, Tom Lane wrote: > > Bruce Momjian writes: > > > On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote: > > >> I would argue to include in 12.1, since

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-09 Thread Bruce Momjian
On Mon, Oct 7, 2019 at 12:34:36PM -0400, Bruce Momjian wrote: > On Mon, Oct 7, 2019 at 12:30:37PM -0400, Robert Haas wrote: > > On Mon, Oct 7, 2019 at 11:48 AM Bruce Momjian wrote: > > > Well, I am starting with the things I _know_ need encrypting, and am > > > then

Re: Transparent Data Encryption (TDE) and encrypted files

2019-10-23 Thread Bruce Momjian
salt to the value you store in the hash index, but when you are looking for a matching value, how do you know what salt to use to find it in the index? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: POC: converting Lists into arrays

2019-03-04 Thread Bruce Momjian
ems look at a structure, and a copy is required? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: POC: converting Lists into arrays

2019-03-04 Thread Bruce Momjian
on is aggregate palloc savings, then yes you > really do want to replace everything...) Could it be that allocating List* structures near the structure it points to is enough of a benefit in terms of cache hits that it is a loss when moving to a List* array? -- Bruce Momjian http:

Re: Remove Deprecated Exclusive Backup Mode

2019-03-04 Thread Bruce Momjian
> > > > Isn't this case problematic even when a backup was taken by pg_basebackup? > > Because of lack of recovery.signal, no archived WAL files are replayed and > > the database may not reach to the latest point. > > It is problematic because we have a message encou

Re: bgwriter_lru_maxpages limits in PG 10 sample conf

2019-03-04 Thread Bruce Momjian
reason to not fix bgwriter_lru_maxpages comment? Frankly, I was surprised postgresql.conf.sample was changed in a back branch since it will cause people who diff $PGDATA/postgresql.conf with share/postgresql.conf.sample to see differences they didn't make. -- Bruce Momjian http://mo

Re: bgwriter_lru_maxpages limits in PG 10 sample conf

2019-03-05 Thread Bruce Momjian
On Tue, Mar 5, 2019 at 12:24:14AM -0300, Alvaro Herrera wrote: > On 2019-Mar-04, Bruce Momjian wrote: > > > On Thu, Feb 28, 2019 at 10:28:44AM +0300, Sergei Kornilov wrote: > > > Hello > > > > > > postgresql.conf.sample was ch

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-03-06 Thread Bruce Momjian
//www.postgresql.org/message-id/20190222035816.uozqvc4wjyag3...@momjian.us -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: pg_upgrade version checking questions

2019-03-18 Thread Bruce Momjian
on, we could surely provide a useful default setting > based on find_my_exec. (I'm amused to notice that pg_upgrade > currently takes the trouble to find out its own path, and then does > precisely nothing with the information.) Good point. You are right that when it was outside of the

Re: pg_upgrade version checking questions

2019-03-18 Thread Bruce Momjian
On Tue, Mar 19, 2019 at 02:43:49AM -0400, Bruce Momjian wrote: > > 3. Actually, I'm kind of wondering why pg_upgrade has a --new-bindir > > option at all, rather than just insisting on finding the new-version > > executables in the same directory it is in. This seems like,

git head crash/regression failures

2019-03-28 Thread Bruce Momjian
est seem to be crashing from \d. For example, in src/test/regress/results/tablespace.out, once this line appears: \d testschema.test_index1 the file ends, while the expected file has many more lines. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: git head crash/regression failures

2019-03-28 Thread Bruce Momjian
On Thu, Mar 28, 2019 at 12:10:23PM -0300, Alvaro Herrera wrote: > On 2019-Mar-28, Bruce Momjian wrote: > > > I am seeing psql crash and massive regression test failures in git head. > > The psql crash happens if .psqlrc contains: > > > > \set COMP_KEYWORD_CAS

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 07:39:20PM -0400, Alvaro Herrera wrote: > On 2019-Jul-15, Bruce Momjian wrote: > > > My point is that doing encryption of only some data might actually make > > the system slower due to the lookups, so I think we need to implement > > all-cluster

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
ilitate online key rotation. Yes, if we do all-cluster encryption, we can just consult pg_control, but if we do per-table/index, that might be needed. There is another email suggesting symlink file to a key file could indicate an encrypted table/index. -- Bruce Momjian http://momjian.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote: > On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > then each row change gets its own LSN. You are asking if an update that > > just expires one row and adds it to a new page gets the same LSN. I > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
milarly, they also > can decrypt WAL as they can know the key id of WDEK. > > Master Key Rotation > > We will support new command-line tool that rotates the master key > offline. It accepts --old-encryption-key-passphrase-command option and > --new-encryption-k

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 01:18:44PM -0400, Bruce Momjian wrote: > > Key Management > > == > > We will use 3-tier key architecture as Joe proposed. > > > > 1. A master key encryption key (KEK): this is the ley supplied by the > > da

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
ach chunk, and I guess store the keys in the file system, encrypted with the master key. My big point is that we need to decide where the IV collisions will happen, and what our encryption limit per key (not per key/IV combination) is. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Fri, Jul 26, 2019 at 02:54:04AM +0900, Masahiko Sawada wrote: > On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian wrote: > > > > On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > > > I've re-considered the design of TDE feature based on the discussion

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
ame code to decrypt the page to add/remove checksums and rewrite it. I don't think we will ever allow add/remove encryption in online mode. Does that help? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
pected to address. Yes, I think we can say that checksums _help_ detect unauthorized database changes, and usually detects database corruption, but it isn't a fully secure solution. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.co

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote: > The second approach is to say they will collide and that we need to mix > a constant into the IV for tables/indexes and a different one for WAL. > In a way I would like to mix the pg_controldata Database system > Ide

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > After talking to Joe Conway, I just want to mention that if we decide > > that the LSN is unique among heap and index, or among heap or index, we

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > On 2019-Jul-15, Bruce Momjian wrote: > > > Uh, if someone modifies a few bytes of the page, we will decrypt it, but > > the checksum (per-page or WAL) will not match our decrypted output. How > > woul

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > > Greetings, > > > > > > * Bruce Momjian (br...@momjian.us)

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
which would avoid the problem of two clusters using the same key, but it wouldn't avoid the problem with promoting two replicas to primaries because they would have the same "Database system identifier" so I think it is just simpler to say "don't do that". -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
inly should be able to easily say "well, > there's no key for this, so just don't go through the decrypt/encrypt > routines". No, we can't easily do different keys for different things since all the keys have to be online for crash recovery, so there isn't much value to having different keys since they always have to be online. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote: > Masahiko Sawada copied this section as a desired direction, so I want to > drill down into it. I think we have five possible approaches for level > 3 listed above. > > The simplest approach would be to say that the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
additional keys for different tables, et al, which are encrypted with > the master key. Joe, I believe, covered all this quite well. Yes, I am disagreeing with that. I posted a 5-option email that went over the issue and explored the value in different keys. I am

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
f it was per key, and it means having a key per 1GB file, > that wouldn't change the point that I was making, so I'm not entirely > sure why it's being mentioned in this context. Because I thought we would use a single key for the entire cluster (heap/index/WAL), and only use another key for encryption key rotation. > I disagree with any approach that lacks a master key with additional > sub-keys, if that helps clarify things. We all know we need a passphrase that unlocks an encryption key. Are you saying we need per-object/table keys? Why, other than for key rotation? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 25, 2019 at 7:51 PM Bruce Momjian wrote: > > Looking at the bits we have, the IV for AES is 16 bytes.  Since we know > we have to use LSN (to change the IV for each page write), and the page >

Re: PG 12 draft release notes

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 08:51:34PM +0200, Laurenz Albe wrote: > On Sat, 2019-05-11 at 16:33 -0400, Bruce Momjian wrote: > > I have posted a draft copy of the PG 12 release notes here: > > I wonder if commits 0ba06e0bf and 40cfe8606 are worth mentioning > in the release

Re: PG 12 draft release notes

2019-07-25 Thread Bruce Momjian
And at the same time we > don't really add in the release notes low-level improvements like > these ones. Well, if we were reporting incorrect results before, that seems like a fix, with updated wording, of course, to mention just the fix. -- Bruce Momjian htt

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 09:11:18PM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 25, 2019 at 8:50 PM Bruce Momjian wrote: > > On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > > You can still use CTR mode and include those to make the key + IV

Re: PG 12 draft release notes

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 06:21:59PM -0700, Peter Geoghegan wrote: > On Wed, Jun 12, 2019 at 7:48 PM Bruce Momjian wrote: > > Great, patch applied. > > I think that it would make sense to have a v12 release note item for > amcheck's new "rootdescend"

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 10:57:08PM -0400, Alvaro Herrera wrote: > On 2019-Jul-25, Bruce Momjian wrote: > > > On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > > > > Why are we encrypting the page header in the first place? It seems to > > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Bruce Momjian
using LSN and page number, we will _never_ reuse the IV, except for cases like promoting two standbys, which I think we have to document as an insecure practice. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Bruce Momjian
On Sat, Jul 27, 2019 at 03:02:02PM -0400, Sehrope Sarkuni wrote: > On Sat, Jul 27, 2019 at 1:32 PM Bruce Momjian wrote: > > Uh, I listed the three options for the CRC and gave the benefits of > > each: > > > > > > https://www.postgresql.org/messag

Re: pg_upgrade fails with non-standard ACL

2019-07-27 Thread Bruce Momjian
problem exists since we added to pg_dump support of ACL changes of > pg_catalog functions in commit 23f34fa4b. > > I think this is a bug since it unpredictably affects user experience, so I > propose to backpatch the fix. > Script to reproduce the problem and the patch to fix it (cr

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-28 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 01:03:06PM -0400, Bruce Momjian wrote: > On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote: > > On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > > then each row change gets its own LSN. You are asking if an update that > > &g

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-29 Thread Bruce Momjian
be done via the creation of a derived key; this was covered here: https://www.postgresql.org/message-id/CAH7T-ap1Q9yHjGSO4ZJaVhU3L=u14tshmr++ccc_hk3eoqk...@mail.gmail.com -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-29 Thread Bruce Momjian
On Sun, Jul 28, 2019 at 10:33:03PM -0400, Bruce Momjian wrote: > I did some more research on which cases use a single LSN to modify > multiple 8k pages. The normal program flow is: > > XLogBeginInsert(); > ... > --> XLogRegisterBuffer(0, meta, ... >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-29 Thread Bruce Momjian
On Mon, Jul 29, 2019 at 04:09:52PM -0400, Alvaro Herrera wrote: > On 2019-Jul-27, Bruce Momjian wrote: > > > I think using LSN and page number, we will _never_ reuse the IV, except > > for cases like promoting two standbys, which I think we have to document > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-29 Thread Bruce Momjian
ent number in both clusters. We can't use the timeline in the WAL IV since they will both be on the same timeline. Anyway, I think the heap/index is still an issue so we should just document "don't do that". -- Bruce Momjian http://momjian.us EnterpriseDB

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-29 Thread Bruce Momjian
On Sun, Jul 28, 2019 at 10:33:03PM -0400, Bruce Momjian wrote: > I am thinking of writing some Assert() code that checks that all buffers > using a single LSN are from the same relation (and therefore different > page numbers). I would do it by creating a static array, clear

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Bruce Momjian
On Tue, Jul 30, 2019 at 07:44:20AM -0400, Sehrope Sarkuni wrote: > On Mon, Jul 29, 2019 at 8:35 PM Bruce Momjian wrote: > From the patch: > > /* > ! * The initialization vector (IV) is used for page-level > ! * encryption.  We use the LSN and page number as the IV, and IV >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-30 Thread Bruce Momjian
and for WAL it would be: uint64 segment_number; uint32counter; /* guarantees this IV doesn't match any relation IV */ uint32 2^32-1 /* all 1's */ Anyway, these are my thoughts so far. -- Bruce Momjian http://momjian.us

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
herefore if we can ensure the uniqueness of IV we can meet > that. Also it's not necessary to encrypt IV as it's okey to be > predictable. So I vote for CTR for at least for tables/indexes > encryption, there already might be consensus though. Yes, you are more likely to get duplic

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
p with changing also > already-flushed-WAL-records. So we might need to change the WAL write > so that it write only WAL records we actually wrote. Uh, I don't understand. We use the LSN to write the 8k page, and we use a different nonce scheme for the WAL. The LSN changes each time the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
. The 8k page LSN changes each time the page is modified, and the is part of the page nonce. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 04:11:03PM +0900, Masahiko Sawada wrote: > On Wed, Jul 31, 2019 at 5:48 AM Bruce Momjian wrote: > > I am thinking for the heap/index IV, it would be: > > > > uint64 lsn; > > unint32 page number; > > /* only uses 11

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-05 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 09:25:01AM -0400, Sehrope Sarkuni wrote: > On Tue, Jul 30, 2019 at 4:48 PM Bruce Momjian wrote: > > I had more time to think about the complexity of adding relfilenode to > the IV.  Since relfilenode is only unique within a database/tablespace, >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-06 Thread Bruce Momjian
counter. Therefore, even if you are encrypting the same 8k page several times in the WAL, the 8k page would be different because of the LSN (and other changes), and the bitstream you encrypt/XOR it with would be different because the counter would be different for that offset in the WAL. -- Bru

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-06 Thread Bruce Momjian
x page, and in WAL, with different key/nonce combinations, which I think is secure. What is insecure is to encrypt two different pieces of data with the same key/nonce combination. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-06 Thread Bruce Momjian
On Tue, Aug 6, 2019 at 01:55:38PM -0400, Bruce Momjian wrote: > CTR mode creates a bit stream for the first 16 bytes with nonce of > (segment_number, counter = 0), and the next 16 bytes with > (segment_number, counter = 1), etc. We only XOR using the parts of the > bit stream we

Re: Refactoring code stripping trailing \n and \r from strings

2019-08-06 Thread Bruce Momjian
the > attached with a new routine in common/string.c? Yes, I think this is a good idea. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + A

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-06 Thread Bruce Momjian
On Tue, Aug 6, 2019 at 06:13:30PM -0400, Jonathan Katz wrote: > Hi, > > On 8/6/19 3:01 PM, Bruce Momjian wrote: > > On Tue, Aug 6, 2019 at 01:55:38PM -0400, Bruce Momjian wrote: > >> CTR mode creates a bit stream for the first 16 bytes with nonce of > >> (segm

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Bruce Momjian
SN and CRC), and WAL (encrypt to the last byte), I assumed we would need to generate a bit stream of a specified size and do the XOR ourselves against the data. I assume ssh does this, so we would have to study the method. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 11:41:51AM -0400, Sehrope Sarkuni wrote: > On Wed, Aug 7, 2019 at 7:19 AM Bruce Momjian wrote: > > On Wed, Aug  7, 2019 at 05:13:31PM +0900, Masahiko Sawada wrote: > > I understood. IIUC in your approach postgres processes encrypt WAL >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > > On Wed, Jul 31, 2019 at 09:25:01AM -0400, Sehrope Sarkuni wrote: > > Even if we do not include a separate per-relation salt or things like >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
s a first pass I'd break it up > as separate writes with the encryption happening at write time. If that works > fine there's no need to complicate things further. OK, sounds like a plan! -- Bruce Momjian http://momjian.us EnterpriseDB h

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > I agree that all of that isn't necessary for an initial implementation, > > I was rather tryi

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Thu, Aug 8, 2019 at 03:07:59PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > > > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
ncrypted index isn't going to be free. It's not clear that > this would be much more expensive than if the entire index is encrypted, > or that people would actually be unhappy if there was such an additional > expense if it meant that they could have vacuum run without the keys. Yes, I think it is information leakage that is always going to make multi-key unable to fulfill all the features of cluster-wide encryption. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
details in how to implement it really helps, so I have no reply to these ideas. People can talk about whatever feature they want on these lists, but I have no way to help discussions that don't address facts. I will close with what I have already stated, that what people want

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
yption key, and the heap/index would already be using its own encryption key. Setting up such a system seems complicated. The big problem is that the base backup would use the primary key, unless we allowed pg_basebackup to decrypt/encrypt with a new heap/index key. -- Bruce Momjian

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
o encrypt shared buffers. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
On Thu, Aug 8, 2019 at 10:17:53PM -0400, Sehrope Sarkuni wrote: > On Thu, Aug 8, 2019 at 2:16 PM Bruce Momjian wrote: > > On Wed, Aug  7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > > Simplest approach for derived keys would be to use immutable attributes >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-10 Thread Bruce Momjian
On Fri, Aug 9, 2019 at 10:54:51PM -0400, Bruce Momjian wrote: > On Thu, Aug 8, 2019 at 10:17:53PM -0400, Sehrope Sarkuni wrote: > > On Thu, Aug 8, 2019 at 2:16 PM Bruce Momjian wrote: > > > > On Wed, Aug  7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-10 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > I was thinking the WAL would use the same key since the nonce is unique > between the two.  What value is there in using a different key? > Never having to

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 08:07:49PM -0400, Bruce Momjian wrote: > On Thu, Jul 11, 2019 at 12:18:47AM +0200, Tomas Vondra wrote: > > On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote: > > > Greetings, > > > > > > * Tomas Vondra (tomas.von...@2ndquad

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
nenrypted user data if it exists at the same page offsets. (The all-zero empty space is a huge known format marker area.) This is why CTR is so sensitive to reuse of the key/IV settings for encrypting different data. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Tue, Jul 30, 2019 at 04:48:31PM -0400, Bruce Momjian wrote: > I am not even clear if pg_upgrade preserving relfilenode is possible --- > when we wrap the relfilenode counter, does it start at 1 or at the > first-user-relation-oid? If the former, it could conflict with oids > ass

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
ing of encrypt(non-zero) at the same spot). No one replied to this comment, so I will state here that we never write zeros to the WAL and go back and write something else --- the WAL is append-only. We might do that for heap/index pages, but they would get a new LSN (and hence a new IV) when that happe

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > I don't think we want to add a MAC at this point since the MAC for 8k > pages seems unattainable. > > Even without a per-page MAC, a MAC at some l

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Sat, Aug 10, 2019 at 08:06:17AM -0400, Bruce Momjian wrote: > So, I just had an indea if we use separate encryption keys for > heap/index and for WAL --- we already know we will have an offline tool > that can rotate the passphrase or encryption keys. If we allow the > encryptio

Re: Bison state table

2019-08-13 Thread Bruce Momjian
On Tue, Aug 13, 2019 at 05:09:30PM +0700, John Naylor wrote: > On Sat, Jan 26, 2019 at 5:39 AM Bruce Momjian wrote: > > > > With our scanner keywords list now more cache-aware, and with us > > planning to use Bison for years to come, has anyone ever looked at > >

Re: pg_upgrade fails with non-standard ACL

2019-08-13 Thread Bruce Momjian
hen pg_upgrade does fail, at least we have a system that clearly points to the cause, but unfortunately usually at run-time, not at --check time. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

<    1   2   3   4   5   6   7   8   9   10   >