Re: Key management with tests

2021-04-06 Thread Neil Chen
Hi Bruce, I went through these patches and executed the test script you added for the KMS section, which looks all good. This is a point that looks like a bug - in patch 10, you changed the location and use of *RelFileNodeSkippingWAL()*, but the modified code logic seems different from the origin

Re: Key management with tests

2021-03-22 Thread Bruce Momjian
On Mon, Mar 22, 2021 at 08:38:37PM -0400, Bruce Momjian wrote: > > This particular patch (introducing the RelationIsPermanent() macro) > > seems like it'd be a nice thing to commit independently of the rest, > > reducing the size of this patch set..? > > Committed as suggested. Also, I have writ

Re: Key management with tests

2021-03-22 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote: > > src/backend/access/gist/gistutil.c | 2 +- > > src/backend/access/heap/heapam_handler.c | 2 +- > > src/backend/catalog/pg_publication.c | 2 +- > > src/backend/commands/tablecmds.c | 10 +- > > src/bac

Re: Key management with tests

2021-03-18 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 01:46:28PM -0400, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > This caught my attention because a comment says "encryption does not > > support WAL-skipped relations", but there's no direct change to the > > definition of RelFileNodeSkippingWA

Re: Key management with tests

2021-03-18 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 02:37:43PM -0300, Álvaro Herrera wrote: > On 2021-Mar-18, Stephen Frost wrote: > > This is discussed in src/backend/access/transam/README, specifically the > > section that talks about Skipping WAL for New RelFileNode. Basically, > > it's the 'wal_level=minimal' optimizatio

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > On 2021-Mar-18, Stephen Frost wrote: > > > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > > > it "relations that are not WAL-logged"? I suppose we alre

Re: Key management with tests

2021-03-18 Thread Alvaro Herrera
On 2021-Mar-18, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > > it "relations that are not WAL-logged"? I suppose we already have a > > term for this; I'm not sure it's a good idea to invent a

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > it "relations that are not WAL-logged"? I suppose we already have a > term for this; I'm not sure it's a good idea to invent a different term > that is only used

Re: Key management with tests

2021-03-18 Thread Alvaro Herrera
Patch 10 uses the term "WAL-skip relations". What does that mean? Is it "relations that are not WAL-logged"? I suppose we already have a term for this; I'm not sure it's a good idea to invent a different term that is only used in this new place. -- Álvaro Herrera39°

Re: Key management with tests

2021-03-18 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote: > > > I have made significant progress on the cluster file encryption feature so > > > it is time for me to p

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote: > > I have made significant progress on the cluster file encryption feature so > > it is time for me to post a new set of patches. > > Here is a rebase, to keep the cfbot green.

Re: Key management with tests

2021-02-07 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 07:53:18PM -0500, Bruce Momjian wrote: > On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote: > > > I disagree. If we only warn about some parts, attackers will just > > > attack other parts. It will also give users a false sense of security. > > > If you can ge

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote: > > I disagree. If we only warn about some parts, attackers will just > > attack other parts. It will also give users a false sense of security. > > If you can get the keys, it doesn't matter if there is one or ten ways > > of gettin

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > > I looked further. First, I don't think we are going to be able to > > > protect at all against users who have _write_ access on the OS running > > > Postgres. It would be t

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > I looked further. First, I don't think we are going to be able to > > protect at all against users who have _write_ access on the OS running > > Postgres. It would be too easy to just read process memory, or modify > > ~/.profile.

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > > I doubt anyone would actually stipulate that they *guarantee* detection > > > of malicious writes, and I do

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > I doubt anyone would actually stipulate that they *guarantee* detection > > of malicious writes, and I don't think we should either, but certainly > > the other system

Re: Key management with tests

2021-02-03 Thread Bruce Momjian
On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > I am thinking group read-access might be a requirement for cluster file > > encryption to be effective. > > People certainly do use group read-access, but I don't see that as being > a requirement for cluster file encryption to be

Re: Key management with tests

2021-02-03 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote: > > On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > The purpose of cluster file encryption is to prevent us

Re: Key management with tests

2021-02-02 Thread Bruce Momjian
On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote: > On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > The purpose of cluster file encryption is to prevent users with read > > > access to the directories used to store

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > The purpose of cluster file encryption is to prevent users with read > > access to the directories used to store database files and write-ahead > > log files from being able to acces

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Mon, Feb 1, 2021 at 06:34:53PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > > > I propose that we meet to discuss what approach we want to use to move TDE > > > forward.  We then start

Re: Key management with tests

2021-02-01 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > > I propose that we meet to discuss what approach we want to use to move TDE > > forward.  We then start a new thread with a proposal on the approach > > and finalize it via commun

Re: Key management with tests

2021-02-01 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 29, 2021 at 05:40:37PM -0500, Stephen Frost wrote: > > I hope it's pretty clear that I'm also very much in support of both this > > effort with the KMS and of TDE in general- TDE is specifically, > > Yes, thanks. I know we have p

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > I propose that we meet to discuss what approach we want to use to move TDE > forward.  We then start a new thread with a proposal on the approach > and finalize it via community consensus. I will invite Bruce, Stephen and > Masahiko to

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Fri, Jan 29, 2021 at 05:05:06PM +0900, Masahiko Sawada wrote: > TBH I’m confused a bit about the recent situation of this patch, but > I Yes, it is easy to get confused. > can contribute to KMS work by discussing, writing, reviewing, and > testing the patch. Also, I can work on the data encryp

Re: Key management with tests

2021-01-31 Thread Moon, Insung
Dear All. Thank you for all opinions and discussions regarding the KMS/TDE function. First of all, to get to the point of this email, I want to participate in anything I can do (review or development) when TDE related development is in progress. If there is a meeting related to it, I can't commun

Re: Key management with tests

2021-01-30 Thread Tom Kincaid
Thanks Stephen, Bruce and Masahiko, > > discussions so far and the point behind the design so that everyone > > can understand why this feature is designed in that way. To do that, > > it might be a good start to sort the wiki page since it has data > > encryption part, KMS, and ToDo mixed. > > I

Re: Key management with tests

2021-01-29 Thread Stephen Frost
Greetings, * Masahiko Sawada (sawada.m...@gmail.com) wrote: > On Fri, Jan 29, 2021 at 5:22 AM Bruce Momjian wrote: > > On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote: > > > I would also like to add a "not wanted" entry for this feature on the > > > TODO list, baaed on the fea

Re: Key management with tests

2021-01-29 Thread Masahiko Sawada
On Fri, Jan 29, 2021 at 5:22 AM Bruce Momjian wrote: > > On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote: > > I would also like to add a "not wanted" entry for this feature on the > > TODO list, baaed on the feature's limited usefulness, but I already > > asked about that a

Re: Key management with tests

2021-01-28 Thread Bruce Momjian
On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote: > I would also like to add a "not wanted" entry for this feature on the > TODO list, baaed on the feature's limited usefulness, but I already > asked about that and no one seems to feel we don't want it. > > > I want to avoi

Re: Key management with tests

2021-01-28 Thread Tom Kincaid
Hello, > > I don't think it makes sense to think about committing this to v14. I > > believe it only makes sense if we have a TDE patch that is relatively > > close to committable that can be used with it. I also don't think that > > this patch is in good enough shape to commit yet in terms of wh

Re: Key management with tests

2021-01-27 Thread Bruce Momjian
On Tue, Jan 26, 2021 at 05:53:01PM -0500, Bruce Momjian wrote: > On Tue, Jan 26, 2021 at 03:24:30PM -0500, Robert Haas wrote: > > I'm wondering whether you've considered storing all the keys in one > > file instead of a file per key. The reason I ask is that it seems to > > me that the key rotation

Re: Key management with tests

2021-01-26 Thread Robert Haas
On Tue, Jan 26, 2021 at 11:15 AM Bruce Momjian wrote: > This version fixes OpenSSL detection and improves docs for initdb > interactions. Hi, I'm wondering whether you've considered storing all the keys in one file instead of a file per key. The reason I ask is that it seems to me that the key r

Re: Key management with tests

2021-01-25 Thread Bruce Momjian
On Mon, Jan 25, 2021 at 08:12:01PM -0300, Álvaro Herrera wrote: > In patch 1, > > * The docs are not clear on what happens if --auth-prompt is not given > but an auth prompt is required for the program to work. Should it exit > with a status other than 0? Uh, I think the docs talk about this:

Re: Key management with tests

2021-01-25 Thread Alvaro Herrera
In patch 1, * The docs are not clear on what happens if --auth-prompt is not given but an auth prompt is required for the program to work. Should it exit with a status other than 0? * BootStrapKmgr claims it is called by initdb, but that doesn't seem to be the case. * Also, BootStrapKmgr is the

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 04:38:47PM -0500, Robert Haas wrote: > To me, it wouldn't make sense to commit a full README for a TDE > feature that we don't have yet with a key management patch, but the > way that they'll interact with each other has to be clear. The > doc/database-encryption.sgml file t

Re: Key management with tests

2021-01-18 Thread Tom Kincaid
I met with Bruce and Stephen this afternoon to discuss the feedback we received so far (prior to Robert's note which I haven't fully digested yet) on this patch. Here is what we plan to do: 1) Bruce is going to gather all the details from the Wiki and build a README for the TDE Key Management pa

Re: Key management with tests

2021-01-18 Thread Robert Haas
On Mon, Jan 18, 2021 at 2:00 PM Tom Kincaid wrote: > Some of the missing things you mention above are about the design of > TDE feature in general. However, this patch is about Key Management > which is going part of the larger TDE feature. So it feels as though > there is the need for a general

Re: Key management with tests

2021-01-18 Thread Andres Freund
On 2021-01-18 13:58:20 -0500, Bruce Momjian wrote: > On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote: > > Personally, but I admit that there's legitimate reasons to differ on > > that note, I don't think it's reasonable for a feature this invasive to > > commit preliminary patches wit

Re: Key management with tests

2021-01-18 Thread Tom Kincaid
> > I have to admit I was kind of baffled that the wiki page wasn't > > sufficient, because it is one of the longest Postgres feature > > explanations I have seen, but I now think the missing part is tying > > the wiki contents to the code implementation. If that is it, please > > confirm. If it

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote: > Personally, but I admit that there's legitimate reasons to differ on > that note, I don't think it's reasonable for a feature this invasive to > commit preliminary patches without the major subsequent patches being in > a shape that a

Re: Key management with tests

2021-01-18 Thread Andres Freund
Hi, On 2021-01-18 12:06:35 -0500, Bruce Momjian wrote: > On Mon, Jan 18, 2021 at 10:50:37AM -0500, Bruce Momjian wrote: > > OK, I looked at that and it is good, and I see my patch is missing that. > > Are people looking for me to take the wiki content, expand on it and tie > > it to the code that

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 10:50:37AM -0500, Bruce Momjian wrote: > OK, I looked at that and it is good, and I see my patch is missing that. > Are people looking for me to take the wiki content, expand on it and tie > it to the code that will be applied, or something else like all the > various crypto

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Sat, Jan 16, 2021 at 10:58:47PM -0800, Andres Freund wrote: > Hi, > > On 2021-01-17 11:54:57 +0530, Amit Kapila wrote: > > On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid > > wrote: > > > Admittedly I am a novice on this topic, and the majority of the > > > PostgreSQL source code, however I am ho

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Sun, Jan 17, 2021 at 11:54:57AM +0530, Amit Kapila wrote: > > Is there a design document for a Postgres feature of this size and > > scope that people feel would serve as a good example? Alternatively, > > is there a design document template that has been successfully used in > > the past? > >

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Sun, Jan 17, 2021 at 07:50:13PM -0500, Robert Haas wrote: > On Fri, Jan 15, 2021 at 7:56 PM Andres Freund wrote: > > I think that's not at all acceptable. I don't mind hashing out details > > on calls / off-list, but the design needs to be public, documented, and > > reviewable. And if it's so

Re: Key management with tests

2021-01-17 Thread Robert Haas
On Fri, Jan 15, 2021 at 7:56 PM Andres Freund wrote: > I think that's not at all acceptable. I don't mind hashing out details > on calls / off-list, but the design needs to be public, documented, and > reviewable. And if it's something the community can't understand, then > it can't get in. We're

Re: Key management with tests

2021-01-16 Thread Andres Freund
Hi, On 2021-01-17 11:54:57 +0530, Amit Kapila wrote: > On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid wrote: > > Admittedly I am a novice on this topic, and the majority of the > > PostgreSQL source code, however I am hopeful enough (those of you who > > know me understand that I suffer from eternal

Re: Key management with tests

2021-01-16 Thread Amit Kapila
On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid wrote: > > > > > I think that's not at all acceptable. I don't mind hashing out details > > > > on calls / off-list, but the design needs to be public, documented, and > > > > reviewable. And if it's something the community can't understand, then > > >

Re: Key management with tests

2021-01-16 Thread Tom Kincaid
> > > I think that's not at all acceptable. I don't mind hashing out details > > > on calls / off-list, but the design needs to be public, documented, and > > > reviewable. And if it's something the community can't understand, then > > > it can't get in. We're going to have to maintain this going

Re: Key management with tests

2021-01-15 Thread Michael Paquier
On Fri, Jan 15, 2021 at 08:20:36PM -0800, Andres Freund wrote: > On 2021-01-15 20:49:10 -0500, Bruce Momjian wrote: >> What Perl tap tests run initdb and manage the cluster? I didn't find >> any. > > find . -name '*.pl'|xargs grep 'use PostgresNode;' > > should give you a nearly complete list.

Re: Key management with tests

2021-01-15 Thread Andres Freund
Hi, On 2021-01-15 20:49:10 -0500, Bruce Momjian wrote: > On Fri, Jan 15, 2021 at 04:56:24PM -0800, Andres Freund wrote: > > On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote: > > > You have to understand cryptography and Postgres internals to understand > > > the design, and I don't think it is re

Re: Key management with tests

2021-01-15 Thread Bruce Momjian
On Fri, Jan 15, 2021 at 04:56:24PM -0800, Andres Freund wrote: > On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote: > > You have to understand cryptography and Postgres internals to understand > > the design, and I don't think it is realistic to explain that all to the > > community. We did much o

Re: Key management with tests

2021-01-15 Thread Andres Freund
Hi, On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote: > On Fri, Jan 15, 2021 at 02:37:56PM -0800, Andres Freund wrote: > > On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote: > > > > I am not even sure there is a consensus on the design, without which > > > > any commit is always premature. > > >

Re: Key management with tests

2021-01-15 Thread Andres Freund
Hi, On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote: > On Fri, Jan 15, 2021 at 04:23:22PM -0500, Robert Haas wrote: > > On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian wrote: > > I don't think that's appropriate. Several prominent community members > > have told you that the patch, as committed t

Re: Key management with tests

2021-01-15 Thread Bruce Momjian
On Fri, Jan 15, 2021 at 04:59:17PM -0500, Robert Haas wrote: > On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian wrote: > > If people want changes, I need to hear about it here. I have address > > everything people have mentioned in these threads so far. > > That does not match my perception of the

Re: Key management with tests

2021-01-15 Thread David G. Johnston
On Fri, Jan 15, 2021 at 2:59 PM Robert Haas wrote: > On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian wrote: > > If people want changes, I need to hear about it here. I have address > > everything people have mentioned in these threads so far. > > That does not match my perception of the situation

Re: Key management with tests

2021-01-15 Thread Robert Haas
On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian wrote: > If people want changes, I need to hear about it here. I have address > everything people have mentioned in these threads so far. That does not match my perception of the situation. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Key management with tests

2021-01-15 Thread Bruce Momjian
On Fri, Jan 15, 2021 at 04:23:22PM -0500, Robert Haas wrote: > On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian wrote: > > I am planning to apply this next week. > > I don't think that's appropriate. Several prominent community members > have told you that the patch, as committed the first time, nee

Re: Key management with tests

2021-01-15 Thread Robert Haas
On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian wrote: > I am planning to apply this next week. I don't think that's appropriate. Several prominent community members have told you that the patch, as committed the first time, needed a lot more work. There hasn't been enough time between then and now

Re: Key management with tests

2021-01-13 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:46:53PM -0500, Bruce Momjian wrote: > On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote: > > Well, we have eight unused bits in the IV, so we could just increment > > that for every hint bit change that uses the same LSN, and then force a > > dummy WAL record

Re: Key management with tests

2021-01-12 Thread Neil Chen
Thank you for your reply, On Wed, Jan 13, 2021 at 12:08 AM Stephen Frost wrote: > > No, we can't 'modify the page format as we wish'- if we change away from > using a C structure then we're going to be modifying quite a bit of > code which otherwise doesn't need to be changed. The proposed flag

Re: Key management with tests

2021-01-12 Thread Andres Freund
Hi, On 2021-01-11 20:12:00 +0900, Masahiko Sawada wrote: > diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c > index 32b5d62e1f..d474af753c 100644 > --- a/contrib/bloom/blinsert.c > +++ b/contrib/bloom/blinsert.c > @@ -177,6 +177,7 @@ blbuildempty(Relation index) >* XLOG_DB

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:57:11PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > Well, we have eight unused bits in the IV, so we coul

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Well, we have eight unused bits in the IV, so we could just increment > > > that for every hint bit change that uses the same LS

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Well, we have eight unused bits in the IV, so we could just increment > > that for every hint bit change that uses the same LSN, and then force a > > dummy WAL record when that 8-bit count

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > I don't think there's any doubt that we need to make sure that the IV is > > distinct and advancing the LSN to get a new one when needed for this > > case seems like i

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > > I think one big question is that, since we are using a streaming cipher, > > > do we care about hint bit changes showing to users? I actually don't > > > know. If we do, so

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > I think one big question is that, since we are using a streaming cipher, > > do we care about hint bit changes showing to users? I actually don't > > know. If we do, some kind of dummy LSN record might be required, as you > > sugg

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > > This says: > > > > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > > > wal_log_hints will be enabled auto

Re: Key management with tests

2021-01-12 Thread Andres Freund
On 2021-01-12 13:03:14 -0500, Bruce Momjian wrote: > I think one big question is that, since we are using a streaming cipher, > do we care about hint bit changes showing to users? I actually don't > know. If we do, some kind of dummy LSN record might be required, as you > suggested. That'd lead

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > This says: > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > wal_log_hints will be enabled automatically in encryption mode. > > > > Does that help? > > IIUC it h

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Neil Chen (carpenter.nail...@gmail.com) wrote: > On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost wrote: > > This is an interesting question but ultimately I don't think we should > > be looking at this from the perspective of allowing arbitrary changes to > > the page format. The cha

Re: Key management with tests

2021-01-12 Thread Masahiko Sawada
On Tue, Jan 12, 2021 at 11:09 AM Bruce Momjian wrote: > > On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote: > > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > > Right, or ensure that the actual IV used is distinct (such as by using > > > another bit in the IV to distingui

Re: Key management with tests

2021-01-11 Thread Neil Chen
Hi Stephen, On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost wrote: > > This is an interesting question but ultimately I don't think we should > be looking at this from the perspective of allowing arbitrary changes to > the page format. The challenge is that much of the page format, today, > is d

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote: > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > Right, or ensure that the actual IV used is distinct (such as by using > > another bit in the IV to distinguish logged-vs-unlogged), but it seems > > saner to just use a diff

Re: Key management with tests

2021-01-11 Thread Masahiko Sawada
On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > > Although, another approach and one that I've discussed a bit with Bruce, > > > is to have more keys- such as a

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote: > > outputs from the GCM encryption and is what provides the integrity / > > authentication of the encrypted data to be able to detect if it's been > > modified. Unfortunately, wh

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > > > Yes, and it avoids the issue of using a single key for too much, which > > > is also a concern. The re

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > > Yes, and it avoids the issue of using a single key for too much, which > > is also a concern. The remaining larger issues are to figure out a > > place to put the tag for each

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > > Although, another approach and one that I've discussed a bit with Bruce, > > > is to have more keys- su

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > Although, another approach and one that I've discussed a bit with Bruce, > > is to have more keys- such as a key for temporary files, and perhaps > > even a key for logged rela

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > Although, another approach and one that I've discussed a bit with Bruce, > is to have more keys- such as a key for temporary files, and perhaps > even a key for logged relations and a different for unlogged.. Or Yes, we have to make

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote: > > Looking at the patch, it supports three algorithms but only > > PG_CIPHER_AES_KWP is used in the core for now: > > > > +/* > > + * Supported symmetric encryption algorithm.

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote: > On Sun, Jan 10, 2021 at 11:51 PM Bruce Momjian wrote: > > OK, here they are with numeric prefixes. It was actually tricky to > > figure out how to create a squashed format-patch based on another branch. > > Thank you for attachin

Re: Key management with tests

2021-01-11 Thread Masahiko Sawada
On Sun, Jan 10, 2021 at 11:51 PM Bruce Momjian wrote: > > On Sun, Jan 10, 2021 at 06:04:12PM +1300, Thomas Munro wrote: > > On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian wrote: > > > Does anyone know why the cfbot applied the patch listed second first > > > here? > > > > > > http://cfbot.

Re: Key management with tests

2021-01-09 Thread Thomas Munro
On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian wrote: > Does anyone know why the cfbot applied the patch listed second first > here? > > http://cfbot.cputube.org/patch_31_2925.log > > Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed > it would apply attachments in th

Re: Key management with tests

2021-01-09 Thread Bruce Momjian
On Sat, Jan 9, 2021 at 08:08:16PM -0500, Bruce Momjian wrote: > On Sat, Jan 9, 2021 at 01:17:36PM -0500, Bruce Momjian wrote: > > I know we are still working on the hex patch (dest-len) and the crypto > > tests, but I wanted to post this so people can see where we are, and we > > can get some cur

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote: > > > Anyway, I think we need to figure out how to trim. The first part would > > > be to figure out whether we need 128 _and_ 256-bit tests, and then see > > > what items are rea

Re: Key management with tests

2021-01-08 Thread Bruce Momjian
On Fri, Jan 8, 2021 at 03:34:23PM -0500, Stephen Frost wrote: > > All the tests pass now. The current src/test directory is 19MB, and > > adding these tests takes it to 23MB, or a 20% increase. That seems like > > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer > > tests, o

Re: Key management with tests

2021-01-08 Thread Bruce Momjian
On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote: > > No, I don't think so. Stephen imported the entire NIST test suite. It > > was so comperhensive, it detected several OpenSSL bugs for zero-length > > strings, which I already reported, but we would never be encrypting > > zero-leng

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings Bruce, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote: > > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > > > I have completed the key management patch with tests created by Stephen > > > Frost. Original patch

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote: > > On 2021-Jan-07, Bruce Momjian wrote: > > > > > All the tests pass now. The current src/test directory is 19MB, and > > > adding these tests takes it to 23MB, or a 20% increa

Re: Key management with tests

2021-01-07 Thread Bruce Momjian
On Thu, Jan 7, 2021 at 10:02:14AM -0500, Bruce Momjian wrote: > My next step is to add the high-level tests. Here is the high-level script, and the log output. I used the pg_upgrade test.sh as a model. It uses "CFE DEBUG" lines that are already in the code to compare the initdb encryption with

Re: Key management with tests

2021-01-07 Thread Bruce Momjian
On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote: > On 2021-Jan-07, Bruce Momjian wrote: > > > All the tests pass now. The current src/test directory is 19MB, and > > adding these tests takes it to 23MB, or a 20% increase. That seems like > > a lot. It is testing 128-bit and 256-b

Re: Key management with tests

2021-01-07 Thread Alvaro Herrera
On 2021-Jan-07, Bruce Momjian wrote: > All the tests pass now. The current src/test directory is 19MB, and > adding these tests takes it to 23MB, or a 20% increase. That seems like > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer > tests, or just test 256, or use gzip to c

Re: Key management with tests

2021-01-07 Thread Bruce Momjian
On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote: > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > > I have completed the key management patch with tests created by Stephen > > Frost. Original patch by Masahiko Sawada. It requires the hex > > reorganization patch fi

Re: Key management with tests

2020-12-31 Thread Bruce Momjian
On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > I have completed the key management patch with tests created by Stephen > Frost. Original patch by Masahiko Sawada. It requires the hex > reorganization patch first. The key patch is now 2.1MB because of the > tests, so attaching i