Re: Proposed patch for key managment

2020-12-15 Thread Alastair Turner
Hi Bruce et al Firstly, thanks for shaping the patch, getting it down to a manageable scope of cluster file encryption. I think this is a great feature and it matters to a lot of the customers I talk to at VMware about adopting Postgres. Since it's exciting stuff, I've been trying to lash togethe

Re: Proposed patch for key managment

2020-12-16 Thread Alastair Turner
Hi Bruce On Wed, 16 Dec 2020 at 00:12, Bruce Momjian wrote: > ... > > The second approach is to make a new API for what you want I am trying to motivate for an alternate API. Specifically, an API which allows any potential adopter of Postgres and Cluster File Encryption to adopt them without

Re: Proposed patch for key managment

2020-12-16 Thread Alastair Turner
On Wed, 16 Dec 2020 at 21:32, Stephen Frost wrote: > > Greetings, > > * Alastair Turner (min...@decodable.me) wrote: > > On Wed, 16 Dec 2020 at 00:12, Bruce Momjian wrote: > > > The second approach is to make a new API for what you want > > > > I am

Re: Proposed patch for key managment

2020-12-16 Thread Alastair Turner
On Wed, 16 Dec 2020 at 22:43, Stephen Frost wrote: > > Greetings, ... > > If I'm following, you're suggesting something like: > > cluster_passphrase_command = 'aws get %q' > > and then '%q' gets replaced with "Please provide the WAL DEK: ", or > something like that? Prompting the user for each ke

Re: Proposed patch for key managment

2020-12-19 Thread Alastair Turner
Hi Stephen On Fri, 18 Dec 2020 at 21:36, Stephen Frost wrote: > > Greetings Alastair, > > * Alastair Turner (min...@decodable.me) wrote: > > On Wed, 16 Dec 2020 at 22:43, Stephen Frost wrote: ... > > passphrase key wrapper, the secret store and the cloud/HW KMS. &g

Re: Proposed patch for key managment

2020-12-19 Thread Alastair Turner
Hi Bruce On Sat, 19 Dec 2020 at 02:38, Bruce Momjian wrote: > > I am not going be as kind. Our workflow is: > > Desirability -> Design -> Implement -> Test -> Review -> Commit > https://wiki.postgresql.org/wiki/Todo#Development_Process > > I have already asked about the first ite

Re: Proposed patch for key managment

2020-12-20 Thread Alastair Turner
Thanks, Bruce On Sat, 19 Dec 2020 at 16:58, Bruce Momjian wrote: > ... > > To enable the direct injection of keys into the server, we would need a > new command for this, since trying to make the passphrase command do > this will lead to unnecessary complexity. The passphrase command should > do

Re: Proposed patch for key managment

2020-12-20 Thread Alastair Turner
Hi Stephen On Sun, 20 Dec 2020 at 22:59, Stephen Frost wrote: > ... > However, after chatting with Bruce about it for a bit this weekend, I'm > not sure that we need to tackle the second case today. I don't think > there's any doubt that there will be users who will want PG to manage > the keyri

Re: Proposed patch for key managment

2020-12-20 Thread Alastair Turner
Thanks Stephen, On Mon, 21 Dec 2020 at 00:33, Stephen Frost wrote: > > Greetings, > > * Alastair Turner (min...@decodable.me) wrote: ... > > > > What I'd like specifically is to have the option of an external > > keyring as a first class key store, where

Re: Proposed patch for key managment

2020-12-22 Thread Alastair Turner
Hi Bruce In ckey_passphrase.sh.sample + +echo "$PASS" | sha256sum | cut -d' ' -f1 + Under the threat model discussed, a copy of the keyfile could be attacked offline. So getting from passphrase to DEKs should be as resource intensive as possible to slow down brute-force attempts. Instead of just

Re: Proposed patch for key management

2021-01-01 Thread Alastair Turner
sumption is that a pg account could be > > > compromised, and the "master key" (if any, that is just one particular > > > cryptographic design) should not be accessible in that case. The first > > > barrier would be pg admin account. > > > >

Re: Proposed patch for key management

2021-01-02 Thread Alastair Turner
Hi Fabien On Sat, 2 Jan 2021 at 09:50, Fabien COELHO wrote: > ... > ISTM that pg at the core level should (only) directly provide: > > (1) a per-file encryption scheme, with loadable (hook-replaceable > functions??) to manage pages, maybe: > >encrypt(page_id, *key, *clear_page, *encrypted_pag

Re: Proposed patch for key management

2021-01-05 Thread Alastair Turner
Hi Bruce On Mon, 4 Jan 2021 at 18:23, Bruce Momjian wrote: > > On Fri, Jan 1, 2021 at 06:26:36PM +, Alastair Turner wrote: > > After the long intro, my question - If using a standard format, > > managed by a library, for the internal keystore does not result in a >

Re: Proposed patch for key management

2021-01-05 Thread Alastair Turner
On Mon, 4 Jan 2021 at 17:56, Bruce Momjian wrote: > > On Sat, Jan 2, 2021 at 12:47:19PM +, Alastair Turner wrote: > > > > There is also a further validation task - probably beyond the scope of > > the key management patch and into the encryption patch[es] territory

Re: Send duration output to separate log files

2024-07-11 Thread Alastair Turner
On Wed, 10 Jul 2024 at 16:58, Greg Sabino Mullane wrote: --snip-- > Why not build a more generic log filtering case? > > I looked into this, but it would be a large undertaking, given the way our > logging system works. And as per above, I don't think the pain would be > worth it, as duration cov

Re: Send duration output to separate log files

2024-07-22 Thread Alastair Turner
On Fri, 12 Jul 2024 at 15:58, Greg Sabino Mullane wrote: > On Thu, Jul 11, 2024 at 6:47 AM Alastair Turner > wrote: > >> The other category of logging which would benefit from a separate file >> is audit. It also can create massive volumes of log content. Splitting >

Re: Parallel copy

2020-02-14 Thread Alastair Turner
On Fri, 14 Feb 2020 at 11:57, Amit Kapila wrote: > On Fri, Feb 14, 2020 at 3:36 PM Thomas Munro > wrote: > > > > On Fri, Feb 14, 2020 at 9:12 PM Amit Kapila > wrote: ... > > > Another approach that came up during an offlist discussion with Robert > > > is that we have one dedicated worker fo

Re: Parallel copy

2020-02-15 Thread Alastair Turner
On Sat, 15 Feb 2020 at 04:55, Amit Kapila wrote: > > On Fri, Feb 14, 2020 at 7:16 PM Alastair Turner wrote: > > ... > > > > Parsing rows from the raw input (the work done by CopyReadLine()) in a > > single process would accommodate line returns in quoted fields. I

Re: Parallel copy

2020-02-26 Thread Alastair Turner
On Wed, 26 Feb 2020 at 10:54, Amit Kapila wrote: > > On Tue, Feb 25, 2020 at 9:30 PM Tomas Vondra > wrote: > > ... > > > > Perhaps. I guess it'll depend on the CSV file (number of fields, ...), > > so I still think we need to do some measurements first. > > > > Agreed. > > > I'm willing to > > do

Re: Count and log pages set all-frozen by vacuum

2024-10-31 Thread Alastair Turner
Hi Melanie On Wed, 30 Oct 2024 at 21:42, Melanie Plageman wrote: > ... > The number of pages set all-frozen in the visibility map by a given > vacuum can be useful when analyzing which normal vacuums reduce the > number of pages future aggressive vacuum need to scan. > > Also, empty pages that a

Re: Count and log pages set all-frozen by vacuum

2024-10-31 Thread Alastair Turner
On Thu, 31 Oct 2024 at 15:26, Melanie Plageman wrote: > On Thu, Oct 31, 2024 at 11:15 AM Peter Geoghegan wrote: > > > > On Thu, Oct 31, 2024 at 10:22 AM Melanie Plageman > > wrote: > > > It seems a better solution would be to find a way to document it or > > > phrase it clearly in the log. It i

Re: Count and log pages set all-frozen by vacuum

2024-11-01 Thread Alastair Turner
On Thu, 31 Oct 2024 at 18:51, Melanie Plageman wrote: > > Would it also be useful to have the number set all-visible? It seems > like if we added a new line prefixed with visibility map, it ought to > include all-visible and all-frozen then. > Something like this: > visibility map: %u pages set

Re: Proposal for Integrating Data Masking and anonymization into PostgreSQL

2024-10-23 Thread Alastair Turner
Hi Hosney On Wed, 23 Oct 2024, 15:17 Hosney Osman, wrote: > Dear PostgreSQL Development Team, > > I am writing to propose the development of native data masking and > anonymization features within PostgreSQL. As a long-time user of > PostgreSQL, > > I have observed a growing need for efficient a

Re: Add llvm version into the version string

2024-09-24 Thread Alastair Turner
On Mon, 23 Sept 2024 at 19:45, Tom Lane wrote: ... > Maybe another idea could be a new system view? > > => select * from pg_system_version; > property | value > > core version | 18.1 > architecture | x86_64-pc-linux-gnu > word size

Extensibility proposal - transaction start callbacks

2025-01-28 Thread Alastair Turner
I see that various comments in xact.c talk about start and end callbacks, but only end seems to be implemented at the moment. I also can't find any discussion about start of transaction callbacks in the mailing list archives. Would a patch in this direction be of interest? I was looking at this ba

Re: Built-in Raft replication

2025-04-18 Thread Alastair Turner
Hi Konstantin On Wed, 16 Apr 2025 at 15:07, Konstantin Osipov wrote: > * Alastair Turner [25/04/16 15:58]: > > > > > If you use build-in failover you have to resort to 3 big Postgres > > > machines because you need 2/3 majority. Of course, you can install > >

Re: Built-in Raft replication

2025-04-16 Thread Alastair Turner
On Wed, 16 Apr 2025 at 07:18, Ashutosh Bapat wrote: > On Wed, Apr 16, 2025 at 10:29 AM Andrey Borodin > wrote: > > > > If you use build-in failover you have to resort to 3 big Postgres > machines because you need 2/3 majority. Of course, you can install > MySQL-stype arbiter - host that had no r

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Alastair Turner
On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: > On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik > wrote: > > > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar > wrote: > > > > > > > > So logically for PostgreSQL its an > > > > user tab

Re: Proposal: Conflict log history table for Logical Replication

2025-09-07 Thread Alastair Turner
Hi Dilip Thanks for working on this, I think it will make conflict detection a lot more useful. On Sat, 6 Sept 2025, 10:38 Dilip Kumar, wrote: > While working on the patch, I see there are some open questions > > 1. We decided to pass the conflict history table name during > subscription creati

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Alastair Turner
On Wed, 10 Sept 2025 at 11:15, Dilip Kumar wrote: > On Wed, Sep 10, 2025 at 3:25 PM Amit Kapila > wrote: > > > ... > > > > How about having this as a built-in type? > > Here we will have to create a built-in type of type table which is I > think typcategory => 'C' and if we create this type it