Re: pg_receivewal documentation

2019-07-25 Thread Michael Paquier
On Wed, Jul 24, 2019 at 03:03:04PM +0200, Jehan-Guillaume de Rorthais wrote: > Unless I am missing something, another solution might be to use a dedicated > role to pg_receive{xlog|wal} with synchronous_commit lower than > remote_apply. Aren't you confused by the same thing as I was upthread [1]?

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Sergei Kornilov
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, failed Spec compliant: not tested Documentation:tested, passed Hi I did some review and have few notes about behavior. reindex

add_path() for Path without InitPlan: cost comparison vs. Paths that require one

2019-07-25 Thread Dent John
Hi folks, I’ve run into a planning conundrum with my query rewriting extension for MVs when attempting to rewrite a RECURSIVE CTE. RECURSIVE CTEs are expensive — and presumably tricky to optimise — and so a good use case for query rewrite against an MV; all the more so if Yugo’s Incremental Vi

Re: pg_receivewal documentation

2019-07-25 Thread Jehan-Guillaume de Rorthais
On Thu, 25 Jul 2019 16:58:17 +0900 Michael Paquier wrote: > On Wed, Jul 24, 2019 at 03:03:04PM +0200, Jehan-Guillaume de Rorthais wrote: > > Unless I am missing something, another solution might be to use a dedicated > > role to pg_receive{xlog|wal} with synchronous_commit lower than > > remote_a

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Julien Rouhaud
Thanks for the review! On Thu, Jul 25, 2019 at 10:17 AM Sergei Kornilov wrote: > > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed > Implements feature: tested, failed > Spec compliant: not tested > Documentation:

Re: dropdb --force

2019-07-25 Thread Pavel Stehule
čt 25. 7. 2019 v 5:11 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > [ drop-database-force-20190708.patch ] > > I took a brief look at this, but I don't think it's really close to > being committable. > > * The documentation claims FORCE will fail if you don't have privileges > to termi

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Sergei Kornilov
Hi Thank you, v9 code and behavior looks good for me. Builds cleanly, works with older releases. I'll mark Ready to Commiter. > I don't have a strong opinion here. If we change for >=, it'd be > better to also adapt vacuumdb for consistency. I didn't change it for > now, to stay consistent with

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Michael Paquier
On Thu, Jul 25, 2019 at 12:12:40PM +0300, Sergei Kornilov wrote: > Thank you, v9 code and behavior looks good for me. Builds cleanly, > works with older releases. I'll mark Ready to Commiter. The restriction with --jobs and --system is not documented, and that's good to have from the start. This

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Sergei Kornilov
Hi >>>  I don't have a strong opinion here. If we change for >=, it'd be >>>  better to also adapt vacuumdb for consistency. I didn't change it for >>>  now, to stay consistent with vacuumdb. >> >>  Yep, no strong opinion from me too. > > My opinion tends towards consistency. Consistency sounds go

Re: POC: Cleaning up orphaned files using undo logs

2019-07-25 Thread Amit Kapila
On Wed, Jul 24, 2019 at 2:45 PM Amit Kapila wrote: > > On Thu, Jul 18, 2019 at 5:10 PM Amit Kapila wrote: > > > > > Yep, that was completely wrong. Here's a new version. > > > > > > > One comment/question related to > > 0022-Use-undo-based-rollback-to-clean-up-files-on-abort.patch. > > > > I hav

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Julien Rouhaud
On Thu, Jul 25, 2019 at 12:03 PM Michael Paquier wrote: > > On Thu, Jul 25, 2019 at 12:12:40PM +0300, Sergei Kornilov wrote: > > Thank you, v9 code and behavior looks good for me. Builds cleanly, > > works with older releases. I'll mark Ready to Commiter. > > The restriction with --jobs and --syst

Re: Index Skip Scan

2019-07-25 Thread Kyotaro Horiguchi
Hello. At Wed, 24 Jul 2019 22:49:32 +0200, Dmitry Dolgov <9erthali...@gmail.com> wrote in > > On Mon, Jul 22, 2019 at 7:10 PM Jesper Pedersen > > wrote: > > > > On 7/22/19 1:44 AM, David Rowley wrote: > > > Here are the comments I noted down during the review: > > > > > > cost_index: > > > > >

Re: Initdb failure

2019-07-25 Thread Rafia Sabih
On Thu, 25 Jul 2019 at 07:39, vignesh C wrote: > > Hi, > > Initdb fails when following path is provided as input: > datasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsda

Re: POC: Cleaning up orphaned files using undo logs

2019-07-25 Thread vignesh C
Hi Thomas, Few review comments on 0003-Add-undo-log-manager.patch: 1) Upgrade may fail +/* + * Compute the new redo, and move the pg_undo file to match if necessary. + * Rather than renaming it, we'll create a new copy, so that a failure that + * occurs before the controlfile is rewritten won't be

Re: Index Skip Scan

2019-07-25 Thread Kyotaro Horiguchi
Sorry, there's a too-hard-to-read part. At Thu, 25 Jul 2019 20:17:37 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190725.201737.192223037.horikyota@gmail.com> > Hello. > > At Wed, 24 Jul 2019 22:49:32 +0200, Dmitry Dolgov <9erthali...@gmail.com> > wrote in > > > On Mon, Jul

Re: Initdb failure

2019-07-25 Thread vignesh C
On Thu, Jul 25, 2019 at 4:52 PM Rafia Sabih wrote: > > On Thu, 25 Jul 2019 at 07:39, vignesh C wrote: > > > > Hi, > > > > Initdb fails when following path is provided as input: > > datasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadf

Re: Built-in connection pooler

2019-07-25 Thread Konstantin Knizhnik
Hello Ryan, Thank you very much for review and benchmarking. My answers are inside. On 25.07.2019 0:58, Ryan Lambert wrote: Applying the patch [1] has improved from v9, still getting these: Fixed.  used a DigitalOcean droplet with 2 CPU and 2 GB RAM and SSD for this testing, Ubuntu 18.04. 

Re: add_path() for Path without InitPlan: cost comparison vs. Paths that require one

2019-07-25 Thread Tom Lane
Dent John writes: > However, if I factor back in the cost of the InitPlan, things net out much > more in favour of a scan against the MV. Of course, the add_path() comparison > logic doesn’t include the InitPlan cost, so the point is moot. Please explain yourself. InitPlans will, as a rule, g

Re: Initdb failure

2019-07-25 Thread Rafia Sabih
On Thu, 25 Jul 2019 at 13:50, vignesh C wrote: > > On Thu, Jul 25, 2019 at 4:52 PM Rafia Sabih wrote: > > > > On Thu, 25 Jul 2019 at 07:39, vignesh C wrote: > > > > > > Hi, > > > > > > Initdb fails when following path is provided as input: > > > datasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsa

Re: pg_upgrade version checking questions

2019-07-25 Thread Daniel Gustafsson
> On 24 Jul 2019, at 22:32, Peter Eisentraut > wrote: > > On 2019-07-23 17:30, Daniel Gustafsson wrote: >> The reason for moving is that we print default values in usage(), and that >> requires the value to be computed before calling usage(). We already do this >> for resolving environment valu

Re: Initdb failure

2019-07-25 Thread Tom Lane
Rafia Sabih writes: > On Thu, 25 Jul 2019 at 13:50, vignesh C wrote: >>> Initdb fails when following path is provided as input: >>> datasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdat

Re: sepgsql seems rather thoroughly broken on Fedora 30

2019-07-25 Thread Tom Lane
Mike Palmiotto writes: > On Fri, Jul 19, 2019 at 4:29 PM Tom Lane wrote: >> I can confirm that the 0001 patch fixes things on my Fedora 30 box. >> So that's good, though I don't know enough to evaluate it for style >> or anything like that. > I think the policy is in need of review/rewriting any

Re: Initdb failure

2019-07-25 Thread Rafia Sabih
On Thu, 25 Jul 2019 at 16:44, Tom Lane wrote: > > Rafia Sabih writes: > > On Thu, 25 Jul 2019 at 13:50, vignesh C wrote: > >>> Initdb fails when following path is provided as input: > >>> datasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafddsdatasadfasfdsafdds

Re: [proposal] de-TOAST'ing using a iterator

2019-07-25 Thread Binguo Bao
Hi John! Sorry for the late reply. It took me some time to fix a random bug. In the case where we don't know the slice size, how about the other > aspect of my question above: Might it be simpler and less overhead to > decompress entire chunks at a time? If so, I think it would be > enlightening t

Re: [GSoC] artbufmgr

2019-07-25 Thread pantilimonov michael
Greetings all, > So, this is a plan, that I would like to stick with subsequently: > > 1) Work on synchronization. > 2) Examine bulk operations on buffers, replace them with tree (prefix) > iterator. > 3) A reasonable way to size memory for tree maintenance. > previously i described the plan, th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 07:39:20PM -0400, Alvaro Herrera wrote: > On 2019-Jul-15, Bruce Momjian wrote: > > > My point is that doing encryption of only some data might actually make > > the system slower due to the lookups, so I think we need to implement > > all-cluster encryption and then see wha

Re: "localtime" value in TimeZone

2019-07-25 Thread Shay Rojansky
> Yeah, this is something that some tzdb packagers do --- they put a > "localtime" file into /usr/share/zoneinfo that is a symlink or hard link > to the active zone file, and then initdb tends to seize on that as being > the shortest available spelling of the active zone. I see, I wasn't aware tha

Re: add_path() for Path without InitPlan: cost comparison vs. Paths that require one

2019-07-25 Thread Dent John
Hi Tom, > On 25 Jul 2019, at 14:25, Tom Lane wrote: > > Please explain yourself. InitPlans will, as a rule, get stuck into the > same place in the plan tree regardless of which paths are chosen; that's > why we need not consider them in path cost comparisons. Ah that’s true. I didn’t realise t

Re: Initdb failure

2019-07-25 Thread vignesh C
On Thu, Jul 25, 2019 at 8:39 PM Rafia Sabih wrote: > > On Thu, 25 Jul 2019 at 16:44, Tom Lane wrote: > > > > Rafia Sabih writes: > > > On Thu, 25 Jul 2019 at 13:50, vignesh C wrote: > > >>> Initdb fails when following path is provided as input: > > >>> datasadfasfdsafddsdatasadfasfdsafddsdatasa

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Sat, Jul 20, 2019 at 03:39:25PM -0400, Sehrope Sarkuni wrote: > How about storing the CRC of the encrypted pages? It would not leak > any additional information and serves the same purpose as a > non-encrypted one, namely I/O corruption detection. I took a look at > pg_checksum and besides check

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote: > On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > then each row change gets its own LSN. You are asking if an update that > > just expires one row and adds it to a new page gets the same LSN. I > > don't know. > > The fo

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > I've re-considered the design of TDE feature based on the discussion > so far. The one of the main open question is the granular of > encryption objects: cluster encryption or more-granular-than-cluster > encryption. The followings

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 01:18:44PM -0400, Bruce Momjian wrote: > > Key Management > > == > > We will use 3-tier key architecture as Joe proposed. > > > > 1. A master key encryption key (KEK): this is the ley supplied by the > > database admin using something akin to ssl_passphra

Re: Fetching timeline during recovery

2019-07-25 Thread Jehan-Guillaume de Rorthais
Hello, On Wed, 24 Jul 2019 14:33:27 +0200 Jehan-Guillaume de Rorthais wrote: > On Wed, 24 Jul 2019 09:49:05 +0900 > Michael Paquier wrote: > > > On Tue, Jul 23, 2019 at 06:05:18PM +0200, Jehan-Guillaume de Rorthais > > wrote: [...] > > I think that there are arguments for being more flexible

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Masahiko Sawada
On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian wrote: > > On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > > I've re-considered the design of TDE feature based on the discussion > > so far. The one of the main open question is the granular of > > encryption objects: cluster encryp

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote: > In my email I linked the wrong page for [2]. The correct one is here: > [2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html > > Following that, I think we could end up with three tiers: > > 1. A master key encryption ke

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Fri, Jul 26, 2019 at 02:54:04AM +0900, Masahiko Sawada wrote: > On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian wrote: > > > > On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > > > I've re-considered the design of TDE feature based on the discussion > > > so far. The one of the m

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Sat, Jul 20, 2019 at 07:30:30PM +0200, Tomas Vondra wrote: > Forbid checksums? I don't see how that could be acceptable. We either have > to accept the limitations of the current design (having to decrypt > everything before checking the checksums) or change the design. Yes, checksums certainly

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Fri, Jul 19, 2019 at 01:59:41PM +0200, Tomas Vondra wrote: > On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote: > > We can guarantee integrity and authenticity of backup, but that's a separate > > feature: someone may need this although it's o.k. for him to run the cluster > > unenc

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote: > The second approach is to say they will collide and that we need to mix > a constant into the IV for tables/indexes and a different one for WAL. > In a way I would like to mix the pg_controldata Database system > Identifier into ther

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > After talking to Joe Conway, I just want to mention that if we decide > that the LSN is unique among heap and index, or among heap or index, we > will need to make sure future WAL records retain this uniqueness. One thing comes to mind regard

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Alvaro Herrera
On 2019-Jul-15, Bruce Momjian wrote: > Uh, if someone modifies a few bytes of the page, we will decrypt it, but > the checksum (per-page or WAL) will not match our decrypted output. How > would they make it match the checksum without already knowing the key. > I read [1] but could not see that e

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > After talking to Joe Conway, I just want to mention that if we decide > > that the LSN is unique among heap and index, or among heap or index, we > > will need to make sure

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > Greetings, > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > After talking to Joe Conway, I just want to mention that if we decide > > > that the LSN is unique among hea

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > On 2019-Jul-15, Bruce Momjian wrote: > > > Uh, if someone modifies a few bytes of the page, we will decrypt it, but > > the checksum (per-page or WAL) will not match our decrypted output. How > > would they make it match the checks

[DOC] Document auto vacuum interruption

2019-07-25 Thread James Coleman
We've discussed this internally many times, but today finally decided to write up a doc patch. Autovacuum holds a SHARE UPDATE EXCLUSIVE lock, but other processes can cancel autovacuum if blocked by that lock unless the autovacuum is to prevent wraparound.This can result in very surprising behavio

Question about MemoryContexts / possible memory leak in CachedPlanSource usage

2019-07-25 Thread Daniel Migowski
Hello, I am just starting to get my feet wet with PostgreSQL development and am starting to understand the source, so please be kind 😊. I am working on the REL_11_4 tag. When CachedPlanSource instances are created the field query_string is filled with pstrdup(query_string) in CreateCachedPlan,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > > Greetings, > > > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > After talking to Joe Conway,

Re: Question about MemoryContexts / possible memory leak in CachedPlanSource usage

2019-07-25 Thread Andres Freund
Hi, On 2019-07-25 20:21:06 +, Daniel Migowski wrote: > When CachedPlanSource instances are created the field query_string is > filled with pstrdup(query_string) in CreateCachedPlan, > plancache.c:182, which is just a wrapper for strdup. According to the > docs the returned pointer should be fr

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-07-25 Thread Tom Lane
Thomas Munro writes: > FWIW this is now fixed for FreeBSD 13-CURRENT, with a good chance of > back-patch. I don't know if there are any other operating systems > that are shipping zoneinfo but failing to install zone1970.tab, but if > there are it's a mistake IMHO and they'll probably fix that if

Re: "localtime" value in TimeZone

2019-07-25 Thread Tom Lane
Shay Rojansky writes: > I followed the conversations you linked to, and disagreements seem to be > mostly about other aspects of timezone selection. Does it make sense to > have a limited, restricted conversation specifically about avoiding > "localtime"? I've tried to kick-start the other thread

AW: Question about MemoryContexts / possible memory leak in CachedPlanSource usage

2019-07-25 Thread Daniel Migowski
Ah, you are right, I looked in fe_memutils.c. Makes sense now, thanks!! -Ursprüngliche Nachricht- Von: Andres Freund Gesendet: Donnerstag, 25. Juli 2019 22:31 An: Daniel Migowski Cc: pgsql-hack...@postgresql.org Betreff: Re: Question about MemoryContexts / possible memory leak in Cach

Re: psql - add SHOW_ALL_RESULTS option

2019-07-25 Thread Daniel Verite
Fabien COELHO wrote: > Attached a "do it always version", which does the necessary refactoring. > There is seldom new code, it is rather moved around, some functions are > created for clarity. Thanks for the update! FYI you forgot to remove that bit: --- a/src/bin/psql/tab-complete.c +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 06:08:28PM -0400, Sehrope Sarkuni wrote: > Hi, > > Some more thoughts on CBC vs CTR modes. There are a number of > advantages to using CTR mode for page encryption. > > CTR encryption modes can be fully parallelized, whereas CBC can only > parallelized for decryption. Whil

Re: Server crash due to assertion failure in CheckOpSlotCompatibility()

2019-07-25 Thread Andres Freund
Hi, On 2019-05-30 16:31:39 +0530, Ashutosh Sharma wrote: > > *Analysis:* > > I did some quick investigation on this and found that when the aggregate > > is performed on the first group i.e. group by 'a', all the input tuples are > > fetched from the outer plan and stored into the tuplesort object

Re: Server crash due to assertion failure in CheckOpSlotCompatibility()

2019-07-25 Thread Andres Freund
Hi, On 2019-06-10 22:42:52 -0400, Alvaro Herrera wrote: > On 2019-May-30, Andres Freund wrote: > > On 2019-05-30 16:31:39 +0530, Ashutosh Sharma wrote: > > > Here are some more details on the crash reported in my previous e-mail for > > > better clarity: > > > > I'll look into this once pgcon is

Re: psql - add SHOW_ALL_RESULTS option

2019-07-25 Thread Fabien COELHO
Bonsoir Daniel, FYI you forgot to remove that bit: + "SINGLELINE|SINGLESTEP|SHOW_ALL_RESULTS")) Indeed. I found another such instance in "help.c". Also copydml does not seem to be exercised with combined queries, so do we need this chunk: --- a/src/test/regress/sql/copydml.sql Yep,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > > After

Re: ON CONFLICT (and manual row locks) cause xmax of updated tuple to unnecessarily be set

2019-07-25 Thread Andres Freund
Hi, On 2019-07-24 17:14:39 -0700, Peter Geoghegan wrote: > On Wed, Jul 24, 2019 at 4:24 PM Andres Freund wrote: > > but we really don't need to do any of that in this case - the only > > locker is the current backend, after all. > > > > I think this isn't great, because it'll later will cause unn

Re: On the stability of TAP tests for LDAP

2019-07-25 Thread Thomas Munro
On Thu, Jul 25, 2019 at 12:51 PM Michael Paquier wrote: > Thanks for the updated patch, this looks good. I have done a series > of tests keeping my laptop busy and I haven't seen a failure where I > usually see problems 10%~20% of the time. So that seems to help, > thanks! Pushed, thanks. --

Re: On the stability of TAP tests for LDAP

2019-07-25 Thread Michael Paquier
On Fri, Jul 26, 2019 at 10:18:48AM +1200, Thomas Munro wrote: > Pushed, thanks. Thanks for fixing! I'll update this thread if there are still some problems. -- Michael signature.asc Description: PGP signature

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 05:50:57PM -0400, Stephen Frost wrote: > > > > pg_upgrade seems immune to must of this, and that is by design. > > > > However, I am hesitant to change the heap/index page format for > > > > encryption because if we add fields, old pages might not fit as > > > > encrypted p

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 05:50:57PM -0400, Stephen Frost wrote: > > > > > pg_upgrade seems immune to must of this, and that is by design. > > > > > However, I am hesitant to change the heap/index page format for > > > > > encryption because if

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote: > Masahiko Sawada copied this section as a desired direction, so I want to > drill down into it. I think we have five possible approaches for level > 3 listed above. > > The simplest approach would be to say that the LSN/page-number a

Re: Compile from source using latest Microsoft Windows SDK

2019-07-25 Thread Michael Paquier
On Thu, Jul 25, 2019 at 09:02:14AM +0900, Michael Paquier wrote: > Interesting. I am not actually sure in which version of VS this has > been introduced. But it would be fine enough to do nothing if the > variable is not defined and rely on the default. Except for the > formatting and indentatio

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 07:41:14PM -0400, Stephen Frost wrote: > > You are right that we can allow it online, but we haven't been > > discussing these cases since it is easy to do this because we have > > access to the keys. I do think whatever code we use for checksum online > > changes will be u

Re: ON CONFLICT (and manual row locks) cause xmax of updated tuple to unnecessarily be set

2019-07-25 Thread Peter Geoghegan
On Thu, Jul 25, 2019 at 3:10 PM Andres Freund wrote: > > I agree that this is unfortunate. Are you planning on working on it? > > Not at the moment, no. Are you planning / hoping to take a stab at it? The current behavior ought to be fixed, and it seems like it falls to me to do that. OTOH, anyth

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 07:41:14PM -0400, Stephen Frost wrote: > > > You are right that we can allow it online, but we haven't been > > > discussing these cases since it is easy to do this because we have > > > access to the keys. I do think

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Sehrope Sarkuni
On Thu, Jul 25, 2019 at 7:51 PM Bruce Momjian wrote: > Looking at the bits we have, the IV for AES is 16 bytes. Since we know > we have to use LSN (to change the IV for each page write), and the page > number (so WAL updates that change multiple pages with the same LSN use > different IVs), that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 08:07:28PM -0400, Stephen Frost wrote: > > Yes, we need to see how we are going to do that for checksums and > > encryption and come up with a plan. > > This is already being done though- Andres has a patch posted already and > my recollection from a quick look at that is t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 25, 2019 at 7:51 PM Bruce Momjian wrote: > > Looking at the bits we have, the IV for AES is 16 bytes.  Since we know > we have to use LSN (to change the IV for each page write), and the page > number (so WA

Re: Seek failure at end of FSM file during WAL replay (in 11)

2019-07-25 Thread Michael Paquier
On Wed, Jul 24, 2019 at 01:30:42PM -0400, Tom Lane wrote: > Hm. AFAICS the immediate issuer of the error must have been > _mdnblocks(); there are other matches to that error string but > they are in places where we can tell which file the seek must > have been applied to, and it wasn't a FSM file.

Re: PG 12 draft release notes

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 08:51:34PM +0200, Laurenz Albe wrote: > On Sat, 2019-05-11 at 16:33 -0400, Bruce Momjian wrote: > > I have posted a draft copy of the PG 12 release notes here: > > I wonder if commits 0ba06e0bf and 40cfe8606 are worth mentioning > in the release notes. They make "pg_test_f

Re: PG 12 draft release notes

2019-07-25 Thread Bruce Momjian
On Tue, Jul 16, 2019 at 03:26:31PM +0900, Michael Paquier wrote: > On Mon, Jul 15, 2019 at 08:51:34PM +0200, Laurenz Albe wrote: > > I wonder if commits 0ba06e0bf and 40cfe8606 are worth mentioning > > in the release notes. They make "pg_test_fsync" work correctly > > on Windows for the first time

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Sehrope Sarkuni
On Thu, Jul 25, 2019 at 8:50 PM Bruce Momjian wrote: > On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > > You can still use CTR mode and include those to make the key + IV unique > by > > adding them to the derived key rather than the IV. > > > > The IV per-page would still be L

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 09:11:18PM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 25, 2019 at 8:50 PM Bruce Momjian wrote: > > On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > > You can still use CTR mode and include those to make the key + IV unique > by > > adding

Re: PG 12 draft release notes

2019-07-25 Thread Bruce Momjian
On Mon, Jul 15, 2019 at 06:21:59PM -0700, Peter Geoghegan wrote: > On Wed, Jun 12, 2019 at 7:48 PM Bruce Momjian wrote: > > Great, patch applied. > > I think that it would make sense to have a v12 release note item for > amcheck's new "rootdescend" verification option: > > https://git.postgresql

Re: PG 12 draft release notes

2019-07-25 Thread Peter Geoghegan
On Thu, Jul 25, 2019 at 6:37 PM Bruce Momjian wrote: > Attached patch applied, thanks. Thanks Bruce, -- Peter Geoghegan

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Jonathan S. Katz
Buffer Encryption == We will use AES-CBC for buffer encryption. We will add key id (4byte) >>> >>> I think we might want to use CTR for this, and will post after this. > > Not sure if I missed this post or not (as several people mentioned, it > is easy to get lost in th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Jonathan S. Katz
Hi, Before my reply, I wanted to say that I've been lurking on this thread for a bit as I've tried to better inform myself on encryption at rest and how it will apply to what we want to build. I actually built a (poor) prototype in Python of the key management system that Joe & Masahiko both laid

Re: Implementing Incremental View Maintenance

2019-07-25 Thread Yugo Nagata
Hi, I've updated the wiki page of Incremental View Maintenance. https://wiki.postgresql.org/wiki/Incremental_View_Maintenance On Thu, 11 Jul 2019 13:28:04 +0900 Yugo Nagata wrote: > Hi Thomas, > > Thank you for your review and discussion on this patch! > > > > 2019年7月8日(月) 15:32 Thomas Munro

Re: Patch for SortSupport implementation on inet/cdir

2019-07-25 Thread Peter Geoghegan
On Fri, Feb 8, 2019 at 11:13 PM Edmund Horner wrote: > I have some comments on the comments: Seems reasonable to me. Where are we on this? I'd like to get the patch committed soon. -- Peter Geoghegan

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Alvaro Herrera
On 2019-Jul-25, Bruce Momjian wrote: > On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > > Why are we encrypting the page header in the first place? It seems to > > me that the encrypted area should cover only the line pointers and the > > tuple data area; the page header needs t

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-25 Thread Michael Paquier
On Thu, Jul 25, 2019 at 01:00:34PM +0200, Julien Rouhaud wrote: > The problem is that a user doing something like: > > reindexdb -j48 -i some_index -S s1 -S s2 -S s3 > > will probably be disappointed to learn that he has to run a specific > command for the index(es) that should be reindexed.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-25 Thread Alvaro Herrera
On 2019-Jul-25, Alvaro Herrera wrote: > > Uh, there are no known attacks on AES with known plain-text, e.g., SSL > > uses AES, so I think we are good with encrypting everything after the > > first 16 bytes. > > Well, maybe there aren't any attacks *now*, but I don't know what will > happen in the

Re: POC: Cleaning up orphaned files using undo logs

2019-07-25 Thread Amit Kapila
On Thu, Jul 25, 2019 at 11:22 AM Thomas Munro wrote: > > On Wed, Jul 24, 2019 at 9:15 PM Amit Kapila wrote: > > I have done some more review of undolog patch series and here are my > > comments: > > Hi Amit, > > Thanks! There a number of actionable changes in your review. I'll be > posting a n

Re: SegFault on 9.6.14

2019-07-25 Thread Amit Kapila
On Tue, Jul 23, 2019 at 5:28 PM Amit Kapila wrote: > > On Tue, Jul 23, 2019 at 9:11 AM Thomas Munro wrote: > > > > > Another idea from the band-aid-solutions-that-are-easy-to-back-patch > > department: in ExecutePlan() where we call ExecShutdownNode(), we > > could write EXEC_FLAG_DONE into estat

Re: psql - add SHOW_ALL_RESULTS option

2019-07-25 Thread Kyotaro Horiguchi
Hello. At Thu, 25 Jul 2019 21:42:11 + (GMT), Fabien COELHO wrote in > > Bonsoir Daniel, > > > FYI you forgot to remove that bit: > > > > + "SINGLELINE|SINGLESTEP|SHOW_ALL_RESULTS")) > > Indeed. I found another such instance in "help.c". > > > Also copydml does not seem to be exercised

pqParseInput3 overruns

2019-07-25 Thread Kyotaro Horiguchi
Hello. While looking [1], I noticed that NOTICE messages of the next command is processed before PQgetResult returns. Clients can receive such spurious NOTICE messages. Looking pqParseInput3, its main loop seems considered to exit after complete messages is processed. (As I read.) > * Loop to pa

Warning messages appearing twice

2019-07-25 Thread vignesh C
Hi, I have noticed in some cases the warning messages appear twice, one such instance is given below: postgres=# begin; BEGIN postgres=# prepare transaction 't1'; PREPARE TRANSACTION postgres=# rollback; *WARNING: there is no transaction in progressWARNING: there is no transaction in progress*

Re: POC: Cleaning up orphaned files using undo logs

2019-07-25 Thread Dilip Kumar
On Thu, Jul 25, 2019 at 11:25 AM Dilip Kumar wrote: > > Hi Thomas, > > I have started reviewing 0003-Add-undo-log-manager, I haven't yet > reviewed but some places I noticed that instead of UndoRecPtr you are > directly > using UndoLogOffset. Which seems like bugs to me > > 1. > +UndoRecPtr > +Un

Re: block-level incremental backup

2019-07-25 Thread Jeevan Ladhe
Hi Vignesh, Please find my comments inline below: 1) If relation file has changed due to truncate or vacuum. > During incremental backup the new files will be copied. > There are chances that both the old file and new file > will be present. I'm not sure if cleaning up of the > o

Re: Warning messages appearing twice

2019-07-25 Thread Dilip Kumar
On Fri, Jul 26, 2019 at 11:04 AM vignesh C wrote: > > Hi, > > I have noticed in some cases the warning messages appear twice, one such > instance is given below: > postgres=# begin; > BEGIN > postgres=# prepare transaction 't1'; > PREPARE TRANSACTION > postgres=# rollback; > WARNING: there is no

Re: Warning messages appearing twice

2019-07-25 Thread vignesh C
On Fri, Jul 26, 2019 at 11:23 AM Dilip Kumar wrote: > On Fri, Jul 26, 2019 at 11:04 AM vignesh C wrote: > > > > Hi, > > > > I have noticed in some cases the warning messages appear twice, one such > instance is given below: > > postgres=# begin; > > BEGIN > > postgres=# prepare transaction 't1';

RE: [Patch] PQconnectPoll() is blocked if target_session_attrs is read-write

2019-07-25 Thread Matsumura, Ryo
Tsunakawa-san Thank you for your comment. I understand the sense. I don't require an explicit rule. Regards Ryo Matsumura

Re: POC: Cleaning up orphaned files using undo logs

2019-07-25 Thread Amit Kapila
On Tue, Jul 23, 2019 at 8:12 PM Amit Khandekar wrote: > > On Tue, 23 Jul 2019 at 08:48, Amit Kapila wrote: > > > > > > > -- > > > > > > + if (!InsertRequestIntoErrorUndoQueue(urinfo)) > > > I was thinking what happens if for some reason > > > InsertRequestIntoErrorUndoQueue() itself e