Re: LLVM compile failing in seawasp

2019-07-27 Thread Fabien COELHO
c.h defines a C Min macro conflicting with llvm new class llvm:ElementCount Min member Really? Well, we will hardly be the only code they broke with that. I think we can just wait for them to reconsider. FYI This is now on LLVM's release_90 branch, due out on August 28. Maybe we should co

Re: LLVM compile failing in seawasp

2019-07-27 Thread Thomas Munro
On Sat, Jul 27, 2019 at 7:06 PM Fabien COELHO wrote: > >>> c.h defines a C Min macro conflicting with llvm new class > >>> llvm:ElementCount Min member > >> > >> Really? Well, we will hardly be the only code they broke with that. > >> I think we can just wait for them to reconsider. > > > > FYI T

Re: LLVM compile failing in seawasp

2019-07-27 Thread Thomas Munro
On Sat, Jul 27, 2019 at 7:12 PM Thomas Munro wrote: > On Sat, Jul 27, 2019 at 7:06 PM Fabien COELHO wrote: > > Maybe we should consider doing an explicit bug report, but I would not bet > > that they are going to fold… or fixing the issue pg side, eg "pg_Min", > > less than 400 hundred instances,

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-27 Thread Julien Rouhaud
On Fri, Jul 26, 2019 at 9:41 AM Michael Paquier wrote: > > On Fri, Jul 26, 2019 at 09:36:32AM +0200, Julien Rouhaud wrote: > > I see that you iterate over the SimpleStringList after it's generated. > > Why not computing that while building it in get_parallel_object_list > > (and keep the provided

Re: Built-in connection pooler

2019-07-27 Thread Dave Cramer
Responses inline. I just picked up this thread so please bear with me. On Fri, 26 Jul 2019 at 16:24, Tomas Vondra wrote: > Hi Konstantin, > > I've started reviewing this patch and experimenting with it, so let me > share some initial thoughts. > > > 1) not handling session state (yet) > > I unde

Re: Built-in connection pooler

2019-07-27 Thread Thomas Munro
On Tue, Jul 16, 2019 at 2:04 AM Konstantin Knizhnik wrote: > I have committed patch which emulates epoll EPOLLET flag and so should > avoid busy loop with poll(). > I will be pleased if you can check it at FreeBSD box. I tried your v12 patch and it gets stuck in a busy loop during make check. Y

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-27 Thread Michael Paquier
On Sat, Jul 27, 2019 at 11:44:47AM +0200, Julien Rouhaud wrote: > That's probably still more intuitive than having the count coming from > either main() or from get_parallel_object_list() depending on the > process type, so I'm fine with that alternative. Maybe we could bite > the bullet and add a

Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Since we have three or four different NOTIFY improvement proposals floating around in the current CF, I got a bit distressed at the lack of test coverage for that functionality. While the code coverage report makes it look like commands/async.c isn't so badly covered, that's all coming from src/te

Re: fsync error handling in pg_receivewal, pg_recvlogical

2019-07-27 Thread Sehrope Sarkuni
Hi, Tried out this patch and it applies, compiles, and passes check-world. Also flipped things around in pg_recvlogical.c to exit-on-success to ensure it's actually being called and that worked too. Outside of a more complicated harness that simulates fsync errors not sure how else to test this fu

Re: fsync error handling in pg_receivewal, pg_recvlogical

2019-07-27 Thread Sehrope Sarkuni
While reviewing this patch I read through some of the other fsync callsites and noticed this typo (walkdir is in file_utils.c, not initdb.c) too: diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c index 315c74c745..9b79df2d7f 100644 --- a/src/backend/storage/file/fd.c +++ b

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 10:57:08PM -0400, Alvaro Herrera wrote: > On 2019-Jul-25, Bruce Momjian wrote: > > > On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > > > > Why are we encrypting the page header in the first place? It seems to > > > me that the encrypted area should cover

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 11:30:55PM -0400, Alvaro Herrera wrote: > On 2019-Jul-25, Alvaro Herrera wrote: > > > > Uh, there are no known attacks on AES with known plain-text, e.g., SSL > > > uses AES, so I think we are good with encrypting everything after the > > > first 16 bytes. > > > > Well, ma

Re: [PATCH] get rid of StdRdOptions, use individual binary reloptions representation for each relation kind instead

2019-07-27 Thread Tomas Vondra
Hi Nikolay, thanks for sending a new version of the patch. I've done a basic review today, so let me share some comments about the patch. Firstly, there's an important question why should we actually do this. At the beginning of this thread you mentioned memory usage - e.g. for indexes the redu

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 12:46:51 -0400, Tom Lane wrote: > So, if we'd like to have more thorough NOTIFY coverage without going > to that much work, what to do? I thought of a few alternatives: > > 1. Write a TAP test instead of using the old test frameworks, and > use regexps to check the expected out

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Andres Freund writes: > On 2019-07-27 12:46:51 -0400, Tom Lane wrote: >> I'm finding alternative #3 the most attractive, because we really >> want isolation-style testing for LISTEN/NOTIFY, and this solution >> doesn't require designing a psql feature that we'd need to get >> consensus on. > Perh

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
While I'm looking at isolationtester ... my eye was immediately drawn to this bit, because it claims to be dealing with NOTIFY messages --- though that's wrong, it's really blocking NOTICE messages: /* * Suppress NOTIFY messages, which otherwise pop into results at odd *

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-27 Thread Julien Rouhaud
On Sat, Jul 27, 2019 at 3:27 PM Michael Paquier wrote: > > On Sat, Jul 27, 2019 at 11:44:47AM +0200, Julien Rouhaud wrote: > > That's probably still more intuitive than having the count coming from > > either main() or from get_parallel_object_list() depending on the > > process type, so I'm fine

Adding column "mem_usage" to view pg_prepared_statements

2019-07-27 Thread Daniel Migowski
Hello, I just implemented a small change that adds another column "mem_usage" to the system view "pg_prepared_statements". It returns the memory usage total of CachedPlanSource.context, CachedPlanSource.query_content and if available CachedPlanSource.gplan.context. Looks like this: IKOffice_D

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 14:15:39 -0400, Tom Lane wrote: > So I think we should apply something like the attached, and if the > buildfarm shows any instability as a result, dealing with that by > taking out the RAISE NOTICE commands. +1 > diff --git a/src/test/isolation/expected/insert-conflict-speccon

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Sehrope Sarkuni
On Sat, Jul 27, 2019 at 1:32 PM Bruce Momjian wrote: > Uh, I listed the three options for the CRC and gave the benefits of > each: > > > https://www.postgresql.org/message-id/20190725200343.xo4dcjm5azrfn...@momjian.us > > Obviously I was not clear on the benefits. To quote: > > 1

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Joe Conway
On 7/27/19 3:02 PM, Sehrope Sarkuni wrote: > More generally, without a cryptographic MAC I don't think it's > possible to provide any meaningful malicious tamper detection. And > even that would have to be off-page to deal with page replay (which I > think is out of scope). > > [1]: https://en.wik

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 18:29:23 +, Daniel Migowski wrote: > I just implemented a small change that adds another column "mem_usage" > to the system view "pg_prepared_statements". It returns the memory > usage total of CachedPlanSource.context, > CachedPlanSource.query_content and if available > Cach

tap tests driving the database via psql

2019-07-27 Thread Andres Freund
Hi, The discussion in [1] again reminded me how much I dislike that we currently issue database queries in tap tests by forking psql and writing/reading from it's stdin/stdout. That's quite cumbersome to write, and adds a good number of additional failure scenarios to worry about. For a lot of ta

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Andres Freund writes: > On 2019-07-27 14:15:39 -0400, Tom Lane wrote: >> So I think we should apply something like the attached, and if the >> buildfarm shows any instability as a result, dealing with that by >> taking out the RAISE NOTICE commands. > +1 >> diff --git a/src/test/isolation/expect

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 15:39:44 -0400, Tom Lane wrote: > Andres Freund writes: > >> diff --git a/src/test/isolation/expected/insert-conflict-specconflict.out > >> b/src/test/isolation/expected/insert-conflict-specconflict.out > >> index 5726bdb..20cc421 100644 > >> --- a/src/test/isolation/expected/i

Re: tap tests driving the database via psql

2019-07-27 Thread David Fetter
On Sat, Jul 27, 2019 at 12:15:23PM -0700, Andres Freund wrote: > Hi, > > The discussion in [1] > again reminded me how much I dislike that we currently issue database > queries in tap tests by forking psql and writing/reading from it's > stdin/stdout. > > That's quite cumbersome to write, and add

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-27 Thread Bruce Momjian
On Sat, Jul 27, 2019 at 03:02:02PM -0400, Sehrope Sarkuni wrote: > On Sat, Jul 27, 2019 at 1:32 PM Bruce Momjian wrote: > > Uh, I listed the three options for the CRC and gave the benefits of > > each: > > > > > > https://www.postgresql.org/message-id/20190725200343.xo4dcjm5azrfn...@momji

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Peter Geoghegan
On Sat, Jul 27, 2019 at 12:39 PM Tom Lane wrote: > Unfortunately, I just found out that on a slow enough machine > (prairiedog's host) there *is* some variation in when that test's > notices come out. I am unsure whether that's to be expected or > whether there's something wrong there --- Peter,

Re: tap tests driving the database via psql

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 22:32:37 +0200, David Fetter wrote: > On Sat, Jul 27, 2019 at 12:15:23PM -0700, Andres Freund wrote: > > 4) We develop a fairly minimal pure perl database driver, that doesn't > >depend on DBI. Include it somewhere as part of the test code, instead > >of src/interfaces, s

Re: tap tests driving the database via psql

2019-07-27 Thread Andrew Dunstan
On 7/27/19 3:15 PM, Andres Freund wrote: > Hi, > > The discussion in [1] > again reminded me how much I dislike that we currently issue database > queries in tap tests by forking psql and writing/reading from it's > stdin/stdout. > > That's quite cumbersome to write, and adds a good number of add

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-07-27 Thread Tomas Vondra
Hi, I've started reviewing this patch, thinking that maybe I could get it committed as it's marked as RFC. In general I agree with having this fuature, but I think we need to rethink the GUC because the current approach is just confusing. The way the current patch works is that we have three GUC

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
I wrote: > Andres Freund writes: >> Perhaps we could just have isolationtester check to which >> isolationtester session the backend pid belongs? And then print the >> session name instead of the pid? That should be fairly easy, and would >> probably give us all we need? > Oh, that's a good idea

Re: tap tests driving the database via psql

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 17:48:39 -0400, Andrew Dunstan wrote: > On 7/27/19 3:15 PM, Andres Freund wrote: > > I'm not volunteering to do 4), my perl skills aren't great (if the test > > infra were python, otoh... I have the skeleton of a pure perl driver > > that I used for testing somewhere). But I am l

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Andres Freund writes: > On 2019-07-27 15:39:44 -0400, Tom Lane wrote: >> Unfortunately, I just found out that on a slow enough machine >> (prairiedog's host) there *is* some variation in when that test's >> notices come out. I am unsure whether that's to be expected or >> whether there's somethin

Re: tap tests driving the database via psql

2019-07-27 Thread Andrew Dunstan
On 7/27/19 6:37 PM, Andres Freund wrote: > Hi, > > On 2019-07-27 17:48:39 -0400, Andrew Dunstan wrote: >> On 7/27/19 3:15 PM, Andres Freund wrote: >>> I'm not volunteering to do 4), my perl skills aren't great (if the test >>> infra were python, otoh... I have the skeleton of a pure perl driver >

Re: tap tests driving the database via psql

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 18:57:58 -0400, Andrew Dunstan wrote: > Maybe I don't write much python but I can read it without too much > difficulty :-) > > > But you did say your skeleton was pure perl ... slip of the fingers? Ooops, yea. > > /me once more regrets that perl, not python, has been chosen

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 18:20:52 -0400, Tom Lane wrote: > diff --git a/src/test/isolation/isolationtester.c > b/src/test/isolation/isolationtester.c > index 6ab19b1..98e5bf2 100644 > --- a/src/test/isolation/isolationtester.c > +++ b/src/test/isolation/isolationtester.c > @@ -23,10 +23,12 @@ > > /*

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Andres Freund writes: > We could of course just send the pids in binary ;). No, not worth it > just to avoid a small redundant array ;) IIRC, we'd have to do htonl on them, so we'd still end up with two representations ... > Hm. I wonder if all that's happening with prairedog is that the notice

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
I wrote: > Andres Freund writes: >> Hm. I wonder if all that's happening with prairedog is that the notice >> is sent a bit later. I think that could e.g. conceivably happen because >> it TCP_NODELAY isn't supported on prariedog? Or just because the machine >> is very slow? > The notices (not not

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 19:27:17 -0400, Tom Lane wrote: > Andres Freund writes: > > We could of course just send the pids in binary ;). No, not worth it > > just to avoid a small redundant array ;) > > IIRC, we'd have to do htonl on them, so we'd still end up with > two representations ... Yea. Altho

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Andres Freund writes: > Polling for notices on the blocked connection before printing anything > ought to practically be reliable. Theoretically I think it still allows > for some reordering, e.g. because there was packet loss on one, but not > the other connection. As long as it's a local connec

Re: pg_upgrade fails with non-standard ACL

2019-07-27 Thread Bruce Momjian
On Thu, Jul 18, 2019 at 06:53:12PM +0300, Anastasia Lubennikova wrote: > pg_upgrade from 9.6 fails if old cluster had non-standard ACL > on pg_catalog functions that have changed between versions, > for example pg_stop_backup(boolean). > > Error: > > pg_restore: creating ACL "pg_catalog.FUNCTION

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Andres Freund
Hi, On 2019-07-27 20:02:13 -0400, Tom Lane wrote: > Andres Freund writes: > I'm slightly more worried about the case of more than one bufferful > of NOTICE messages: calling PQconsumeInput isn't entirely guaranteed to > absorb *all* available input. But for the cases we actually need to > deal w

Re: Testing LISTEN/NOTIFY more effectively

2019-07-27 Thread Tom Lane
Andres Freund writes: > I wonder if there aren't similar dangers around the notify handling. In > your patch we don't print them particularly eagerly. Doesn't that also > open us up to timing concerns? I think probably not, because of the backend-side restrictions on when notify messages will be

Re: pg_upgrade fails with non-standard ACL

2019-07-27 Thread Tom Lane
Bruce Momjian writes: > On Thu, Jul 18, 2019 at 06:53:12PM +0300, Anastasia Lubennikova wrote: >> pg_upgrade from 9.6 fails if old cluster had non-standard ACL >> on pg_catalog functions that have changed between versions, >> for example pg_stop_backup(boolean). > Uh, wouldn't this affect any def

minor fixes after pgindent prototype fixes

2019-07-27 Thread Andres Freund
Hi, I noticed that after commit 8255c7a5eeba8f1a38b7a431c04909bde4f5e67d Author: Tom Lane Date: 2019-05-22 13:04:48 -0400 Phase 2 pgindent run for v12. Switch to 2.1 version of pg_bsd_indent. This formats multiline function declarations "correctly", that is with addition

Re: minor fixes after pgindent prototype fixes

2019-07-27 Thread Tom Lane
Andres Freund writes: > a few prototypes look odd. It appears to be cases where previously the > odd indentation was put to some use, by indenting parameters less: > ... > but now that looks odd: > extern void DefineCustomBoolVariable( > const char *name, >

Fix typos and inconsistencies for HEAD (take 8)

2019-07-27 Thread Alexander Lakhin
Hello hackers, Please consider fixing the next set of typos and inconsistencies in the tree: 8.1. LABORT -> LIKE_ABORT 8.2. LagTrackerWriter -> LagTrackerWrite 8.3. lag_with_offset_and_default, * -> window_lag_with_offset_and_default, window_* (in windowfuncs.c) 8.4. language-name -> language_name

Re: Adding column "mem_usage" to view pg_prepared_statements

2019-07-27 Thread Daniel Migowski
Hello Andres, how do you want to generalize it? Are you thinking about a view solely for the display of the memory usage of different objects? Like functions or views (that also have a plan associated with it, when I think about it)? While being interesting I still believe monitoring the mem us