Re: Postgres 11: Table Partitioning and Primary Keys

2019-07-10 Thread Michael Paquier
On Tue, Jul 09, 2019 at 06:59:59PM -0400, Tom Lane wrote: > Alvaro Herrera writes: >> >> >>Unique constraints on partitioned tables (as well as primary keys) >>must constrain all the partition key columns. This limitation exists >>because PostgreSQL can only e

make -jn fails by requiring not-yet-generated include files.

2019-07-10 Thread Kyotaro Horiguchi
Sorry, the subject of the previous mail was wrong. I resend it with the correct subject. I found the following make's behavior is annoying (at dab81b9953). make distclean ./configure make all make -j4 clean all relpath.c:21:10: fatal error: catalog/pg_tablespace_d.h: No such file or

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Amit Kapila
On Wed, Jul 10, 2019 at 10:12 AM Michael Paquier wrote: > > On Tue, Jul 09, 2019 at 11:54:29AM -0700, Melanie Plageman wrote: > > It might be worth post-processing results files to ignore row ordering > > in some cases to allow for easier comparison. Has this been proposed > > in the past? > > Not

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-10 Thread Michael Paquier
On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote: > Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could > use some common code, but for Windows build, I'm not sure where are those > window build files. Does anyone know about that? Thanks. The VS scripts are located

Re: Contribution to Perldoc for TestLib module in Postgres

2019-07-10 Thread Michael Paquier
On Tue, Jul 09, 2019 at 03:16:01PM +0200, Daniel Gustafsson wrote: > The v2 patch is somewhat confused as it has Windows carriage returns rather > than newlines, so it replaces the entire file making the diff hard to read. > It > also includes a copy of TestLib and the v1 patch and has a lot of w

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Amit Langote
On Tue, Jul 9, 2019 at 6:21 AM Tom Lane wrote: > > Amit Langote writes: > > [ parse-plan-memcxt_v2.patch ] > > I got around to looking at this finally. Thanks for the review. > I'm not at all happy with > the fact that it's added a plantree copy step to the only execution > path through exec_si

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: > It would be good if we can come up with something like that. It will > be helpful for zheap, where in some cases we get different row > ordering due to in-place updates. As of now, we try to add Order By > or do some extra magic to ge

Re: FETCH FIRST clause PERCENT option

2019-07-10 Thread Kyotaro Horiguchi
Hello. At Wed, 10 Jul 2019 15:02:57 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190710.150257.260806103.horikyota@gmail.com> > It is seen by a simpler test. > > create table t as select a from generate_series(0, 9) a; > analyze t; > explain analyze select * from t order

Re: pg_receivewal documentation

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 12:22:02AM +0200, Laurenz Albe wrote: > Works for me. > > Marked as "ready for committer". Hmm. synchronous_commit is user-settable, which means that it is possible to enforce a value in the connection string doing the connection. Isn't that something we had better enfor

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Antonin Houska
Joe Conway wrote: > On 7/8/19 6:04 PM, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > >> Uh, well, renaming the user was a big problem, but that is the only case > >> I can think of. I don't see that as an issue for block or WAL sequence > >> numbers. If we want to use a d

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Kyotaro Horiguchi
Hi, At Wed, 10 Jul 2019 16:35:18 +0900, Amit Langote wrote in > On Tue, Jul 9, 2019 at 6:21 AM Tom Lane wrote: > > > > Amit Langote writes: > > > [ parse-plan-memcxt_v2.patch ] > > > > I got around to looking at this finally. > > Thanks for the review. > > > I'm not at all happy with > > th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Antonin Houska
Tomas Vondra wrote: > On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote: > >On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote: > >> On 7/9/19 11:11 AM, Bruce Momjian wrote: > >> > Good point about nonce and IV. I wonder if running the nonce > >> > through the cipher with the

Re: pg_log_fatal vs pg_log_error

2019-07-10 Thread Antonin Houska
Peter Eisentraut wrote: > On 2019-06-17 14:19, Antonin Houska wrote: > > Can anyone please give me a hint (and possibly add some comments to the > > code) > > when pg_log_fatal() should be used in frontend code and when it's > > appropriate > > to call pg_log_error()? The current use does not s

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-07-10 Thread David Rowley
On Wed, 3 Jul 2019 at 19:35, Michael Paquier wrote: > This has been reverted as of f5db56f, still it seems to me that this > was moving in the right direction. I've pushed this again, this time with the cleanup code done in the right order. -- David Rowley http://www.2ndQuadr

pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Magnus Hagander
How is this intended to work? pg_checksums enumerate the files. What if there are files there from a different tableam? Isn't pg_checksums just going to badly fail then, since it assumes everything is heap? Also, do we allow AMs that don't support checksumming data? Do we have any checks for tabl

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/9/19 7:28 PM, Stephen Frost wrote: > Greetings, > > * Joe Conway (m...@joeconway.com) wrote: >> On 7/9/19 5:42 PM, Tomas Vondra wrote: >> > There are two basic ways to construct nonces - CSPRNG and sequences, and >> > then a combination of both, i.e. one part is generated from a sequence >> >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/9/19 10:06 PM, Stephen Frost wrote: > Greetings, > > * Ryan Lambert (r...@rustprooflabs.com) wrote: >> > What I think Tomas is getting at here is that we don't write a page only >> > once. >> >> > A nonce of tableoid+pagenum will only be unique the first time we write >> > out that page. Se

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 2:40 AM, Masahiko Sawada wrote: > On Tue, Jul 9, 2019 at 10:16 PM Joe Conway wrote: >> >> On 7/9/19 8:39 AM, Ryan Lambert wrote: >> > Hi Thomas, >> > >> >> CBC mode does require >> >> random nonces, other modes may be fine with even sequences as long as >> >> the values are not reused.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 2:38 AM, Masahiko Sawada wrote: > On Tue, Jul 9, 2019 at 9:01 PM Joe Conway wrote: >> >> On 7/9/19 6:07 AM, Peter Eisentraut wrote: >> > On 2019-07-08 18:09, Joe Conway wrote: >> >> In my mind, and in practice to a >> >> large extent, a postgres tablespace == a unique mount point. >> >

Re: Minimal logical decoding on standbys

2019-07-10 Thread Amit Khandekar
On Wed, 10 Jul 2019 at 08:44, Andres Freund wrote: > > Hi, > > Thanks for the new version! Looks like we're making progress towards > something committable here. > > I think it'd be good to split the patch into a few pieces. I'd maybe do > that like: > 1) WAL format changes (plus required other ch

Re: Comment typo in tableam.h

2019-07-10 Thread Amit Kapila
On Mon, Jul 8, 2019 at 10:21 PM Ashwin Agrawal wrote: > > > On Sat, Jul 6, 2019 at 12:05 AM Amit Kapila wrote: >> >> On Tue, Jul 2, 2019 at 1:00 AM Ashwin Agrawal wrote: >> > Please find attached v2 of patch 1 without objectionable comment change. >> > v1 of patch 2 attaching here just for conv

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 4:47 AM, Antonin Houska wrote: > Tomas Vondra wrote: >> I don't think that works, because that'd mean we're encrypting the same >> page with the same nonce over and over, which means reusing the reuse >> (even if you hash/encrypt it). Or did I miss something? > > I found out that it's

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 4:24 AM, Antonin Houska wrote: > Joe Conway wrote: > >> On 7/8/19 6:04 PM, Stephen Frost wrote: >> > * Bruce Momjian (br...@momjian.us) wrote: >> >> Uh, well, renaming the user was a big problem, but that is the only case >> >> I can think of. I don't see that as an issue for block or

Re: pg_receivewal documentation

2019-07-10 Thread Jesper Pedersen
Hi, On 7/9/19 6:22 PM, Laurenz Albe wrote: Works for me. Marked as "ready for committer". Thank you ! Best regards, Jesper

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Joe Conway (m...@joeconway.com) wrote: > On 7/9/19 7:28 PM, Stephen Frost wrote: > > * Joe Conway (m...@joeconway.com) wrote: > >> On 7/9/19 5:42 PM, Tomas Vondra wrote: > >> > There are two basic ways to construct nonces - CSPRNG and sequences, and > >> > then a combination of both,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 2:45 AM, Masahiko Sawada wrote: > On Wed, Jul 10, 2019 at 11:06 AM Stephen Frost wrote: >> >> Greetings, >> >> * Ryan Lambert (r...@rustprooflabs.com) wrote: >> > > What I think Tomas is getting at here is that we don't write a page only >> > > once. >> > >> > > A nonce of tableoid+page

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 8:34 AM, Stephen Frost wrote: > Greetings, > > * Joe Conway (m...@joeconway.com) wrote: >> On 7/9/19 7:28 PM, Stephen Frost wrote: >> > * Joe Conway (m...@joeconway.com) wrote: >> >> On 7/9/19 5:42 PM, Tomas Vondra wrote: >> >> > There are two basic ways to construct nonces - CSPRNG and

Re: extension patch of CREATE OR REPLACE TRIGGER

2019-07-10 Thread Surafel Temesgen
Hi Takamichi Osumi, On Tue, Jul 9, 2019 > I've rebased the previous patch to be applied > I don't test your patch fully yet but here are same comment. There are same white space issue like here - bool is_internal) + bool is_internal, + Oid existing_constraint_oid) in a few place + // trigoid

Re: pg_receivewal documentation

2019-07-10 Thread Jesper Pedersen
Hi, On 7/10/19 4:04 AM, Michael Paquier wrote: On Wed, Jul 10, 2019 at 12:22:02AM +0200, Laurenz Albe wrote: Works for me. Marked as "ready for committer". Hmm. synchronous_commit is user-settable, which means that it is possible to enforce a value in the connection string doing the connect

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 11:42:34AM +0200, Magnus Hagander wrote: > pg_checksums enumerate the files. What if there are files there from a > different tableam? Isn't pg_checksums just going to badly fail then, since > it assumes everything is heap? > > Also, do we allow AMs that don't support check

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Melanie Plageman
On Wed, Jul 10, 2019 at 12:40 AM Michael Paquier wrote: > On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: > > It would be good if we can come up with something like that. It will > > be helpful for zheap, where in some cases we get different row > > ordering due to in-place updates.

Re: Index Skip Scan

2019-07-10 Thread Jesper Pedersen
Hi, On 7/9/19 10:14 PM, Thomas Munro wrote: Thomas, do you have any ideas for this ? I can see that MySQL did the functionality in two change sets (base and function support), but like you said we shouldn't paint ourselves into a corner. I think amskip() could be augmented by later patches to

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Robert Haas
On Tue, Jul 9, 2019 at 2:32 PM Dave Cramer wrote: > So did this die from lack of interest? > > I have proposed in another thread adding more GUC REPORT variables, but I see > this as a much better way. > > I'm willing to code the patch if we can get some buy in here ? It seemed like most people

Re: progress report for ANALYZE

2019-07-10 Thread Robert Haas
On Tue, Jul 9, 2019 at 6:12 PM Alvaro Herrera wrote: > Hmm, ok. In CREATE INDEX, we use the block counters multiple times. Why do we do that? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: progress report for ANALYZE

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Robert Haas wrote: > On Tue, Jul 9, 2019 at 6:12 PM Alvaro Herrera > wrote: > > Hmm, ok. In CREATE INDEX, we use the block counters multiple times. > > Why do we do that? Because we scan the table first, then the index, then the table again (last two for the validation phase o

Re: global / super barriers (for checksums)

2019-07-10 Thread Magnus Hagander
On Tue, Oct 30, 2018 at 6:16 AM Andres Freund wrote: > Hi, > > Magnus cornered me at pgconf.eu and asked me whether I could prototype > the "barriers" I'd been talking about in the online checksumming thread. > > The problem there was to make sure that all processes, backends and > auxiliary proc

Re: Contribution to Perldoc for TestLib module in Postgres

2019-07-10 Thread Alvaro Herrera
On 2019-Apr-11, Iwata, Aya wrote: > In the above document, why not write a description after the function name? > I think it is better to write the function name first and then the > description. > In your code; > #Checks if all the tests passed or not > all_tests_passing() > > In my suggesti

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Tom Lane
Michael Paquier writes: > On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: >> It would be good if we can come up with something like that. It will >> be helpful for zheap, where in some cases we get different row >> ordering due to in-place updates. As of now, we try to add Order By

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Dave Cramer
On Wed, 10 Jul 2019 at 09:11, Robert Haas wrote: > On Tue, Jul 9, 2019 at 2:32 PM Dave Cramer wrote: > > So did this die from lack of interest? > > > > I have proposed in another thread adding more GUC REPORT variables, but > I see this as a much better way. > > > > I'm willing to code the patch

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Tue, Jul 09, 2019 at 10:06:33PM -0400, Stephen Frost wrote: Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > What I think Tomas is getting at here is that we don't write a page only > once. Yes, that's what I meant. > A nonce of tableoid+pagenum will only be unique the first

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-09, Julien Rouhaud wrote: > I finished to do a better refactoring, and ended up with this API in > scripts_parallel: Looking good! I'm not sure about the "Consume" word in ConsumeIdleSlot; maybe "Reserve"? "Obtain"? "Get"? Code commentary: I think the comment that sits atop the func

Re: [proposal] de-TOAST'ing using a iterator

2019-07-10 Thread Binguo Bao
Hi Thomas, I've fixed the warnings. Thomas Munro 于2019年7月5日周五 下午12:21写道: > On Thu, Jun 20, 2019 at 1:51 AM Binguo Bao wrote: > > Hi hackers! > > This proposal aims to provide the ability to de-TOAST a fully TOAST'd > and compressed field using an iterator and then update the appropriate > parts

Re: pg_receivewal documentation

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-09, Jesper Pedersen wrote: > + > +Note that while WAL will be flushed with this setting, > +it will never be applied, so linkend="guc-synchronous-commit"/> must > +not be set to remote_apply if > pg_receivewal > +is the only synchronous standby.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 03:38:54PM +0900, Masahiko Sawada wrote: On Tue, Jul 9, 2019 at 9:01 PM Joe Conway wrote: On 7/9/19 6:07 AM, Peter Eisentraut wrote: > On 2019-07-08 18:09, Joe Conway wrote: >> In my mind, and in practice to a >> large extent, a postgres tablespace == a unique mount poi

Re: Index Skip Scan

2019-07-10 Thread Floris Van Nee
> Here is finally a new version of the patch, where all the mentioned issues > seems to be fixed and the corresponding new tests should keep it like that > (I've skipped all the pubs at PostgresLondon for that). Thanks for the new patch! Really appreciate the work you're putting into it. I ver

Re: Optimize partial TOAST decompression

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 01:35:25PM +0800, Binguo Bao wrote: Tomas Vondra 于2019年7月10日周三 上午5:12写道: On Sat, Jul 06, 2019 at 05:23:37PM +0200, Tomas Vondra wrote: >On Sat, Jul 06, 2019 at 02:27:56AM +0800, Binguo Bao wrote: >>Hi, Tomas! >>Thanks for your testing and the suggestion. >> >>That's qui

Re: Index Skip Scan

2019-07-10 Thread Dmitry Dolgov
> On Tue, Jul 2, 2019 at 2:27 PM David Rowley > wrote: > > The more I think about these UniqueKeys, the more I think they need to > be a separate concept to PathKeys. For example, UniqueKeys: { x, y } > should be equivalent to { y, x }, but with PathKeys, that's not the > case, since the order of

Re: Index Skip Scan

2019-07-10 Thread Floris Van Nee
> Thanks for testing! Could you provide a test case to show what exactly is the > problem? create table a (a int, b int, c int); insert into a (select vs, ks, 10 from generate_series(1,5) vs, generate_series(1, 1) ks); create index on a (a,b); analyze a; set enable_indexskipscan=1; // setti

Re: range_agg

2019-07-10 Thread Paul Jungwirth
On 7/9/19 11:24 PM, David Fetter wrote: I seem to recall that the usual convention (at least in math) is to use intervals that are generally represented as open on the infinity side, but that might not fit how we do things. I think it does, unless I'm misunderstanding? Oh, I was just wonderin

Re: Index Skip Scan

2019-07-10 Thread Dmitry Dolgov
> On Wed, Jul 10, 2019 at 4:52 PM Floris Van Nee > wrote: > > > Thanks for testing! Could you provide a test case to show what exactly is > > the > > problem? > > create table a (a int, b int, c int); > insert into a (select vs, ks, 10 from generate_series(1,5) vs, > generate_series(1, 1) k

Re: Index Skip Scan

2019-07-10 Thread Floris Van Nee
> Thanks for testing! Could you provide a test case to show what exactly is the > problem? Note that in the case of a regular non-skip scan, this cursor backwards works because the Unique node on top does not support backwards scanning at all. Therefore, when creating the cursor, the actual pla

Re: Index Skip Scan

2019-07-10 Thread Dmitry Dolgov
> On Wed, Jul 10, 2019 at 5:00 PM Floris Van Nee > wrote: > > > > Thanks for testing! Could you provide a test case to show what exactly is > > the > > problem? > > Note that in the case of a regular non-skip scan, this cursor backwards works > because the Unique node on top does not support bac

Re: pg_receivewal documentation

2019-07-10 Thread Jesper Pedersen
Hi, On 7/10/19 10:24 AM, Alvaro Herrera wrote: +1 to document this caveat. How about Note that while WAL will be flushed with this setting, pg_receivewal never applies it, so must not be set to remote_apply if pg_receivewal is the only synchronous

Re: FETCH FIRST clause PERCENT option

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-08, Surafel Temesgen wrote: > Hi Thomas, > Thank you for informing me > > Hi Surafel, > > > > There's a call to adjust_limit_rows_costs() hiding under > > contrib/postgres_fdw, so this fails check-world. > > > > Fixed . I also include the review given by Ryan in attached patch What'

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Ashwin Agrawal
On Wed, Jul 10, 2019 at 6:46 AM Tom Lane wrote: > Michael Paquier writes: > > On Wed, Jul 10, 2019 at 12:51:28PM +0530, Amit Kapila wrote: > >> It would be good if we can come up with something like that. It will > >> be helpful for zheap, where in some cases we get different row > >> ordering

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Magnus Hagander
On Wed, Jul 10, 2019 at 3:05 PM Michael Paquier wrote: > On Wed, Jul 10, 2019 at 11:42:34AM +0200, Magnus Hagander wrote: > > pg_checksums enumerate the files. What if there are files there from a > > different tableam? Isn't pg_checksums just going to badly fail then, > since > > it assumes ever

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Andres Freund
Hi, On July 10, 2019 9:12:18 AM PDT, Magnus Hagander wrote: >On Wed, Jul 10, 2019 at 3:05 PM Michael Paquier >wrote: > >> On Wed, Jul 10, 2019 at 11:42:34AM +0200, Magnus Hagander wrote: >> > pg_checksums enumerate the files. What if there are files there >from a >> > different tableam? Isn't pg

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote: > Please see my other reply (and > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf > appendix C as pointed out by Ryan downthread). > > At least in my mind, I trust a published specification from the > nation

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 2:32 AM Amit Kapila wrote: > As of now, after we finish executing the rollback actions, the entry > from the hash table is removed. Now, at a later time (when queues are > full and we want to insert a new entry) when we access the queue entry > (to check whether we can rem

Re: [proposal] de-TOAST'ing using a iterator

2019-07-10 Thread Binguo Bao
This is the patch that fix warnings. Best Regards, Binguo Bao Binguo Bao 于2019年7月10日周三 下午10:18写道: > Hi Thomas, > I've fixed the warnings. > > Thomas Munro 于2019年7月5日周五 下午12:21写道: > >> On Thu, Jun 20, 2019 at 1:51 AM Binguo Bao wrote: >> > Hi hackers! >> > This proposal aims to provide the abi

buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Tom Lane
The current HEAD typedefs list available from https://buildfarm.postgresql.org/cgi-bin/typedefs.pl has the following interesting additions compared to where things were on July 1: 2 ECPGt_bytea connection_name in_addr pg_fprintf send_appname The "2"

Re: (select query)/relation as first class citizen

2019-07-10 Thread Dent John
Hi Roman, Pavel, I was interested in this post, as it’s a topic I’ve stumbled upon in the past. There are two topics at play here: 1. The ability to flexibly craft queries from procedural language functions 2. Support for pipelined access to SETOF/TABLEs from procedural language functions Po

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Bruce Momjian wrote: > * Using the LSN as part of the nonce fixes both problems, and has a >third benefit: > > * We don't need to decrypt/re-encrypt during CREATE DATABASE since >the page contents are the same in both places, and once one >database change

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Andres Freund
Hi, On 2019-07-10 12:57:08 -0400, Tom Lane wrote: > The current HEAD typedefs list available from > https://buildfarm.postgresql.org/cgi-bin/typedefs.pl > has the following interesting additions compared to where > things were on July 1: > > 2 > ECPGt_bytea > connection_name >

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Tom Lane
Andres Freund writes: > On 2019-07-10 12:57:08 -0400, Tom Lane wrote: >> Was there any change in calliphoridae's toolchain this month? > Hm, it has gotten gcc-9 installed recently, but calliphoridae isn't > using that. So it's probably not the compiler side. But I also see a > binutils upgrade: >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 01:04:47PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > * Using the LSN as part of the nonce fixes both problems, and has a > >third benefit: > > > > * We don't need to decrypt/re-encrypt during CREATE DATABASE since > >the pa

Re: block-level incremental backup

2019-07-10 Thread Anastasia Lubennikova
23.04.2019 14:08, Anastasia Lubennikova wrote: I'm volunteering to write a draft patch or, more likely, set of patches, which will allow us to discuss the subject in more detail. And to do that I wish we agree on the API and data format (at least broadly). Looking forward to hearing your though

Re: doc: minor update for description of "pg_roles" view

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:35:56PM +0900, Ian Barwick wrote: > Hi > > Here: > > https://www.postgresql.org/docs/12/view-pg-roles.html > > we state: > > "This view explicitly exposes the OID column of the underlying table, >since that is needed to do joins to other catalogs." > > I thin

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Ryan Lambert
> what is it that gets stored in the page for > decryption use, the nonce or the IV derived from it? I believe storing the IV is preferable and still secure per [1]: "The IV need not be secret" Beyond needing the database oid, if every decrypt function has to regenerate the IV from the nonce tha

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > > what is it that gets stored in the page for > > decryption use, the nonce or the IV derived from it? > > I believe storing the IV is preferable and still secure per [1]: "The IV > need not be secret" > > Beyond needing the database oi

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > what is it that gets stored in the page for > decryption use, the nonce or the IV derived from it? > > > I believe storing the IV is preferable and still secure per [1]: "The IV need > not be secret" > > Beyond needing th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > > > what is it that gets stored in the page for > > decryption use, the nonce or the IV derived from it? > > > > > > I believe storing the IV is preferable and still

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Tom Lane
Amit Langote writes: > Attached updated patch. Thanks again. Pushed with a bit of further cleanup --- most notably, the way you had execute_sql_string(), it was still leaking any cruft ProcessUtility might generate. We can fix that by running ProcessUtility in the per-statement context too. I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:44:30PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > > > > > what is it that gets stored in the page for > > > decryption use, the nonce or the IV der

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Ryan Lambert
> I didn't either, except it was referenced above as "forward hash". I > don't know why that was suggested, which is why I listed it as an > option/suggestion. My bad, sorry for the confusion! I meant to say "cipher" not "hash". I was (trying to) refer to the method of generating unpredictable

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Ryan Lambert (r...@rustprooflabs.com) wrote: > > I didn't either, except it was referenced above as "forward hash". I > > don't know why that was suggested, which is why I listed it as an > > option/suggestion. > > My bad, sorry for the confusion! I meant to say "cipher" not "hash"

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Robert Haas
On Tue, Jul 9, 2019 at 6:28 AM Dilip Kumar wrote: > PFA, updated patch version which includes > - One defect fix in undo interface related to undo page compression > for handling persistence level > - Implemented pending TODO optimization in undo page compression. > - One defect fix in undo proces

Re: pg_receivewal documentation

2019-07-10 Thread Laurenz Albe
On Wed, 2019-07-10 at 17:04 +0900, Michael Paquier wrote: > Hmm. synchronous_commit is user-settable, which means that it is > possible to enforce a value in the connection string doing the > connection. Isn't that something we had better enforce directly in > the tool? In this case what could b

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-10 Thread Julien Rouhaud
Hi Alvaro, Thanks a lot for the review On Wed, Jul 10, 2019 at 4:15 PM Alvaro Herrera wrote: > > On 2019-Jul-09, Julien Rouhaud wrote: > > > I finished to do a better refactoring, and ended up with this API in > > scripts_parallel: > > Looking good! Thanks! > I'm not sure about the "Consume"

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:57:54PM -0400, Stephen Frost wrote: > Greetings, > > * Ryan Lambert (r...@rustprooflabs.com) wrote: > > > I didn't either, except it was referenced above as "forward hash". I > > > don't know why that was suggested, which is why I listed it as an > > > option/suggestion

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Bruce Momjian wrote: > Good, so I think we all now agree we have to put the nonce > (pg_class.oid, LSN, page-number) though the cipher using the secret. Actually, why do you need the page number in the nonce? The LSN already distinguishes pages -- you can't have two pages with th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > Good, so I think we all now agree we have to put the nonce > > (pg_class.oid, LSN, page-number) though the cipher using the secret. > > Actually, why do you need the page number in the nonc

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 12:36 PM Robert Haas wrote: > Broadly, you are correct to point out that you need to avoid chasing > stale pointers, and there are a bunch of ways to accomplish that: > approach #1 avoids using real pointers, and the rest just make sure > that any stale pointers don't stick

Re: coypu: "FATAL: sorry, too many clients already"

2019-07-10 Thread Rémi Zara
> Le 10 juil. 2019 à 04:09, Tom Lane a écrit : > > I don't see a really good reason to be using force_parallel_mode on > such a low-end box, and would recommend taking that out in any case. > If the box's SysV IPC limits can't be raised, it might be a good idea > to restrict the maximum test p

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 03:53:55PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > Good, so I think we all now agree we have to put the nonce > > (pg_class.oid, LSN, page-number) though the cipher using the secret. > > Actually, why do you need the page number in the non

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Bruce Momjian wrote: > Uh, what if a transaction modifies page 0 and page 1 of the same table > --- don't those pages have the same LSN. No, because WAL being a physical change log, each page gets its own WAL record with its own LSN. -- Álvaro Herrerahttps://www.

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 9:59 AM Dave Cramer wrote: > I'm still a bit conflicted about what to do with search_path as I do believe > this is potentially a security issue. > It may be that we always want to report that and possibly back patch it. I don't see that as a feasible option unless we mak

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-07-10 Thread Tom Lane
Robert Haas writes: > On Wed, Jul 10, 2019 at 9:59 AM Dave Cramer wrote: >> I'm still a bit conflicted about what to do with search_path as I do believe >> this is potentially a security issue. >> It may be that we always want to report that and possibly back patch it. > I don't see that as a f

[sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Andreas Seltenreich
Hi, running sqlsmith on the regression database of REL_12_STABLE at ff597b656f yielded a crash in mcv_get_match_bitmap. I can reproduce it with the following query on the regression database: select filler1 from mcv_lists where a is not null and (select 42) <= c; Backtrace below. regards,

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Andrew Dunstan
On 7/10/19 1:34 PM, Andres Freund wrote: > > Hm, it has gotten gcc-9 installed recently, but calliphoridae isn't > using that. So it's probably not the compiler side. But I also see a > binutils upgrade: > > 2019-07-08 06:22:48 upgrade binutils-multiarch:amd64 2.31.1-16 > 2.32.51.20190707-1 > >

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Andreas Seltenreich writes: > running sqlsmith on the regression database of REL_12_STABLE at > ff597b656f yielded a crash in mcv_get_match_bitmap. I can reproduce it > with the following query on the regression database: > select filler1 from mcv_lists where a is not null and (select 42) <=

Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Sehrope Sarkuni
Hi, While working on adding a new log_destination I noticed that the syslogger piping would need to be updated. At the moment both ends only handle stderr/csvlog as the pipe message header has a char "is_last" that is either t/f (stderr last, stderr partial) or T/F (csvlog last, csvlog partial). C

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 04:57:54PM -0400, Tom Lane wrote: Andreas Seltenreich writes: running sqlsmith on the regression database of REL_12_STABLE at ff597b656f yielded a crash in mcv_get_match_bitmap. I can reproduce it with the following query on the regression database: select filler1 f

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Tomas Vondra writes: > BTW which other thread? I don't see any other threads mentioning this > function. https://www.postgresql.org/message-id/flat/CA%2Bu7OA65%2BjEFb_TyV5g%2BKq%2BonyJ2skMOPzgTgFH%2BqgLwszRqvw%40mail.gmail.com regards, tom lane

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Joe Conway
On 7/10/19 3:53 PM, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > >> Good, so I think we all now agree we have to put the nonce >> (pg_class.oid, LSN, page-number) though the cipher using the secret. (been traveling -- just trying to get caught up on this thread) > Actually, why

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Tomas Vondra writes: > Yeah, that's a bug. Will fix (not sure how yet). You could do worse than replace this: ok = (NumRelids(clause) == 1) && (is_pseudo_constant_clause(lsecond(expr->args)) || (varonleft = false, is_pseudo_constant_clause(linitial(

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 05:45:24PM -0400, Tom Lane wrote: Tomas Vondra writes: Yeah, that's a bug. Will fix (not sure how yet). You could do worse than replace this: ok = (NumRelids(clause) == 1) && (is_pseudo_constant_clause(lsecond(expr->args)) || (varonleft =

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: On 2019-Jul-10, Bruce Momjian wrote: Uh, what if a transaction modifies page 0 and page 1 of the same table --- don't those pages have the same LSN. No, because WAL being a physical change log, each page gets its own WAL record w

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: > >On 2019-Jul-10, Bruce Momjian wrote: > > > >>Uh, what if a transaction modifies page 0 and page 1 of the same table > >>--- don't those pages have the same LSN. > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Tomas Vondra
On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: >On 2019-Jul-10, Bruce Momjian wrote: > >>Uh, what if a transaction modifies page 0 and page 1 of the sa

  1   2   >