Re: Tid scan improvements

2019-07-10 Thread David Rowley
On Sun, 7 Jul 2019 at 15:32, Edmund Horner wrote: > I'm not really sure how to proceed. I started with a fairly pragmatic > solution to "WHERE ctid > ? AND ctid < ?" for tables, and then tableam > came along. > > The options I see are: > > A. Continue to target only heapam tables, making the bar

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Joe Conway wrote: > On 7/10/19 3:53 PM, Alvaro Herrera wrote: > > (I do think you can have multiple writes of the same page with > > different LSNs, if you change hint bits and don't write WAL about it, > > Do you mean "multiple writes of the same page without..."? Right, "twice

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Alvaro Herrera
Hi Sehrope, On 2019-Jul-10, Sehrope Sarkuni wrote: > While working on adding a new log_destination I noticed that the > syslogger piping would need to be updated. At the moment both ends > only handle stderr/csvlog as the pipe message header has a char > "is_last" that is either t/f (stderr last,

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Tom Lane
Oh ... while we're piling on here, it just sunk into me that mcv_get_match_bitmap is deciding what the semantics of an operator are by seeing what it's using for a selectivity estimator. That is just absolutely, completely wrong. For starters, it means that the whole mechanism fails for any operat

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Tom Lane
Alvaro Herrera writes: > Maybe we can use something like a shared memory queue, working in a > similar way to wal_buffers -- where backends send over the shm queue to > syslogger, and syslogger writes in order to the actual log file. No way that's going to be acceptable for postmaster output.

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Alvaro Herrera
On 2019-Jul-10, Tom Lane wrote: > Alvaro Herrera writes: > > Maybe we can use something like a shared memory queue, working in a > > similar way to wal_buffers -- where backends send over the shm queue to > > syslogger, and syslogger writes in order to the actual log file. > > No way that's goin

Re: Refactoring syslogger piping to simplify adding new log destinations

2019-07-10 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jul-10, Tom Lane wrote: >> No way that's going to be acceptable for postmaster output. > Well, we can use both mechanisms simultaneously. Postmaster doesn't emit > all that much output anyway, so I don't think that's a concern. And > actually, we still need the p

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Ryan Lambert
> As I understand, the reason we > want to avoid using the same IV for too many pages is to dodge a > cryptanalysis attack, which requires a large amount of data encrypted > with the same key/IV in order to be effective. But if we have two > copies of the same page encrypted with the same key/IV,

Re: accounting for memory used for BufFile during hash joins

2019-07-10 Thread Melanie Plageman
Okay, so, while I do have specific, actual code review/commitfest-y feedback for the patch in this thread registered for this commitfest, I wanted to defer that for a later email and use this one to cover off on a few higher level issues. 1) How this patch's approach fits into the wider set of pro

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:11:41AM -0700, Ashwin Agrawal wrote: > Will post patch for the tool, once I get in little decent shape. That would be nice! We may be able to get something into v13 this way then. -- Michael signature.asc Description: PGP signature

Re: Adversarial case for "many duplicates" nbtree split strategy in v12

2019-07-10 Thread Peter Geoghegan
On Tue, Jul 2, 2019 at 3:51 PM Peter Geoghegan wrote: > I've already written a rough patch that fixes the issue by taking this > second view of the problem. The patch makes nbtsplitloc.c more > skeptical about finishing with the "many duplicates" strategy, > avoiding the problem -- it can just fal

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 06:28:42PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Joe Conway wrote: > > > On 7/10/19 3:53 PM, Alvaro Herrera wrote: > > > > (I do think you can have multiple writes of the same page with > > > different LSNs, if you change hint bits and don't write WAL about it, >

Re: doc: minor update for description of "pg_roles" view

2019-07-10 Thread Ian Barwick
On 7/11/19 3:24 AM, Bruce Momjian wrote: On Wed, Jul 10, 2019 at 02:35:56PM +0900, Ian Barwick wrote: Hi Here: https://www.postgresql.org/docs/12/view-pg-roles.html we state: "This view explicitly exposes the OID column of the underlying table, since that is needed to do joins to o

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Thu, Jul 11, 2019 at 12:18:47AM +0200, Tomas Vondra wrote: > On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote: > > Greetings, > > > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > > > On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote: > > > >On 2019-Jul-10, Br

Re: buildfarm's typedefs list has gone completely nutso

2019-07-10 Thread Andres Freund
Hi, On 2019-07-10 16:40:20 -0400, Andrew Dunstan wrote: > On 7/10/19 1:34 PM, Andres Freund wrote: > > > > Hm, it has gotten gcc-9 installed recently, but calliphoridae isn't > > using that. So it's probably not the compiler side. But I also see a > > binutils upgrade: > > > > 2019-07-08 06:22:48

Re: pg_checksums (or checksums in general) vs tableam

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:19:03AM -0700, Andres Freund wrote: > On July 10, 2019 9:12:18 AM PDT, Magnus Hagander wrote: >> That would be fine, if we actually knew. Should we (or have we already?) >> defined a rule that they are not allowed to use the same naming standard >> unless they have the s

Re: using explicit_bzero

2019-07-10 Thread Thomas Munro
On Sun, Jul 7, 2019 at 1:11 AM Peter Eisentraut wrote: > I see. My premise, which should perhaps be explained in a comment at > least, is that on an operating system that does not provide > explicit_bzero() (or an obvious alternative), we don't care about > addressing this particular security con

Re: [sqlsmith] Crash in mcv_get_match_bitmap

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 11:26:09PM +0200, Tomas Vondra wrote: > Yeah, that's a bug. Will fix (not sure how yet). Please note that I have added an open item for it. -- Michael signature.asc Description: PGP signature

Re: using explicit_bzero

2019-07-10 Thread Michael Paquier
On Mon, Jun 24, 2019 at 02:08:50PM +0900, Michael Paquier wrote: > CreateRole() and AlterRole() can manipulate a password in plain format > in memory. The cleanup could be done just after calling > encrypt_password() in user.c. > > Could it be possible to add the new flag in pg_config.h.win32? W

Re: progress report for ANALYZE

2019-07-10 Thread Robert Haas
On Wed, Jul 10, 2019 at 9:26 AM Alvaro Herrera wrote: > On 2019-Jul-10, Robert Haas wrote: > > On Tue, Jul 9, 2019 at 6:12 PM Alvaro Herrera > > wrote: > > > Hmm, ok. In CREATE INDEX, we use the block counters multiple times. > > > > Why do we do that? > > Because we scan the table first, then

Re: Copy data to DSA area

2019-07-10 Thread Thomas Munro
On Wed, Jul 10, 2019 at 6:03 PM Thomas Munro wrote: > Hmm. I wonder if we should just make ShmContextFree() do nothing! And > make ShmContextAlloc() allocate (say) 8KB chunks (or larger if needed > for larger allocation) and then hand out small pieces from the > 'current' chunk as needed. Then t

Re: Implementing Incremental View Maintenance

2019-07-10 Thread Tatsuo Ishii
> I am quite interested to learn how IVM interacts with SERIALIZABLE. Just for a fun, I have added: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE; right after every BEGIN; in incremental_matview.sql in regression test and it seems it works. > A couple of superficial review comments: > > +

Re: Index Skip Scan

2019-07-10 Thread Thomas Munro
On Thu, Jul 11, 2019 at 2:40 AM Floris Van Nee wrote: > I verified that the backwards index scan is indeed functioning now. However, > I'm afraid it's not that simple, as I think the cursor case is broken now. I > think having just the 'scan direction' in the btree code is not enough to get > t

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Dilip Kumar
On Thu, Jul 11, 2019 at 12:38 AM Robert Haas wrote: > > On Tue, Jul 9, 2019 at 6:28 AM Dilip Kumar wrote: > > PFA, updated patch version which includes > > - One defect fix in undo interface related to undo page compression > > for handling persistence level > > - Implemented pending TODO optimiz

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:44:14PM +0200, Julien Rouhaud wrote: > On Wed, Jul 10, 2019 at 4:15 PM Alvaro Herrera > wrote: >> Looking good! > > Thanks! Confirmed. The last set is much easier to go through. >> I'm not sure about the "Consume" word in ConsumeIdleSlot; >> maybe "Reserve"? "Obtai

Re: Implementing Incremental View Maintenance

2019-07-10 Thread Yugo Nagata
Hi Thomas, Thank you for your review and discussion on this patch! > > 2019年7月8日(月) 15:32 Thomas Munro : > > > > > On Fri, Jun 28, 2019 at 10:56 PM Yugo Nagata wrote: > > > > Attached is a WIP patch of IVM which supports some aggregate functions. > > > > > > Hi Nagata-san and Hoshiai-san, > > >

Re: POC: Cleaning up orphaned files using undo logs

2019-07-10 Thread Amit Kapila
On Wed, Jul 10, 2019 at 10:06 PM Robert Haas wrote: > > On Wed, Jul 10, 2019 at 2:32 AM Amit Kapila wrote: > > As of now, after we finish executing the rollback actions, the entry > > from the hash table is removed. Now, at a later time (when queues are > > full and we want to insert a new entry

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-10 Thread Peter Geoghegan
On Sat, Jul 6, 2019 at 4:08 PM Peter Geoghegan wrote: > I took a closer look at this patch, and have some general thoughts on > its design, and specific feedback on the implementation. I have some high level concerns about how the patch might increase contention, which could make queries slower.

Re: pg_receivewal documentation

2019-07-10 Thread Michael Paquier
On Wed, Jul 10, 2019 at 09:12:46PM +0200, Laurenz Albe wrote: > Are you talking about the replication connection from pg_receivewal > to the PostgreSQL server? That wouldn't do anything, because it is > the setting of "synchronous_commit" for an independent client > connection that is the problem:

Re: warning to publication created and wal_level is not set to logical

2019-07-10 Thread Thomas Munro
On Wed, Jul 10, 2019 at 12:47 PM Tom Lane wrote: > 1. > > + errmsg("insufficient wal_level to publish logical > changes"), > > Might read better as "wal_level is insufficient to publish logical changes"? > > 2. > > + errhint("Set wal_level to logical be

Re: Index Skip Scan

2019-07-10 Thread David Rowley
On Thu, 11 Jul 2019 at 14:50, Thomas Munro wrote: > > On Thu, Jul 11, 2019 at 2:40 AM Floris Van Nee > wrote: > > I verified that the backwards index scan is indeed functioning now. > > However, I'm afraid it's not that simple, as I think the cursor case is > > broken now. I think having just

Re: FETCH FIRST clause PERCENT option

2019-07-10 Thread Surafel Temesgen
Hi Alvaro, On Wed, Jul 10, 2019 at 6:44 PM Alvaro Herrera wrote: > What's with the new tuplestore function for getting heap tuples? That > looks really odd. > Previously I create new TTSOpsMinimalTuple type slots for every tuple returned in order to fetch it from tuplestore because tuplestore

<    1   2