Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Antonin Houska
Bruce Momjian wrote: > On Thu, Aug 15, 2019 at 09:01:05PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Why would it not be simpler to have the cluster_passphrase_command run > > > whatever command-line program it wants? If you don't want to use a > > > shell co

Re: REL_12_STABLE crashing with assertion failure in ExtractReplicaIdentity

2019-08-16 Thread Tom Lane
Andres Freund writes: > On 2019-08-16 09:44:15 -0700, Hadi Moshayedi wrote: >> It seems that sometimes when DELETE cascades to referencing tables we fail >> to acquire locks on replica identity index. > I suspect this "always" has been broken, it's just that we previously > didn't have checks in

Re: Can't we give better table bloat stats easily?

2019-08-16 Thread Andres Freund
Hi, On 2019-08-16 20:39:21 -0400, Greg Stark wrote: > But isn't this all just silliiness these days? We could actually sum up the > space recorded in the fsm and get a much more trustworthy number in > milliseconds. You mean like pgstattuple_approx()? https://www.postgresql.org/docs/current/pgst

Can't we give better table bloat stats easily?

2019-08-16 Thread Greg Stark
Everywhere I've worked I've seen people struggle with table bloat. It's hard to even measure how much of it you have or where, let alone actually fix it. If you search online you'll find dozens of different queries estimating how much empty space are in your tables and indexes based on pg_stats st

Re: Unused header file inclusion

2019-08-16 Thread Andres Freund
Hi, On 2019-08-03 12:37:33 -0700, Andres Freund wrote: > Think the first three are pretty clearly a good idea, I'm a bit less > sanguine about the fourth: > Headers like utils/timestamp.h are often included just because we need a > TimestampTz type somewhere, or call GetCurrentTimestamp(). Approxi

Re: default_table_access_method is not in sample config file

2019-08-16 Thread Andres Freund
On 2019-08-13 15:03:13 +0900, Michael Paquier wrote: > On Fri, Aug 09, 2019 at 11:34:05AM +0300, Heikki Linnakangas wrote: > > On 11/04/2019 19:49, Andres Freund wrote: > >> Hm, I think we should rather add it to sample. That's an oversight, not > >> intentional. > > > > I just noticed that this i

Re: allocation limit for encoding conversion

2019-08-16 Thread Andres Freund
Hi, On 2019-08-16 17:31:49 -0400, Tom Lane wrote: > Alvaro Herrera writes: > > Somebody ran into issues when generating large XML output (upwards of > > 256 MB) and then sending via a connection with a different > > client_encoding. This occurs because we pessimistically allocate 4x as > > much

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Bruce Momjian
On Fri, Aug 16, 2019 at 07:47:37PM +0200, Antonin Houska wrote: > Bruce Momjian wrote: > > > I have seen no one present a clear description of how anything beyond > > all-cluster encryption would work or be secure. Wishing that were not > > the case doesn't change things. > > Since this email t

Re: allocation limit for encoding conversion

2019-08-16 Thread Tom Lane
Alvaro Herrera writes: > Somebody ran into issues when generating large XML output (upwards of > 256 MB) and then sending via a connection with a different > client_encoding. This occurs because we pessimistically allocate 4x as > much memory as the string needs, and we run into the 1GB palloc >

Re: pgsql: doc: Add some images

2019-08-16 Thread Alvaro Herrera
On 2019-Mar-27, Peter Eisentraut wrote: > doc: Add some images > > Add infrastructure for having images in the documentation, in SVG > format. Add two images to start with. See the included README file > for instructions. > Author: Jürgen Purtz > Author: Peter Eisentraut Now when I test Jür

Re: allocation limit for encoding conversion

2019-08-16 Thread Alvaro Herrera
On 2019-Aug-16, Alvaro Herrera wrote: > Somebody ran into issues when generating large XML output (upwards of > 256 MB) and then sending via a connection with a different > client_encoding. ref: https://postgr.es/m/43a889a1-45fb-1d60-31ae-21e607307...@gmail.com (pgsql-es-ayuda) -- Álvaro Herrer

allocation limit for encoding conversion

2019-08-16 Thread Alvaro Herrera
Somebody ran into issues when generating large XML output (upwards of 256 MB) and then sending via a connection with a different client_encoding. This occurs because we pessimistically allocate 4x as much memory as the string needs, and we run into the 1GB palloc limitation. ISTM we can do better

Re: Add "password_protocol" connection parameter to libpq

2019-08-16 Thread Jonathan S. Katz
On 8/15/19 9:28 PM, Stephen Frost wrote: > Greetings, > > * Jeff Davis (pg...@j-davis.com) wrote: >> On Wed, 2019-08-14 at 11:38 +0900, Michael Paquier wrote: >>> What I got in mind was a comma-separated list of authorized protocols >>> which can be specified as a connection parameter, which exten

Re: REL_12_STABLE crashing with assertion failure in ExtractReplicaIdentity

2019-08-16 Thread Andres Freund
Hi, On 2019-08-16 09:44:15 -0700, Hadi Moshayedi wrote: > It seems that sometimes when DELETE cascades to referencing tables we fail > to acquire locks on replica identity index. > > To reproduce, set wal_level to logical, and run 1.sql. > > I can look into this, but I thought first I should sen

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Antonin Houska
Bruce Momjian wrote: > I have seen no one present a clear description of how anything beyond > all-cluster encryption would work or be secure. Wishing that were not > the case doesn't change things. Since this email thread has grown a lot and is difficult to follow, it might help if we summariz

Re: Global temporary tables

2019-08-16 Thread Pavel Stehule
pá 16. 8. 2019 v 16:12 odesílatel Konstantin Knizhnik < k.knizh...@postgrespro.ru> napsal: > I did more investigations of performance of global temp tables with shared > buffers vs. vanilla (local) temp tables. > > 1. Combination of persistent and temporary tables in the same query. > > Preparatio

REL_12_STABLE crashing with assertion failure in ExtractReplicaIdentity

2019-08-16 Thread Hadi Moshayedi
It seems that sometimes when DELETE cascades to referencing tables we fail to acquire locks on replica identity index. To reproduce, set wal_level to logical, and run 1.sql. I can look into this, but I thought first I should send it here in case someone who is more familiar with these related fun

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-08-16 Thread Anastasia Lubennikova
13.08.2019 18:45, Anastasia Lubennikova wrote:   I also added a nearby FIXME comment to _bt_insertonpg_in_posting() -- I don't think think that the code for splitting a posting list in two is currently crash-safe. Good catch. It seems, that I need to rearrange the code. I'll send updated patch

Re: UNION ALL

2019-08-16 Thread Tom Lane
Mark Pasterkamp writes: > I am comparing two queries, q1 and q2 respectively. > Query q1 is the original query and q2 is an attempt to reduce the cost of > execution via leveraging the materialized view ci_t_15. > ... > Running explain analyze on both queries I get the following execution plans.

Re: block-level incremental backup

2019-08-16 Thread Ibrar Ahmed
On Fri, Aug 16, 2019 at 4:12 PM Ibrar Ahmed wrote: > > > > > On Fri, Aug 16, 2019 at 3:24 PM Jeevan Chalke < > jeevan.cha...@enterprisedb.com> wrote: > >> >> >> On Fri, Aug 2, 2019 at 6:43 PM vignesh C wrote: >> >>> Some comments: >>> 1) There will be some link files created for tablespace, we m

Re: Unexpected "shared memory block is still in use"

2019-08-16 Thread Tom Lane
Peter Eisentraut writes: > On 2019-08-14 01:22, Tom Lane wrote: >> Attached is a draft patch to change both shmem and sema key selection >> to be based on data directory inode rather than port. > For the POSIX APIs where the numbers are just converted to a string, why > not use both -- or forget

Re: Global temporary tables

2019-08-16 Thread Konstantin Knizhnik
I did more investigations of performance of global temp tables with shared buffers vs. vanilla (local) temp tables. 1. Combination of persistent and temporary tables in the same query. Preparation: create table big(pk bigint primary key, val bigint); insert into big values (generate_series(1,1

Re: Add test case for sslinfo

2019-08-16 Thread Peter Eisentraut
On 2019-07-08 10:18, Michael Paquier wrote: > On Mon, Jul 08, 2019 at 02:11:34PM +0800, Hao Wu wrote: >> Thank you for your quick response! I work on greenplum, and I didn't see >> this folder(src/test/ssl/ssl) before. >> I will add more certificates to test and resend again. > > Not having duplic

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Bruce Momjian
On Fri, Aug 16, 2019 at 05:58:59PM +0500, Ibrar Ahmed wrote: > > > On Thu, Aug 15, 2019 at 8:21 PM Bruce Momjian wrote: > > On Thu, Aug 15, 2019 at 11:24:46AM +0200, Antonin Houska wrote: > > > I think there are several directions we can go after all-cluster > > > encryption, >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Bruce Momjian
On Thu, Aug 15, 2019 at 09:01:05PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > I assume you are talking about my option #1. I can see if you only need > > a few tables encrypted, e.g., credit card numbers, it can be excessive > > to encrypt the entire cluster. (I t

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2019-08-16 Thread Etsuro Fujita
On Tue, Jul 30, 2019 at 6:00 PM Etsuro Fujita wrote: > On Fri, Jul 19, 2019 at 10:44 PM Robert Haas wrote: > > On Thu, Jul 18, 2019 at 2:55 AM Etsuro Fujita > > wrote: > > > I.e., partition_bounds_merge() is performed for each pair of input > > > partitioned relations for a join relation in try

Re: Unexpected "shared memory block is still in use"

2019-08-16 Thread Peter Eisentraut
On 2019-08-14 01:22, Tom Lane wrote: > Attached is a draft patch to change both shmem and sema key selection > to be based on data directory inode rather than port. > > I considered using "st_ino ^ st_dev", or some such, but decided that > that would largely just make it harder to manually correla

Re: [PATCH] Implement INSERT SET syntax

2019-08-16 Thread Ibrar Ahmed
On Fri, Aug 16, 2019 at 8:19 AM Amit Kapila wrote: > On Wed, Jul 17, 2019 at 10:00 AM Gareth Palmer > wrote: > > > > Hello, > > > > Attached is a patch that adds the option of using SET clause to specify > > the columns and values in an INSERT statement in the same manner as that > > of an UPDAT

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Ibrar Ahmed
On Thu, Aug 15, 2019 at 8:21 PM Bruce Momjian wrote: > On Thu, Aug 15, 2019 at 11:24:46AM +0200, Antonin Houska wrote: > > > I think there are several directions we can go after all-cluster > > > encryption, > > > > I think I misunderstood. What you summarize in > > > > > https://wiki.postgresql.

Re: Global temporary tables

2019-08-16 Thread Konstantin Knizhnik
On 16.08.2019 11:32, Craig Ringer wrote: You ignore the costs of evicting non-temporary data from shared_buffers, i.e. contention for space. Also increased chance of backends being forced to do direct write-out due to lack of s_b space for dirty buffers. > In case of pulling all content of

Re: Global temporary tables

2019-08-16 Thread Konstantin Knizhnik
On 16.08.2019 11:37, Craig Ringer wrote: On Fri, 16 Aug 2019 at 15:30, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: I forget or do not notice some of your questions, would you be so kind as to repeat them? Sent early by accident. Repeating question

Re: block-level incremental backup

2019-08-16 Thread Ibrar Ahmed
On Fri, Aug 16, 2019 at 3:24 PM Jeevan Chalke < jeevan.cha...@enterprisedb.com> wrote: > > > On Fri, Aug 2, 2019 at 6:43 PM vignesh C wrote: > >> Some comments: >> 1) There will be some link files created for tablespace, we might >> require some special handling for it >> > > Yep. I have that in

Re: block-level incremental backup

2019-08-16 Thread Jeevan Chalke
On Fri, Aug 9, 2019 at 6:07 AM Jeevan Ladhe wrote: > Hi Jeevan, > > I have reviewed the backup part at code level and still looking into the > restore(combine) and functional part of it. But, here are my comments so > far: > Thank you Jeevan Ladhe for reviewing the changes. > > The patches nee

Re: block-level incremental backup

2019-08-16 Thread Jeevan Chalke
On Fri, Aug 2, 2019 at 6:43 PM vignesh C wrote: > Some comments: > 1) There will be some link files created for tablespace, we might > require some special handling for it > Yep. I have that in my ToDo. Will start working on that soon. > 2) > Retry functionality is hanlded only for copying of

Re: POC: Cleaning up orphaned files using undo logs

2019-08-16 Thread Dilip Kumar
On Fri, Aug 16, 2019 at 10:56 AM Andres Freund wrote: > > Hi, > > On 2019-08-16 09:44:25 +0530, Dilip Kumar wrote: > > On Wed, Aug 14, 2019 at 2:48 PM Dilip Kumar wrote: > > > > > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > > > > > I think that batch reading should just copy th

Re: Global temporary tables

2019-08-16 Thread Craig Ringer
> > > On Fri, 16 Aug 2019 at 15:30, Konstantin Knizhnik < > k.knizh...@postgrespro.ru> wrote: > > >> I forget or do not notice some of your questions, would you be so kind as >> to repeat them? >> > > Sent early by accident. Repeating questions: Why do you need to do all this indirection with c

Re: Global temporary tables

2019-08-16 Thread Craig Ringer
On Fri, 16 Aug 2019 at 15:30, Konstantin Knizhnik wrote: > > 1. Statistic for global temporary tables (including number of tuples, > pages and all visible flag). > My position is the following: while in most cases it should not be a > problem, because users rarely create indexes or do analyze fo

Re: UNION ALL

2019-08-16 Thread Mark Pasterkamp
First of all, thank you for the replies. I am using a base installation of postgres 10.10, with no modifications to any of the system defaults. I am trying to speedup a join between two tables: the title table and the cast_info table. The title table is a table containing information about diffe

Re: Global temporary tables

2019-08-16 Thread Konstantin Knizhnik
On 16.08.2019 9:25, Craig Ringer wrote: On Tue, 13 Aug 2019 at 21:50, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: As far as I understand relpages and reltuples are set only when you perform "analyze" of the table. Also autovacuum's autoanalyze. W