Re: Data-only pg_rewind, take 2

2019-07-08 Thread Thomas Munro
On Mon, Mar 18, 2019 at 8:46 PM Chris Travers wrote: > On Mon, Mar 18, 2019 at 4:09 AM Michael Paquier wrote: >> On Sun, Mar 17, 2019 at 09:00:57PM +0800, Chris Travers wrote: >> > I also added test cases and some docs. I don't know if the docs are >> > sufficient. Feedback is appreciated. >> >

Re: FETCH FIRST clause PERCENT option

2019-07-08 Thread Surafel Temesgen
Hi Thomas, Thank you for informing me Hi Surafel, > > There's a call to adjust_limit_rows_costs() hiding under > contrib/postgres_fdw, so this fails check-world. > Fixed . I also include the review given by Ryan in attached patch regards Surafel diff --git a/contrib/postgres_fdw/postgres_fdw.c b

Re: (select query)/relation as first class citizen

2019-07-08 Thread Roman Pekar
Hi, what do you think about this idea in general? If you don't have to think about implementation for now? From my point of view writing Sql queries is very close to how functional language work if you treat "select" queries as functions without side-effects, and having query being first-class-cit

Re: Ltree syntax improvement

2019-07-08 Thread Thomas Munro
On Wed, Apr 17, 2019 at 5:29 AM Dmitry Belyavsky wrote: > I've applied your patch. > From my point of view, there is no major difference between case and chain if > here. > Neither case nor ifs allow extracting the common code to separate function - > just because there seem to be no identical p

Re: Replication & recovery_min_apply_delay

2019-07-08 Thread Thomas Munro
On Thu, Jan 31, 2019 at 3:34 AM Alvaro Herrera wrote: > On 2019-Jan-30, Konstantin Knizhnik wrote: > > I wonder if it can be considered as acceptable solution of the problem or > > there can be some better approach? > > I didn't find one. It sounds like you are in agreement that there is a proble

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-08 Thread Michael Paquier
On Fri, Jul 05, 2019 at 07:25:41PM +0200, Julien Rouhaud wrote: > On Fri, Jul 5, 2019 at 6:16 PM Peter Eisentraut > wrote: >> Isn't that also the case for your proposal? We are not going to release >> a new reindexdb before a new REINDEX. > > Sure, but my point was that once the new reindexdb i

Re: Implementing Incremental View Maintenance

2019-07-08 Thread Tatsuo Ishii
> As for how to make internal columns invisible to SELECT *, previously > there have been discussions about doing that using a new flag in > pg_attribute: > > https://www.postgresql.org/message-id/flat/CAEepm%3D3ZHh%3Dp0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA%40mail.gmail.com Now that I realized th

Re: Replication & recovery_min_apply_delay

2019-07-08 Thread Michael Paquier
On Mon, Jul 08, 2019 at 07:56:25PM +1200, Thomas Munro wrote: > On Thu, Jan 31, 2019 at 3:34 AM Alvaro Herrera > wrote: > > On 2019-Jan-30, Konstantin Knizhnik wrote: > > > I wonder if it can be considered as acceptable solution of the problem or > > > there can be some better approach? > > > > I

Re: [PATCH] kNN for btree

2019-07-08 Thread Thomas Munro
On Tue, Jul 2, 2019 at 5:47 AM Nikita Glukhov wrote: > Attached 12th version of the patches rebased onto the current master. Hi Nikita, make check-world fails for me, and in tmp_install/log/install.log I see: btree_int2.c:97:9: error: implicit declaration of function 'int2dist' is invalid in C9

Re: Add test case for sslinfo

2019-07-08 Thread Michael Paquier
On Mon, Jul 08, 2019 at 02:11:34PM +0800, Hao Wu wrote: > Thank you for your quick response! I work on greenplum, and I didn't see > this folder(src/test/ssl/ssl) before. > I will add more certificates to test and resend again. Not having duplicates would be nice. > Do you have any suggestion abo

Re: dropdb --force

2019-07-08 Thread Pavel Stehule
po 8. 7. 2019 v 0:07 odesílatel Thomas Munro napsal: > On Thu, Jun 27, 2019 at 7:15 AM Pavel Stehule > wrote: > > fixed > > Hi Pavel, > > FYI t/050_dropdb.pl fails consistently with this patch applied: > > https://travis-ci.org/postgresql-cfbot/postgresql/builds/555234838 with attached patch s

RE: [PATCH] memory leak in ecpglib

2019-07-08 Thread kuroda.hay...@fujitsu.com
Dear Meskes, Zhang, I think this modification is not enough, and I have an another idea. >> If your patch is committed, in your example, any operation for cur1 will not >> be accepted. > Although the return value after calling ecpg_get_con_name_by_cursor_name(cur1) > is NULL, in ecpg_get_connec

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-08 Thread Amit Langote
Hi Julien, Thanks for taking a look at this. On Thu, Jul 4, 2019 at 6:52 PM Julien Rouhaud wrote: > On Tue, May 28, 2019 at 6:57 AM Amit Langote wrote: > > >> Maybe like the attached? I'm not sure if we need to likewise be > > >> concerned > > >> about exec_sql_string() being handed multi-quer

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Masahiko Sawada
On Sun, Jul 7, 2019 at 1:05 AM Bruce Momjian wrote: > > On Fri, Jul 5, 2019 at 10:24:39PM +0200, Tomas Vondra wrote: > > I agree this is a pretty crucial challenge, and those requirements seem > > in direct conflict. Users use encryption to protect privacy of the data, > > but we need access to s

Re: (select query)/relation as first class citizen

2019-07-08 Thread Pavel Stehule
Hi po 8. 7. 2019 v 9:33 odesílatel Roman Pekar napsal: > Hi, > > what do you think about this idea in general? If you don't have to think > about implementation for now? From my point of view writing Sql queries is > very close to how functional language work if you treat "select" queries as > f

Re: progress report for ANALYZE

2019-07-08 Thread Tatsuro Yamada
Hi Alvaro, I'll review your patch in this week. :) I tested your patch on 6b854896. Here is a result. See below: - [Session #1] create table hoge as select * from generate_series(1, 100) a; analyze verbose hoge; [Session #2] \a \t s

Re: Synchronizing slots from primary to standby

2019-07-08 Thread Thomas Munro
On Mon, Dec 31, 2018 at 10:23 AM Petr Jelinek wrote: > As Andres has mentioned over at minimal decoding on standby thread [1], > that functionality can be used to add simple worker which periodically > synchronizes the slot state from the primary to a standby. > > Attached patch is rough implement

Re: Re: Reviving the "Stopping logical replication protocol" patch from Vladimir Gordichuk

2019-07-08 Thread Thomas Munro
On Thu, Mar 7, 2019 at 8:19 PM David Steele wrote: > On 2/16/19 10:38 PM, Dave Cramer wrote: > > Thanks for looking at this. FYI, I did not originally write this, rather > > the original author has not replied to requests. > > JDBC could use this, I assume others could as well. > > > > That said I

Re: Re: Reviving the "Stopping logical replication protocol" patch from Vladimir Gordichuk

2019-07-08 Thread Dave Cramer
On Mon, 8 Jul 2019 at 06:40, Thomas Munro wrote: > On Thu, Mar 7, 2019 at 8:19 PM David Steele wrote: > > On 2/16/19 10:38 PM, Dave Cramer wrote: > > > Thanks for looking at this. FYI, I did not originally write this, > rather > > > the original author has not replied to requests. > > > JDBC cou

Re: How to estimate the shared memory size required for parallel scan?

2019-07-08 Thread Daniel Gustafsson
> On 1 Jul 2019, at 13:03, Thomas Munro wrote: > > On Fri, May 3, 2019 at 6:06 AM Thomas Munro wrote: >> Added to commitfest. > > Rebased. Below is a review of this patch. It does what it says on the tin, applies clean without introducing compiler warnings and it seems like a good addition (f

Re: POC: Cleaning up orphaned files using undo logs

2019-07-08 Thread Amit Kapila
On Sat, Jul 6, 2019 at 1:47 AM Robert Haas wrote: > > On Mon, Jul 1, 2019 at 3:54 AM Thomas Munro wrote: > > [ new patches ] > > I took a look at 0012 today, Amit's patch for extending the binary > heap machinery, and 0013, Amit's patch for "Infrastructure to register > and fetch undo action requ

Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

2019-07-08 Thread Peter Eisentraut
On 2019-07-07 21:26, Steven Pousty wrote: > The point of the links I sent from the Python community is that they > wanted Python extinct in the wild as of Jan 1 next year. They are never > fixing it, even for a security vulnerability. The operating systems that most of our users are going to run P

Re: Commitfest 2019-07, the first of five* for PostgreSQL 13

2019-07-08 Thread Thomas Munro
On Tue, Jul 2, 2019 at 12:20 AM Thomas Munro wrote: > It's now July everywhere on Earth, so I marked CF 2019-07 as > in-progress, and 2019-09 as open for bumping patches into. I pinged > most of the "Needs Review" threads that don't apply and will do a few > more tomorrow, and then I'll try to ch

Re: [PATCH] Speedup truncates of relation forks

2019-07-08 Thread Thomas Munro
On Fri, Jul 5, 2019 at 3:03 PM Jamison, Kirk wrote: > I updated the patch which is similar to V3 of the patch, > but addressing my problem in (5) in the previous email regarding > FreeSpaceMapVacuumRange. > It seems to pass the regression test now. Kindly check for validation. Hi Kirk, FYI ther

Re: Fix doc bug in logical replication.

2019-07-08 Thread Peter Eisentraut
On 2019-06-27 18:50, Tomas Vondra wrote: >> Whether we *want* to document that it works, documenting that it >> doesn't work when it does can't be the right answer. If you want to >> couch the language to leave the door open that we may not support this >> the same way in the future I wouldn't be o

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-08 Thread Julien Rouhaud
Hi Amit, On Mon, Jul 8, 2019 at 10:52 AM Amit Langote wrote: > > On Thu, Jul 4, 2019 at 6:52 PM Julien Rouhaud wrote: > > On Tue, May 28, 2019 at 6:57 AM Amit Langote wrote: > > > >> Maybe like the attached? I'm not sure if we need to likewise be > > > >> concerned > > > >> about exec_sql_stri

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra wrote: > We're running query like this: > > SELECT a, sum(b), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 > ORDER BY 1, 2, 3 > > but we're trying to add the incremental sort *before* the aggregation, > because the optimizer also considers g

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra wrote: We're running query like this: SELECT a, sum(b), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 ORDER BY 1, 2, 3 but we're trying to add the incremental sort *before* t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 06:04:28PM +0900, Masahiko Sawada wrote: > On Sun, Jul 7, 2019 at 1:05 AM Bruce Momjian wrote: > > What about referential integrity constraints that need to check primary > > keys in the encrypted tables? I also don't see a way of delaying that, > > and if you can't do ref

Re: allow_system_table_mods stuff

2019-07-08 Thread Bruce Momjian
On Sun, Jul 7, 2019 at 11:45:49PM -0400, Bruce Momjian wrote: > On Mon, Jun 24, 2019 at 11:20:51AM -0400, Tom Lane wrote: > > I do see value in two switches not one, but it's what I said above, > > to not need to give people *more* chance-to-break-things than they > > had before when doing manual

Re: Broken defenses against dropping a partitioning column

2019-07-08 Thread Alvaro Herrera
On 2019-Jul-07, Tom Lane wrote: > Ideally, perhaps, a DROP CASCADE like this would not cascade to > the whole table but only to the table's partitioned-ness property, > leaving you with a non-partitioned table with most of its data > intact. It would take a lot of work to make that happen though,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra wrote: > > On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: > >On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra > > wrote: > >> We're running query like this: > >> > >> SELECT a, sum(b), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3

Re: POC: Cleaning up orphaned files using undo logs

2019-07-08 Thread Robert Haas
On Mon, Jul 8, 2019 at 6:57 AM Amit Kapila wrote: > I didn't have other use cases in mind and I think to some extent this > argument holds true for existing binaryheap_allocate allocate. If we > want to make it more generic, then shouldn't we try to even change the > existing binaryheap_allocate

Re: Duplicated LSN in ReorderBuffer

2019-07-08 Thread Masahiko Sawada
On Mon, Jul 8, 2019 at 9:00 AM Thomas Munro wrote: > > On Wed, Jun 26, 2019 at 2:46 AM Ildar Musin wrote: > > Attached is a simple patch that uses subxid instead of top-level xid > > in ReorderBufferAddNewTupleCids() call. It seems to fix the bug, but > > i'm not sure that this is a valid change.

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 10:32:18AM -0400, James Coleman wrote: On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra wrote: On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: >On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondra > wrote: >> We're running query like this: >> >> SELECT a, sum(b), cou

Re: [RFC] [PATCH] Flexible "partition pruning" hook

2019-07-08 Thread Mike Palmiotto
On Sun, Jul 7, 2019 at 9:46 PM Thomas Munro wrote: > On Sat, Apr 6, 2019 at 3:06 PM Andres Freund wrote: > > I've moved this to the next CF, and marked it as targeting v13. > > Hi Mike, > > Commifest 1 for PostgreSQL 13 is here. I was just going through the > automated build results for the 'fe

Re: Broken defenses against dropping a partitioning column

2019-07-08 Thread Robert Haas
On Mon, Jul 8, 2019 at 10:32 AM Alvaro Herrera wrote: > That said, I'm not sure I see the use case for an ALTER TABLE .. DROP > COLUMN command that turns a partitioned table (with existing partitions > containing data) into one non-partitioned table with all data minus the > partitioning column(s)

Re: Broken defenses against dropping a partitioning column

2019-07-08 Thread Robert Haas
On Mon, Jul 8, 2019 at 11:02 AM Robert Haas wrote: > On Mon, Jul 8, 2019 at 10:32 AM Alvaro Herrera > wrote: > > That said, I'm not sure I see the use case for an ALTER TABLE .. DROP > > COLUMN command that turns a partitioned table (with existing partitions > > containing data) into one non-par

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Joe Conway
On 7/8/19 10:19 AM, Bruce Momjian wrote: > When people are asking for multiple keys (not just for key rotation), > they are asking to have multiple keys that can be supplied by users only > when they need to access the data. Yes, the keys are always in the > datbase, but the feature request is tha

Re: Broken defenses against dropping a partitioning column

2019-07-08 Thread Tom Lane
Robert Haas writes: > On Mon, Jul 8, 2019 at 11:02 AM Robert Haas wrote: >> On Mon, Jul 8, 2019 at 10:32 AM Alvaro Herrera >> wrote: >>> That said, I'm not sure I see the use case for an ALTER TABLE .. DROP >>> COLUMN command that turns a partitioned table (with existing partitions >>> containi

Re: Broken defenses against dropping a partitioning column

2019-07-08 Thread Tom Lane
Alvaro Herrera writes: > That said, I'm not sure I see the use case for an ALTER TABLE .. DROP > COLUMN command that turns a partitioned table (with existing partitions > containing data) into one non-partitioned table with all data minus the > partitioning column(s). Yeah, it'd be a lot of work

Re: Add missing operator <->(box, point)

2019-07-08 Thread Alexander Korotkov
On Mon, Mar 11, 2019 at 2:49 AM Nikita Glukhov wrote: > 2. Add <-> to GiST box_ops. >Extracted gist_box_distance_helper() common for gist_box_distance() and >gist_bbox_distance(). For me it doesn't look worth having two distinct functions gist_box_distance_helper() and gist_bbox_distance(

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 11:18:01AM -0400, Joe Conway wrote: > On 7/8/19 10:19 AM, Bruce Momjian wrote: > > When people are asking for multiple keys (not just for key rotation), > > they are asking to have multiple keys that can be supplied by users only > > when they need to access the data. Yes,

Re: errbacktrace

2019-07-08 Thread Dmitry Dolgov
> On Sat, Jun 29, 2019 at 7:41 AM Jaime Casanova > wrote: > > This is certainly a very useful thing. Sadly, it doesn't seem to compile when > trying to use libunwind. Yeah, the same for me. To make it works I've restricted libunwind to local unwinding only: #ifdef USE_LIBUNWIND #define

Re: tableam vs. TOAST

2019-07-08 Thread Robert Haas
On Tue, Jun 25, 2019 at 2:19 AM Prabhat Sahu wrote: > I have tested the TOAST patches(v3) with different storage options like(MAIN, > EXTERNAL, EXTENDED, etc.), and > combinations of compression and out-of-line storage options. > I have used a few dummy tables with various tuple count say 10k, 20

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jul 8, 2019 at 11:18:01AM -0400, Joe Conway wrote: > > On 7/8/19 10:19 AM, Bruce Momjian wrote: > > > When people are asking for multiple keys (not just for key rotation), > > > they are asking to have multiple keys that can be suppli

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Peter Eisentraut
On 2019-07-08 17:47, Stephen Frost wrote: > Of course, we can discuss if what websites do with over-the-wire > encryption is sensible to compare to what we want to do in PG for > data-at-rest, but then we shouldn't be talking about what websites do, > it'd make more sense to look at other data-at-r

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread James Coleman
On Mon, Jul 8, 2019 at 10:58 AM Tomas Vondra wrote: > > On Mon, Jul 08, 2019 at 10:32:18AM -0400, James Coleman wrote: > >On Mon, Jul 8, 2019 at 9:59 AM Tomas Vondra > > wrote: > >> > >> On Mon, Jul 08, 2019 at 09:22:39AM -0400, James Coleman wrote: > >> >On Sun, Jul 7, 2019 at 5:02 PM Tomas Vondr

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Joe Conway
On 7/8/19 11:56 AM, Peter Eisentraut wrote: > On 2019-07-08 17:47, Stephen Frost wrote: >> Of course, we can discuss if what websites do with over-the-wire >> encryption is sensible to compare to what we want to do in PG for >> data-at-rest, but then we shouldn't be talking about what websites do,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 11:47:33AM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jul 8, 2019 at 11:18:01AM -0400, Joe Conway wrote: > > > On 7/8/19 10:19 AM, Bruce Momjian wrote: > > > > When people are asking for multiple keys (not just for ke

Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

2019-07-08 Thread Tom Lane
Peter Eisentraut writes: > I'm not trying to dismiss the importance of managing the Python > transition. But this issue has been known for many years, and the > current setup is more or less in line with the wider world. For > example, the Debian release that came out over the weekend still ship

Re: errbacktrace

2019-07-08 Thread Alvaro Herrera
On 2019-Jul-08, Dmitry Dolgov wrote: > > On Sat, Jun 29, 2019 at 7:41 AM Jaime Casanova > > wrote: > > > > This is certainly a very useful thing. Sadly, it doesn't seem to compile > > when > > trying to use libunwind. > > Yeah, the same for me. To make it works I've restricted libunwind to loc

Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

2019-07-08 Thread Tom Lane
I wrote: > But I could support having a way for individual installations to change > what the synonym means locally. Perhaps we could think about how to do > that in conjunction with the project of getting rid of pg_pltemplate > that's been kicked around before [1][2][3]. ... actually, if we had

Re: range_agg

2019-07-08 Thread Paul A Jungwirth
On Sat, Jul 6, 2019 at 12:13 PM Jeff Davis wrote: > > On Fri, 2019-07-05 at 09:58 -0700, Paul A Jungwirth wrote: > > user-defined range types. So how about I start on it and see how it > > goes? I expect I can follow the existing code for range types pretty > > closely, so maybe it won't be too ha

Re: Comment typo in tableam.h

2019-07-08 Thread Ashwin Agrawal
On Sat, Jul 6, 2019 at 12:05 AM Amit Kapila wrote: > On Tue, Jul 2, 2019 at 1:00 AM Ashwin Agrawal wrote: > > Please find attached v2 of patch 1 without objectionable comment change. > v1 of patch 2 attaching here just for convenience, no modifications made to > it. > > > > 0001* > * See table

Re: Broken defenses against dropping a partitioning column

2019-07-08 Thread Robert Haas
On Mon, Jul 8, 2019 at 11:08 AM Tom Lane wrote: > FWIW, I was imagining the action as being (1) detach all the child > partitions, (2) make parent into a non-partitioned table, (3) > drop the target column in each of these now-independent tables. > No data movement. Other than the need to acquire

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-08 Thread Tom Lane
Michael Paquier writes: > I have begun playing with regressplans.sh which enforces various > combinations of "-f s|i|n|m|h" when running the regression tests, and > I have noticed that -fh can cause the server to become stuck in the > test join_hash.sql with this query (not sure which portion of t

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-08 Thread Michael Paquier
On Mon, Jul 08, 2019 at 06:49:44PM +1200, Thomas Munro wrote: > "simple" has 20k rows and "extremely_skewed" has 20k rows but the > planner thinks it only has 2. So this going to take O(n^2) time and n > is 20k. Not sure what to do about that. Maybe "join_hash" should be > skipped for the -h (=

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-08 Thread Thomas Munro
On Mon, Jul 8, 2019 at 5:53 PM Michael Paquier wrote: > I have begun playing with regressplans.sh which enforces various > combinations of "-f s|i|n|m|h" when running the regression tests, and > I have noticed that -fh can cause the server to become stuck in the > test join_hash.sql with this quer

Re: [PATCH] kNN for btree

2019-07-08 Thread Alexander Korotkov
On Mon, Jul 1, 2019 at 8:47 PM Nikita Glukhov wrote: > On 01.07.2019 13:41, Thomas Munro wrote: > > On Tue, Mar 26, 2019 at 4:30 AM Nikita Glukhov > > wrote: > Fixed two bugs in patches 3 and 10 (see below). > Patch 3 was extracted from the main patch 5 (patch 4 in v9). > >>> This patc

Re: progress report for ANALYZE

2019-07-08 Thread Robert Haas
On Mon, Jul 8, 2019 at 5:29 AM Tatsuro Yamada wrote: > 17520|13599|postgres|16387|f|16387|scanning table|4425|4425 > 17520|13599|postgres|16387|f|16387|analyzing sample|0|0 > 17520|13599|postgres|16387|f|16387||0|0 <-- Is it Okay?? Why do we zero out the block numbers when we switch phases? The

Re: Multivariate MCV list vs. statistics target

2019-07-08 Thread Tomas Vondra
On Sun, Jul 07, 2019 at 12:02:38AM +0200, Tomas Vondra wrote: Hi, apparently v1 of the ALTER STATISTICS patch was a bit confused about length of the VacAttrStats array passed to statext_compute_stattarget, causing segfaults. Attached v2 patch fixes that, and it also makes sure we print warnings

Re: progress report for ANALYZE

2019-07-08 Thread Alvaro Herrera
On 2019-Jul-08, Robert Haas wrote: > On Mon, Jul 8, 2019 at 5:29 AM Tatsuro Yamada > wrote: > > 17520|13599|postgres|16387|f|16387|scanning table|4425|4425 > > 17520|13599|postgres|16387|f|16387|analyzing sample|0|0 > > 17520|13599|postgres|16387|f|16387||0|0 <-- Is it Okay?? > > Why do we zero

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 11:25:10AM -0400, Bruce Momjian wrote: On Mon, Jul 8, 2019 at 11:18:01AM -0400, Joe Conway wrote: On 7/8/19 10:19 AM, Bruce Momjian wrote: > When people are asking for multiple keys (not just for key rotation), > they are asking to have multiple keys that can be supplied

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jul 8, 2019 at 11:47:33AM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Mon, Jul 8, 2019 at 11:18:01AM -0400, Joe Conway wrote: > > > > On 7/8/19 10:19 AM, Bruce Momjian wrote: > > > > > When peop

Re: progress report for ANALYZE

2019-07-08 Thread Robert Haas
On Mon, Jul 8, 2019 at 2:18 PM Alvaro Herrera wrote: > Yeah, I got the impression that that was determined to be the desirable > behavior, so I made it do that, but I'm not really happy about it > either. We're not too late to change the CREATE INDEX behavior, but > let's discuss what is it that

Re: progress report for ANALYZE

2019-07-08 Thread Julien Rouhaud
On Mon, Jul 8, 2019 at 8:44 PM Robert Haas wrote: > > On Mon, Jul 8, 2019 at 2:18 PM Alvaro Herrera > wrote: > > Yeah, I got the impression that that was determined to be the desirable > > behavior, so I made it do that, but I'm not really happy about it > > either. We're not too late to change

Re: Declared but no defined functions

2019-07-08 Thread Ashwin Agrawal
On Sat, Jul 6, 2019 at 4:32 PM Masahiko Sawada wrote: > Indeed. I've tried to search again with the following script and got > more such functions. > > for func in `git ls-files | egrep "\w+\.h$" | xargs cat | egrep -v > "(^typedef)|(DECLARE)|(BKI)" | egrep "^(extern )*[\_0-9A-Za-z]+ > [\_\*0-9a-

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-08 Thread Julien Rouhaud
On Mon, Jul 8, 2019 at 9:57 AM Michael Paquier wrote: > > On Fri, Jul 05, 2019 at 07:25:41PM +0200, Julien Rouhaud wrote: > > On Fri, Jul 5, 2019 at 6:16 PM Peter Eisentraut > > wrote: > >> Isn't that also the case for your proposal? We are not going to release > >> a new reindexdb before a new

Re: Ltree syntax improvement

2019-07-08 Thread Dmitry Belyavsky
Dear Thomas, Thank you for your proofreading! Please find the updated patch attached. It also contains the missing escaping. On Mon, Jul 8, 2019 at 10:39 AM Thomas Munro wrote: > On Wed, Apr 17, 2019 at 5:29 AM Dmitry Belyavsky > wrote: > > I've applied your patch. > > From my point of view,

Re: tableam: abstracting relation sizing code

2019-07-08 Thread Robert Haas
On Wed, Jun 12, 2019 at 9:14 AM Robert Haas wrote: > On Tue, Jun 11, 2019 at 7:17 PM Andres Freund wrote: > > Just to understand: What version are you targeting? It seems pretty > > clearly v13 material to me? > > My current plan is to commit this to v13 as soon as the tree opens. Committed. --

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-08 Thread Tom Lane
Michael Paquier writes: > Well, another thing I'd like to think about is if there is any point > to keep regressplans.sh and the relevant options in postgres at this > stage. From the top of the file one can read that: The point of regressplans.sh is to see if anything goes seriously wrong when

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 02:39:44PM -0400, Stephen Frost wrote: Greetings, * Bruce Momjian (br...@momjian.us) wrote: On Mon, Jul 8, 2019 at 11:47:33AM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jul 8, 2019 at 11:18:01AM -0400, Joe Conway wrote: > > > On

Re: [HACKERS] Cached plans and statement generalization

2019-07-08 Thread Konstantin Knizhnik
On 08.07.2019 2:23, Thomas Munro wrote: On Tue, Jul 2, 2019 at 3:29 AM Konstantin Knizhnik wrote: Attached please find rebased version of the patch. Also this version can be found in autoprepare branch of this repository https://github.com/postgrespro/postgresql.builtin_pool.git on github.

Re: Declared but no defined functions

2019-07-08 Thread Tom Lane
Ashwin Agrawal writes: > Do we wish to make this a tool and have it in src/tools, either as part of > find_static tool after renaming that one to more generic name or > independent script. Well, the scripts described so far are little more than jury-rigged hacks, with lots of room for false posit

Assertion for logically decoding multi inserts into the catalog

2019-07-08 Thread Daniel Gustafsson
My patch for using heap_multi_insert in the catalog [1] failed the logical decoding part of test/recovery [2]. The assertion it failed on seems to not take multi inserts into the catalog into consideration, while the main logic does. This assertion hasn't tripped since there are no multi inserts

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 12:16:04PM -0400, Bruce Momjian wrote: ... Anyway, I will to research the reasonable data size that can be secured with a single key via AES. I will look at how PGP encrypts large files too. IMO there are various recommendations about this, for example from NIST. But

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Tomas Vondra
On Mon, Jul 08, 2019 at 12:09:58PM -0400, Joe Conway wrote: On 7/8/19 11:56 AM, Peter Eisentraut wrote: On 2019-07-08 17:47, Stephen Frost wrote: Of course, we can discuss if what websites do with over-the-wire encryption is sensible to compare to what we want to do in PG for data-at-rest, but

Re: Ltree syntax improvement

2019-07-08 Thread Alvaro Herrera
On 2019-Jul-08, Dmitry Belyavsky wrote: > Dear Thomas, > > Thank you for your proofreading! > > Please find the updated patch attached. It also contains the missing > escaping. I think all these functions you're adding should have a short sentence explaining what it does. I'm not really convin

Re: Ltree syntax improvement

2019-07-08 Thread Dmitry Belyavsky
Dear Alvaro, On Mon, Jul 8, 2019 at 11:16 PM Alvaro Herrera wrote: > On 2019-Jul-08, Dmitry Belyavsky wrote: > > > Dear Thomas, > > > > Thank you for your proofreading! > > > > Please find the updated patch attached. It also contains the missing > > escaping. > > I think all these functions you'

Re: Built-in connection pooler

2019-07-08 Thread Konstantin Knizhnik
On 08.07.2019 3:37, Thomas Munro wrote: On Tue, Jul 2, 2019 at 3:11 AM Konstantin Knizhnik wrote: On 01.07.2019 12:57, Thomas Munro wrote: Interesting work. No longer applies -- please rebase. Rebased version of the patch is attached. Also all this version of built-ni proxy can be found i

Re: Ltree syntax improvement

2019-07-08 Thread Alvaro Herrera
On 2019-Jul-08, Dmitry Belyavsky wrote: > I did not introduce any functions. I've just changed the parser. I mean the C-level functions -- count_parts_ors() and so on. > I'm not sure that it makes sense to remove any tests as most of them were > written to catch really happened bugs during the i

Re: Add missing operator <->(box, point)

2019-07-08 Thread Nikita Glukhov
Attached 3rd version of the patches. On 02.07.2019 21:55, Alexander Korotkov wrote: On Tue, Jul 2, 2019 at 9:19 PM Nikita Glukhov wrote: We could use commuted "const <-> var" operators for kNN searches, but the current implementation requires the existence of "var <-> const" operators, and or

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 12:09:58PM -0400, Joe Conway wrote: > On 7/8/19 11:56 AM, Peter Eisentraut wrote: > > On 2019-07-08 17:47, Stephen Frost wrote: > >> Of course, we can discuss if what websites do with over-the-wire > >> encryption is sensible to compare to what we want to do in PG for > >> d

Re: Add parallelism and glibc dependent only options to reindexdb

2019-07-08 Thread Julien Rouhaud
On Mon, Jul 8, 2019 at 9:08 PM Julien Rouhaud wrote: > > I'll resubmit the parallel patch using per-table only > approach Attached. 0001-Export-vacuumdb-s-parallel-infrastructure.patch Description: Binary data 0002-Add-parallel-processing-to-reindexdb.patch Description: Binary data

Re: Add missing operator <->(box, point)

2019-07-08 Thread Alexander Korotkov
On Mon, Jul 8, 2019 at 11:39 PM Nikita Glukhov wrote: > On 08.07.2019 18:22, Alexander Korotkov wrote: > For me it doesn't look worth having two distinct functions > gist_box_distance_helper() and gist_bbox_distance(). What about > having just one and leave responsibility for recheck flag to the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 02:39:44PM -0400, Stephen Frost wrote: > > > Of course, we can discuss if what websites do with over-the-wire > > > encryption is sensible to compare to what we want to do in PG for > > > data-at-rest, but then we shouldn't be talking about what websites do, > > > it'd make

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-08 Thread Tom Lane
Amit Langote writes: > [ parse-plan-memcxt_v2.patch ] I got around to looking at this finally. I'm not at all happy with the fact that it's added a plantree copy step to the only execution path through exec_simple_query. That's a very significant overhead, in many use-cases, to solve something

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 09:30:03PM +0200, Tomas Vondra wrote: > I think Bruce's proposal was to minimize the time the key is "unlocked" > in memory by only unlocking them when the user connects and supplies > some sort of secret (passphrase), and remove them from memory when the > user disconnects.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jul 8, 2019 at 02:39:44PM -0400, Stephen Frost wrote: > > > > Of course, we can discuss if what websites do with over-the-wire > > > > encryption is sensible to compare to what we want to do in PG for > > > > data-at-rest, but then we

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 05:41:51PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Well, if it was a necessary features, I assume TLS 1.3 would have found > > a way to make it secure, no? Certainly they are not shipping TLS 1.3 > > with a known weakness. > > As discuss

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jul 8, 2019 at 05:41:51PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Well, if it was a necessary features, I assume TLS 1.3 would have found > > > a way to make it secure, no? Certainly they are n

Detailed questions about pg_xact_commit_timestamp

2019-07-08 Thread Morris de Oryx
I have some specific questions about pg_xact_commit_timestamp, and am hoping that this is the right place to ask. I read a lot of the commentary about the original patch, and the contributors seem to be here. If I'm asking in the wrong place, just let me know. I'm working on a design for a concurr

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 06:04:46PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jul 8, 2019 at 05:41:51PM -0400, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > Well, if it was a necessary features, I assume TLS 1.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Joe Conway
On 7/8/19 6:04 PM, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: >> Uh, well, renaming the user was a big problem, but that is the only case >> I can think of. I don't see that as an issue for block or WAL sequence >> numbers. If we want to use a different nonce, we have to fin

Re: FETCH FIRST clause PERCENT option

2019-07-08 Thread Ryan Lambert
Hi Surafel, The v5 patch [1] applies cleanly and passes make installcheck-world. My concern with this is its performance. As a user I would expect using this feature to enable queries to run faster than they would simply pulling the full table. I tested on tables ranging from 10k rows up to 10

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Alexander Korotkov
On Mon, Jun 3, 2019 at 12:18 AM Tomas Vondra wrote: > For a moment I thought we could/should look at the histogram, becase that > could tell us if there are groups "before" the first MCV one, but I don't > think we should do that, for two reasons. Firstly, rare values may not get > to the histogra

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-07-08 Thread Alexander Korotkov
On Thu, Jul 4, 2019 at 4:25 PM James Coleman wrote: > Process questions: > - Do I need to explicitly move the patch somehow to the next CF? We didn't manage to register it on current (July) commitfest. So, please, register it on next (September) commitfest. > - Since I've basically taken over p

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jul 8, 2019 at 06:04:46PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Mon, Jul 8, 2019 at 05:41:51PM -0400, Stephen Frost wrote: > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > > Well,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 06:23:13PM -0400, Bruce Momjian wrote: > Yes, 'postgres' can be used to create a nice md5 rainbow table that > works on many servers --- good point. Are rainbow tables possible with > something like AES? > > > I appreciate that *some* of this might not be completely releva

  1   2   >