Re: Parallel Aggregates for string_agg and array_agg

2018-06-20 Thread David Rowley
On 2 May 2018 at 10:14, David Rowley wrote: > On 2 May 2018 at 08:59, Tom Lane wrote: >> My estimate for the number of people positively impacted could be off >> by a factor of a thousand, and it still wouldn't change the conclusion >> that this will hurt more people than it helps. > > It's proba

Re: ToDo: show size of partitioned table

2018-06-20 Thread Pavel Stehule
2018-06-20 7:44 GMT+02:00 Amit Langote : > On 2018/06/02 0:15, Ashutosh Bapat wrote: > > I think we should at least display "Type" as "partitioned table" for a > > partitioned table, so that it's easy to understand why the size is 0; > > partitioned tables do not hold any data by themselves. > > T

New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Haribabu Kommi
The pg_stat_statements contains the statistics of the queries that are cumulative. I find that any optimizations that are done to improve the performance of a query are not be visible clearly until the stats are reset. Currently there is a function to reset all the statistics, I find it will be use

Re: New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Daniel Gustafsson
> On 20 Jun 2018, at 09:30, Haribabu Kommi wrote: > > The pg_stat_statements contains the statistics of the queries that are > cumulative. > I find that any optimizations that are done to improve the performance of a > query > are not be visible clearly until the stats are reset. Currently ther

Re: ToDo: show size of partitioned table

2018-06-20 Thread Amit Langote
On 2018/06/20 16:21, Pavel Stehule wrote: > 2018-06-20 7:44 GMT+02:00 Amit Langote : > >> On 2018/06/02 0:15, Ashutosh Bapat wrote: >>> I think we should at least display "Type" as "partitioned table" for a >>> partitioned table, so that it's easy to understand why the size is 0; >>> partitioned t

Re: documentation is now XML

2018-06-20 Thread Liudmila Mantrova
On 04/27/2018 06:36 PM, Tom Lane wrote: Bruce Momjian writes: On Fri, Apr 27, 2018 at 11:00:36AM -0400, Peter Eisentraut wrote: That proposal seemed to indicate not only converting the source code to XML but also the build system to XSL. The latter is out of the question, I think. Why is tha

Re: New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Jeremy Schneider
> On Jun 20, 2018, at 12:30 AM, Haribabu Kommi wrote: > > Attached is a simple patch with implementation. Comments? Seems useful to me too! What are the odds we could have a column telling the timestamp when a particular query was last reset? Would that be complicated to add? -Jeremy

Re: ToDo: show size of partitioned table

2018-06-20 Thread Pavel Stehule
2018-06-20 9:44 GMT+02:00 Amit Langote : > On 2018/06/20 16:21, Pavel Stehule wrote: > > 2018-06-20 7:44 GMT+02:00 Amit Langote : > > > >> On 2018/06/02 0:15, Ashutosh Bapat wrote: > >>> I think we should at least display "Type" as "partitioned table" for a > >>> partitioned table, so that it's ea

ERROR: ORDER/GROUP BY expression not found in targetlist

2018-06-20 Thread Rajkumar Raghuwanshi
Hi, Below test case is failing with ERROR: ORDER/GROUP BY expression not found in targetlist. this issue is reproducible from PGv10. postgres=# CREATE TABLE test(c1 int, c2 text, c3 text); CREATE TABLE postgres=# INSERT INTO test SELECT i % 10, i % 250, to_char(i % 4, 'FM000') FROM GENERATE_

Re: Possible bug in logical replication.

2018-06-20 Thread Arseny Sher
Michael Paquier writes: > On Mon, Jun 18, 2018 at 09:42:36PM +0900, Michael Paquier wrote: >> On Fri, Jun 15, 2018 at 06:27:56PM +0300, Arseny Sher wrote: >> It seems to me that we still want to have the slot forwarding finish in >> this case even if this is interrupted. Petr, isn't that the in

Re: New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Sergei Kornilov
Hello > key.userid = GetUserId(); We can remove query id only from current user, right? Maybe better provide optional argument for userid? Typically user with pg_read_all_stats and user for application queries are different users. At least it should be documented. regards, Sergei

Re: ToDo: show size of partitioned table

2018-06-20 Thread Amit Langote
On 2018/06/20 16:50, Pavel Stehule wrote: > 2018-06-20 9:44 GMT+02:00 Amit Langote : >> Do you mean \dt continues to show size 0 for partitioned tables, but with >> the new option (\dtP+) shows the actual size by aggregating across >> partitions? +1 to such a feature, but we need to agree on an ac

Re: ToDo: show size of partitioned table

2018-06-20 Thread Pavel Stehule
2018-06-20 10:03 GMT+02:00 Amit Langote : > On 2018/06/20 16:50, Pavel Stehule wrote: > > 2018-06-20 9:44 GMT+02:00 Amit Langote : > >> Do you mean \dt continues to show size 0 for partitioned tables, but > with > >> the new option (\dtP+) shows the actual size by aggregating across > >> partition

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-20 Thread Amit Kapila
On Tue, Jun 19, 2018 at 7:45 PM, Robert Haas wrote: > On Mon, Jun 18, 2018 at 11:36 PM, Amit Kapila wrote: >> Fixed in the attached patch. I will wait for a day or two to see if >> Tom or Robert wants to say something and then commit. > > The patch LGTM but the commit message could perhaps use a

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-20 Thread Amit Kapila
On Mon, Jun 18, 2018 at 3:11 PM, Amit Khandekar wrote: > On 16 June 2018 at 10:44, Amit Kapila wrote: >> On Thu, Jun 14, 2018 at 10:05 PM, Tom Lane wrote: >>> >>> It looks to me like traversal of the partial subpaths is the right >>> thing here, in which case we should do >>> >>> - foreach

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-20 Thread Amit Khandekar
On 20 June 2018 at 14:24, Amit Kapila wrote: > On Mon, Jun 18, 2018 at 3:11 PM, Amit Khandekar > wrote: >> On 16 June 2018 at 10:44, Amit Kapila wrote: >>> On Thu, Jun 14, 2018 at 10:05 PM, Tom Lane wrote: It looks to me like traversal of the partial subpaths is the right thing

Re: [HACKERS] GUC for cleanup indexes threshold.

2018-06-20 Thread Alexander Korotkov
On Wed, Jun 20, 2018 at 8:32 AM Masahiko Sawada wrote: > On Wed, Jun 20, 2018 at 1:00 AM, Alexander Korotkov > > Ok. I've rephrased comment a bit. Also, you created "index vacuum" > > subsection in the "resource usage" section. I think it's not > > appropriate for this option to be in "resource

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread Thomas Munro
Adding Simon and David R. On Wed, Jun 20, 2018 at 5:44 AM, Tom Lane wrote: > Hence, I have a modest proposal: use elog(LOG) followed by Assert(false), > so that debug builds report such problems loudly but production builds > do not. We've done that before, see e.g. load_relcache_init_file aroun

Re: documentation is now XML

2018-06-20 Thread Bruce Momjian
On Wed, Jun 20, 2018 at 10:48:47AM +0300, Liudmila Mantrova wrote: > To complete the picture of possible issues with older branches in XML, we > posted a question in packager lists some time ago and didn't receive any > objections. Just to keep record of all related questions in one place, here's >

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread David Rowley
On 19 June 2018 at 17:43, Thomas Munro wrote: > The problem is that StandbyReleaseLocks() does a linear search of all > known AccessExclusiveLocks when a transaction ends. Luckily, since > v10 (commit 9b013dc2) that is skipped for transactions that haven't > taken any AELs and aren't using 2PC, b

Re: pg_verify_checksums review

2018-06-20 Thread Magnus Hagander
On Tue, Jun 19, 2018 at 10:25 AM, Daniel Gustafsson wrote: > In looking over pg_verify_checksums I found a few small things that I think > would improve on it: > > * pg_verify_checksums was placed in the Client Utils section in the docs. > Since it requries physical access to the cluster datafile

Re: Fix some error handling for read() and errno

2018-06-20 Thread Magnus Hagander
On Mon, Jun 18, 2018 at 6:42 AM, Michael Paquier wrote: > On Thu, Jun 14, 2018 at 09:50:33AM -0400, Robert Haas wrote: > > On Mon, Jun 11, 2018 at 6:11 PM, Alvaro Herrera > > wrote: > >> I would go as far as suggesting to remove qualifiers that indicate what > >> the file is for (such as "relati

Re: Adding tests for inheritance trees with temporary tables

2018-06-20 Thread Ashutosh Bapat
On Wed, Jun 20, 2018 at 11:39 AM, Michael Paquier wrote: > > Good point, I have added those. Except that you meant > tableoid::regclass. Thanks. I have checked that make check passes with this patch. I have marked this entry as ready for committer. -- Best Wishes, Ashutosh Bapat EnterpriseDB

Re: Concurrency bug in UPDATE of partition-key

2018-06-20 Thread Amit Kapila
On Tue, Jun 19, 2018 at 9:33 PM, Amit Khandekar wrote: > > Could not add RAISE statement, because the isolation test does not > seem to print those statements in the output. Instead, I have dumped > some rows in a new table through the trigger function, and did a > select from that table. Attached

Re: Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-20 Thread Jeevan Chalke
On Tue, Jun 19, 2018 at 7:14 PM, Jeevan Chalke < jeevan.cha...@enterprisedb.com> wrote: > > In the reported testcase, parallel_workers is set to 0 for all partition >> (child) relations. Which means partial parallel paths are not possible for >> child rels. However, the parent can have partial par

Re: Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-20 Thread Ashutosh Bapat
On Tue, Jun 19, 2018 at 2:13 PM, Jeevan Chalke wrote: > > > In the reported testcase, parallel_workers is set to 0 for all partition > (child) relations. Which means partial parallel paths are not possible for > child rels. However, the parent can have partial parallel paths. Thus, when > we have

Re: documentation is now XML

2018-06-20 Thread Alexander Lakhin
20.06.2018 15:03, Bruce Momjian wrote: > On Wed, Jun 20, 2018 at 10:48:47AM +0300, Liudmila Mantrova wrote: >> It's unfortunate that we'll have to deal with different formats in the >> supported branches for several more years, but we at Postgres Professional >> are >> ready to accept any your dec

Re: documentation is now XML

2018-06-20 Thread Bruce Momjian
On Wed, Jun 20, 2018 at 04:46:24PM +0300, Alexander Lakhin wrote: > 20.06.2018 15:03, Bruce Momjian wrote: > > On Wed, Jun 20, 2018 at 10:48:47AM +0300, Liudmila Mantrova wrote: > > It's unfortunate that we'll have to deal with different formats in the > supported branches for

Re: documentation is now XML

2018-06-20 Thread Tatsuo Ishii
>> It's unfortunate that we'll have to deal with different formats in the >> supported branches for several more years, but we at Postgres Professional >> are >> ready to accept any your decision on this matter for now. > > I had not thought of translations, but wouldn't all translations also > n

Re: Loaded footgun open_datasync on Windows

2018-06-20 Thread Laurenz Albe
Kuntal Ghosh wrote: [pg_test_fsync doesn't use pgwin32_open and hence doesn't respect O_DSYNC] > On Wed, Jun 6, 2018 at 2:39 PM, Amit Kapila wrote: > > On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier > > wrote: > > > On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: > > > > One anoth

Re: New function pg_stat_statements_reset_query() to reset statistics of a specific query

2018-06-20 Thread Euler Taveira
2018-06-20 4:30 GMT-03:00 Haribabu Kommi : > Attached is a simple patch with implementation. Comments? > Why don't you extend the existing function pg_stat_statements_reset()? -- Euler Taveira Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria, D

PATCH: backtraces for error messages

2018-06-20 Thread Craig Ringer
Hi folks I recently needed a way to get backtraces from errors in a convenient, non-interactive and indescriminate way. The attached patch is the result. It teaches Pg to use libunwind to self-backtrace in elog/ereport. Anyone think this is useful/interesting/worth pursuing? (Patch is currently

Re: add default parallel query to v10 release notes? (Re: [PERFORM] performance drop after upgrade (9.6 > 10))

2018-06-20 Thread Bruce Momjian
On Thu, May 24, 2018 at 08:00:25PM -0500, Justin Pryzby wrote: > Moving to -hackers; > > On Sun, Jan 28, 2018 at 06:53:10PM -0500, Bruce Momjian wrote: > > On Thu, Oct 26, 2017 at 02:45:15PM -0500, Justin Pryzby wrote: > > > Is it because max_parallel_workers_per_gather now defaults to 2 ? > > >

Re: Keeping temporary tables in shared buffers

2018-06-20 Thread Bruce Momjian
On Sat, Jun 2, 2018 at 05:18:17PM -0400, Asim Praveen wrote: > Hi Amit > > On Mon, May 28, 2018 at 4:25 AM, Amit Kapila wrote: > > > > This is one way, but I think there are other choices as well. We can > > identify and flush all the dirty (local) buffers for the relation > > being accessed pa

Re: PATCH: backtraces for error messages

2018-06-20 Thread Alvaro Herrera
On 2018-Jun-20, Craig Ringer wrote: > Hi folks > > I recently needed a way to get backtraces from errors in a convenient, > non-interactive and indescriminate way. The attached patch is the result. > It teaches Pg to use libunwind to self-backtrace in elog/ereport. > > Anyone think this is usefu

Re: partition -> partitioned

2018-06-20 Thread Alvaro Herrera
On 2018-Jun-19, Amit Langote wrote: > Noticed that the relevant code changed, so I rebased the patch. Also, > made a minor update to a nearby comment. Pushed, thanks. I made a couple of comments one or two words shorter while (IMO) not losing clarity. -- Álvaro Herrerahttps://

Re: PATCH: backtraces for error messages

2018-06-20 Thread Alexander Kuzmenkov
On 06/20/2018 06:10 PM, Craig Ringer wrote: I recently needed a way to get backtraces from errors in a convenient, non-interactive and indescriminate way. The attached patch is the result. It teaches Pg to use libunwind to self-backtrace in elog/ereport. Anyone think this is useful/interesti

Re: found xmin from before relfrozenxid on pg_catalog.pg_authid

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 15:05:59 +0300, Sergey Burladyan wrote: > Andres Freund writes: > > > You should first make sure it's actually this problem - which tables are > > holding back the xmin horizon? After that, yes, deleting the > > global/pg_internal.init file is the way to go, and I can't think

Re: PATCH: backtraces for error messages

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 11:20:49 -0400, Alvaro Herrera wrote: > > I recently needed a way to get backtraces from errors in a convenient, > > non-interactive and indescriminate way. The attached patch is the result. > > It teaches Pg to use libunwind to self-backtrace in elog/ereport. > > > > Anyone thi

Re: ERROR: ORDER/GROUP BY expression not found in targetlist

2018-06-20 Thread Tom Lane
Rajkumar Raghuwanshi writes: > Below test case is failing with ERROR: ORDER/GROUP BY expression not found > in targetlist. this issue is reproducible from PGv10. Thanks for the report. I traced through this, and the problem seems to be that split_pathtarget_at_srfs() is neglecting to attach sor

line numbers in error messages are off wrt debuggers

2018-06-20 Thread Andres Freund
Hi, I'm fairly frequently annoyed that when I see an error message in the log, I can't just generally set a breakpoint on the included line number. That's because the line number in the error message is from the *end* of the message: 2018-06-20 09:02:39.226 PDT [21145][3/2] LOG: 0: statement

Re: PATCH: backtraces for error messages

2018-06-20 Thread Tom Lane
Andres Freund writes: > On 2018-06-20 11:20:49 -0400, Alvaro Herrera wrote: >> I have no idea how expensive backtrace() and libunwind are, but I doubt >> we want to pay the cost for every message before we know that error >> requires the backtrace to be collected. Something like PGC_USERSET >> se

Re: PATCH: backtraces for error messages

2018-06-20 Thread Andres Freund
On 2018-06-20 12:04:51 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-06-20 11:20:49 -0400, Alvaro Herrera wrote: > >> I have no idea how expensive backtrace() and libunwind are, but I doubt > >> we want to pay the cost for every message before we know that error > >> requires the back

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Tom Lane
Andres Freund writes: > I'm fairly frequently annoyed that when I see an error message in the > log, I can't just generally set a breakpoint on the included line > number. That's because the line number in the error message is from the > *end* of the message: IME it varies depending on which comp

Re: pgsql: Store 2PC GID in commit/abort WAL recs for logical decoding

2018-06-20 Thread Alvaro Herrera
On 2018-Jun-15, Alvaro Herrera wrote: > By the way, why do we need to strlen() the target buffer when strlcpy > already reports the length? (You could argue that there is a difference > if the string is truncated ... but surely we don't care about that case) > I propose the attached. I decided n

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 12:12:26 -0400, Tom Lane wrote: > Andres Freund writes: > > I'm fairly frequently annoyed that when I see an error message in the > > log, I can't just generally set a breakpoint on the included line > > number. That's because the line number in the error message is from the > >

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Bruce Momjian
On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: > > On Fri, May 25, 2018 at 7:33 AM, Tom Lane wrote: > > Ashwin Agrawal writes: > > Proposing to create directory with timestamp at time of creating > tablespace > > and create symbolic link to it instead. > >

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Andres Freund
On 2018-06-20 12:12:26 -0400, Tom Lane wrote: > Andres Freund writes: > > I'm fairly frequently annoyed that when I see an error message in the > > log, I can't just generally set a breakpoint on the included line > > number. That's because the line number in the error message is from the > > *end

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Andres Freund
Hi, Hi, On 2018-05-26 10:08:57 -0400, Tom Lane wrote: > Not sure about the relative-path idea. Seems like that would create > a huge temptation to put tablespaces inside the data directory, which > would force us to deal with that can of worms. It doesn't seem impossible to normalize the path,

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andrew Dunstan
On 06/19/2018 10:46 PM, Andres Freund wrote: On 2018-06-19 22:41:26 -0400, Andrew Dunstan wrote: This unfortunately crashes and burns if we use DirectFunctionCall3 to call array_in, because it uses fn_extra. There is the CallerFInfoFunctionCall stuff, but it only has 1 and 2 arg variants, and

Re: AtEOXact_ApplyLauncher() and subtransactions

2018-06-20 Thread Amit Khandekar
On 18 June 2018 at 15:02, Amit Khandekar wrote: > On 16 June 2018 at 00:03, Amit Khandekar wrote: >> The way I am implementing this can be seen in attached >> apply_launcher_subtrans_WIP.patch. (check launcher.c changes). I >> haven't started testing it yet though. > > Attached patch passes some

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andres Freund
Hi, Just a quick scan-through review: On 2018-06-20 12:51:07 -0400, Andrew Dunstan wrote: > diff --git a/src/backend/utils/adt/pg_upgrade_support.c > b/src/backend/utils/adt/pg_upgrade_support.c > index 0c54b02..2666ab2 100644 > --- a/src/backend/utils/adt/pg_upgrade_support.c > +++ b/src/back

Re: XML/XPath issues: text/CDATA in XMLTABLE, XPath evaluated with wrong context

2018-06-20 Thread Alvaro Herrera
Hi On 2018-Mar-27, Markus Winand wrote: > I’ve found two issues with XML/XPath integration in PostgreSQL. Two > patches are attached. I’ve just separated them because the second one > is an incompatible change. > > * Patch 0001: Accept TEXT and CDATA nodes in XMLTABLE’s column_expression. I pus

Re: PATCH: backtraces for error messages

2018-06-20 Thread Robert Haas
On Wed, Jun 20, 2018 at 12:10 PM, Andres Freund wrote: > I think the problem is that this most frequently is an issue when > investigating an issue for customers. Often enough it'll legally not be > possible to gain direct access to the system, and remotely instructing > people to install an exten

Re: PATCH: backtraces for error messages

2018-06-20 Thread Andres Freund
On 2018-06-20 13:10:57 -0400, Robert Haas wrote: > On Wed, Jun 20, 2018 at 12:10 PM, Andres Freund wrote: > > If we instead had a backtrace enabled for all PANICs and some FATALs by > > default (and perhaps a SIGSEGV handler too), we'd be in a better > > place. That'd obviously only work when comp

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Tom Lane
Andres Freund writes: > On 2018-06-20 12:12:26 -0400, Tom Lane wrote: >> Personally, my habit is to set the breakpoint at errfinish, which works >> independently of just where the call is. > I do that too, occasionally. Doesn't really if there's a lot of error > messages before the bug you're wai

Re: line numbers in error messages are off wrt debuggers

2018-06-20 Thread Tom Lane
Andres Freund writes: > Seems it can be hacked around: https://stackoverflow.com/a/13497879 > But yikes, that's ugly. I bet it's not very portable either. I'm not sure that all compilers will think that HANDLEARGS should be expanded in that situation. regards, tom lane

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Ashwin Agrawal
On Wed, Jun 20, 2018 at 9:39 AM Bruce Momjian wrote: > On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: > > > > On Fri, May 25, 2018 at 7:33 AM, Tom Lane wrote: > > > > Ashwin Agrawal writes: > > > Proposing to create directory with timestamp at time of creating > > t

Re: PATCH: backtraces for error messages

2018-06-20 Thread Daniel Gustafsson
> On 20 Jun 2018, at 17:10, Craig Ringer wrote: > BTW, Álvaro posted a simpler patch at > https://www.postgresql.org/message-id/20180410213203.nl645o5vj5ap66sl@alvherre.pgsql. > It uses backtrace() from glibc, and requires each site you want to bt to be > annotated explicitly. I forgot about b

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andrew Dunstan
On 06/20/2018 12:58 PM, Andres Freund wrote: Hi, Just a quick scan-through review: On 2018-06-20 12:51:07 -0400, Andrew Dunstan wrote: diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index 0c54b02..2666ab2 100644 --- a/src/backend/utils/a

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Andres Freund
On June 20, 2018 10:31:05 AM PDT, Ashwin Agrawal wrote: >On Wed, Jun 20, 2018 at 9:39 AM Bruce Momjian wrote: > >> On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: >> > >> > On Fri, May 25, 2018 at 7:33 AM, Tom Lane >wrote: >> > >> > Ashwin Agrawal writes: >> > > Propo

Re: PATCH: backtraces for error messages

2018-06-20 Thread Robert Haas
On Wed, Jun 20, 2018 at 1:15 PM, Andres Freund wrote: > I don't think that's ok. It's perfectly possible to hit > ERRCODE_INTERNAL_ERROR at a high frequency in some situations, Really? How? > and > there's plenty cases that aren't ERRCODE_INTERNAL_ERROR where we'd want > this. E.g. a lot of gen

Re: [HACKERS] WIP: long transactions on hot standby feedback replica / proof of concept

2018-06-20 Thread Alexander Korotkov
Hi! Thank you for your work on this issue. On Wed, Feb 28, 2018 at 5:25 PM Ivan Kartyshov wrote: > Thank you for your valuable comments. I've made a few adjustments. > > The main goal of my changes is to let long read-only transactions run on > replica if hot_standby_feedback is turned on. > > >

Re: Allow auto_explain to log to NOTICE

2018-06-20 Thread Daniel Gustafsson
> On 27 Apr 2018, at 04:24, Andres Freund wrote: > > On 2018-04-27 11:52:18 +0930, Tom Dunstan wrote: >>> I'd argue this should contain the non-error cases. It's just as >>> reasonable to want to add this as a debug level or such. >> >> So all of warning, info, debug and debug1-5? > > Yea. Like

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Ashwin Agrawal
On Wed, Jun 20, 2018 at 10:50 AM Andres Freund wrote: > > > On June 20, 2018 10:31:05 AM PDT, Ashwin Agrawal > wrote: > >On Wed, Jun 20, 2018 at 9:39 AM Bruce Momjian wrote: > > > >> On Fri, May 25, 2018 at 02:17:23PM -0700, Ashwin Agrawal wrote: > >> > > >> > On Fri, May 25, 2018 at 7:33 AM, T

[PATCH] Include application_name in "connection authorized" log message

2018-06-20 Thread Don Seiler
Good afternoon, all. In trying to troubleshoot the source of a recent connection storm, I was frustrated to find that the app name was not included in the connection messages. It is there in the log_line_prefix on the disconnection messages but I would prefer it be directly visible with the connec

memory leak when serializing TRUNCATE in reorderbuffer

2018-06-20 Thread Tomas Vondra
Hi, While rebasing the logical replication patches on top of PG11, I've noticed that ReorderBufferSerializeChange claims this: case REORDER_BUFFER_CHANGE_TRUNCATE: ... /* ReorderBufferChange contains everything important */ That is not quite correct, though - the OIDs of t

Re: [PATCH] Include application_name in "connection authorized" log message

2018-06-20 Thread Stephen Frost
Greetings, * Don Seiler (d...@seiler.us) wrote: > In trying to troubleshoot the source of a recent connection storm, I was > frustrated to find that the app name was not included in the connection > messages. It is there in the log_line_prefix on the disconnection messages > but I would prefer it

Re: Postgres 10.4 crashing when using PLV8

2018-06-20 Thread David G. Johnston
On Wed, Jun 20, 2018 at 12:46 PM, Mukesh Chhatani wrote: > I am trying to use the PLV8 via function and while using the function > created via PLV8 in one of the create materialized view, postgres crashes, > attached is the log file with DEBUG5 turned on. > ​These are not the correct place to po

Re: Postgres 10.4 crashing when using PLV8

2018-06-20 Thread Mukesh Chhatani
Thanks David for the response, I have opened a issue with PLV8 team. Let me know if I should report this bug to postgres or not, since I was not sure thus I sent email earlier. Regards, Mukesh On Wed, Jun 20, 2018 at 3:02 PM, David G. Johnston < david.g.johns...@gmail.com> wrote: > On Wed, Jun

Re: ERROR: ORDER/GROUP BY expression not found in targetlist

2018-06-20 Thread Tom Lane
I wrote: > Thanks for the report. I traced through this, and the problem seems to > be that split_pathtarget_at_srfs() is neglecting to attach sortgroupref > labeling to the extra PathTargets it constructs. I don't remember > whether that code is my fault or Andres', but I'll take a look at > fix

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Tom Lane
Ashwin Agrawal writes: > Okay just bouncing another approach, how about generating UUID for a > postgres instance during initdb and pg_basebackup ? There's no uuid generation code in core postgres, for excellent reasons (lack of portability and lack of failure modes are the main objections). This

Re: libpq compression

2018-06-20 Thread Robbie Harwood
Konstantin Knizhnik writes: > On 20.06.2018 00:04, Robbie Harwood wrote: >> Konstantin Knizhnik writes: >>> On 18.06.2018 23:34, Robbie Harwood wrote: >>> I also don't like that you've injected into the *startup* path - before authentication takes place. Fundamentally, authentication

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote: > BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to > use the IV in CBC mode for this proposal. > I'd like to hear opinions by security engineer. Well, CBC makes sense, and since AES uses a 16 byte block size

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Wed, Jun 13, 2018 at 09:20:58AM -0400, Joe Conway wrote: > On 06/11/2018 05:22 AM, Masahiko Sawada wrote: > > As per discussion at PGCon unconference, I think that firstly we need > > to discuss what threats we want to defend database data against. > > Exactly. While certainly there is demand f

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 08:29:32AM -0400, Joe Conway wrote: > >> Or > >> maybe key management is really tied into the separately discussed effort > >> to create SQL VARIABLEs somehow. > > > > Could you elaborate on how key management is tied into SQL VARIABLEs? > > Well, the key management probab

Re: Push down Aggregates below joins

2018-06-20 Thread Tomas Vondra
On 06/20/2018 10:12 PM, Heikki Linnakangas wrote: > Currently, the planner always first decides the scan/join order, and > adds Group/Agg nodes on top of the joins. Sometimes it would be legal, > and beneficial, to perform the aggregation below a join. I've been > hacking on a patch to allow that.

Re: Push down Aggregates below joins

2018-06-20 Thread Tomas Vondra
On 06/20/2018 10:12 PM, Heikki Linnakangas wrote: > Currently, the planner always first decides the scan/join order, and > adds Group/Agg nodes on top of the joins. Sometimes it would be legal, > and beneficial, to perform the aggregation below a join. I've been > hacking on a patch to allow that.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote: > figure stuff out. You can infer something about the length of > particular values. Perhaps you can find cases where the same > encrypted value appears multiple times. If there's a btree index, you Most encryption methods use a rando

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote: > > At the same time, having to have a bunch of independently-decipherable > > short field values is not real secure either, especially if they're known > > to all be encrypted with the same key. But what you know or can guess > > about t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Bruce Momjian
On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: > On Mon, Jun 11, 2018 at 06:22:22PM +0900, Masahiko Sawada wrote: > > As per discussion at PGCon unconference, I think that firstly we need > > to discuss what threats we want to defend database data against. If > > We call that a thr

Postgres 10.4 crashing when using PLV8

2018-06-20 Thread Mukesh Chhatani
Hello Team, I am trying to use the PLV8 via function and while using the function created via PLV8 in one of the create materialized view, postgres crashes, attached is the log file with DEBUG5 turned on. SQL which is breaking the code and SQL function is attached. Creating materialized view - m

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: > > Note that unless the pg_catalog is protected against manipulation by > > remote storage, then TDE for user tables might be possible to > > compromise. Like so: the at

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:09 PM, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote: >> know the ordering of the values under whatever ordering semantics >> apply to that index. It's unclear to me how useful such information > > I don't think an ordered index is possible, o

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:05 PM, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 08:29:32AM -0400, Joe Conway wrote: Or maybe key management is really tied into the separately discussed effort to create SQL VARIABLEs somehow. >>> >>> Could you elaborate on how key management is tied into SQL V

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Wed, Jun 20, 2018 at 06:06:56PM -0400, Joe Conway wrote: > On 06/20/2018 05:09 PM, Bruce Momjian wrote: > > On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote: > >> know the ordering of the values under whatever ordering semantics > >> apply to that index. It's unclear to me how useful

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:03 PM, Bruce Momjian wrote: > On Wed, Jun 13, 2018 at 09:20:58AM -0400, Joe Conway wrote: >> The idea has not been extensively fleshed out yet, but the thought was >> that we create column level POLICY, which would transparently apply some >> kind of transform on input and/or outpu

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Joe Conway
On 06/20/2018 05:12 PM, Bruce Momjian wrote: > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote: >>> At the same time, having to have a bunch of independently-decipherable >>> short field values is not real secure either, especially if they're known >>> to all be encrypted with the same k

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread Thomas Munro
Hi David, On Thu, Jun 21, 2018 at 12:25 AM, David Rowley wrote: > On 19 June 2018 at 17:43, Thomas Munro wrote: >> The problem is that StandbyReleaseLocks() does a linear search of all >> known AccessExclusiveLocks when a transaction ends. Luckily, since >> v10 (commit 9b013dc2) that is skipped

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Nico Williams
On Wed, Jun 20, 2018 at 06:19:40PM -0400, Joe Conway wrote: > On 06/20/2018 05:12 PM, Bruce Momjian wrote: > > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote: > > Even if they are encrypted with the same key, they use different > > initialization vectors that are stored inside the encry

Re: Excessive CPU usage in StandbyReleaseLocks()

2018-06-20 Thread Andres Freund
Hi, On 2018-06-21 00:25:03 +1200, David Rowley wrote: > On 19 June 2018 at 17:43, Thomas Munro wrote: > > The problem is that StandbyReleaseLocks() does a linear search of all > > known AccessExclusiveLocks when a transaction ends. Luckily, since > > v10 (commit 9b013dc2) that is skipped for tra

changing xpath() and xpath_exists()

2018-06-20 Thread Alvaro Herrera
Hello Per discussion at https://postgr.es/m/0684a598-002c-42a2-ae12-f024a324e...@winand.at I intend to change xpath() and xpath_exists() in a subtly backwards- incompatible way, so that they match the behavior of SQL-standard- specified XMLTABLE. It seems sane to keep them in sync. Anybody wants

Re: Avoiding Tablespace path collision for primary and standby

2018-06-20 Thread Craig Ringer
On Thu., 21 Jun. 2018, 04:30 Tom Lane, wrote: > Ashwin Agrawal writes: > > Okay just bouncing another approach, how about generating UUID for a > > postgres instance during initdb and pg_basebackup ? > > There's no uuid generation code in core postgres, for excellent reasons > (lack of portabili

Re: partition -> partitioned

2018-06-20 Thread Amit Langote
On 2018/06/21 0:45, Alvaro Herrera wrote: > On 2018-Jun-19, Amit Langote wrote: > >> Noticed that the relevant code changed, so I rebased the patch. Also, >> made a minor update to a nearby comment. > > Pushed, thanks. I made a couple of comments one or two words shorter > while (IMO) not losin

Re: changing xpath() and xpath_exists()

2018-06-20 Thread Andrew Dunstan
On 06/20/2018 07:35 PM, Alvaro Herrera wrote: Hello Per discussion at https://postgr.es/m/0684a598-002c-42a2-ae12-f024a324e...@winand.at I intend to change xpath() and xpath_exists() in a subtly backwards- incompatible way, so that they match the behavior of SQL-standard- specified XMLTABLE.

RE: PATCH: backtraces for error messages

2018-06-20 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > I have no idea how expensive backtrace() and libunwind are, but I doubt > we want to pay the cost for every message before we know that error > requires the backtrace to be collected. Something like PGC_USERSET > server_min_backtraces=PANI

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Tsunakawa, Takayuki
From: Bruce Momjian [mailto:br...@momjian.us] > On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote: > > BTW, I want to support CBC mode encryption[3]. However, I'm not sure how > to use the IV in CBC mode for this proposal. > > I'd like to hear opinions by security engineer. > > Well, CB

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andrew Dunstan
On 06/20/2018 01:46 PM, Andrew Dunstan wrote: Attached deals with all those issues except locking. This version adds a lock on the table owning the attribute. cheers andrew -- Andrew Dunstanhttps://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, T

Re: Fast default stuff versus pg_upgrade

2018-06-20 Thread Andres Freund
Hi, On 2018-06-20 20:53:34 -0400, Andrew Dunstan wrote: > This version adds a lock on the table owning the attribute. Cool. > > /* > + * in binary upgrade mode, update the catalog with any missing > values > + * that might be present. > + */

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-20 Thread Masahiko Sawada
On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams wrote: > On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote: >> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote: >> > Note that unless the pg_catalog is protected against manipulation by >> > remote storage, then TDE for user

  1   2   >