Re: proposal \gcsv

2020-04-01 Thread Vik Fearing
On 4/1/20 1:53 AM, Tom Lane wrote: > Consider some syntax along the lines of > > \gpset (pset-option-name [pset-option-value]) ... filename > > or if you don't like parentheses, choose some other punctuation to wrap > the \pset options in. I initially thought of square brackets, but I'm > afraid

Re: Tab completion for \gx

2020-04-01 Thread Vik Fearing
On 4/1/20 5:01 AM, Bruce Momjian wrote: > > Patch applied though PG 10, thanks. Thanks! -- Vik Fearing

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote: > > On Wed, Apr 1, 2020 at 8:26 AM Masahiko Sawada > wrote: > > > > On Wed, 1 Apr 2020 at 11:46, Amit Kapila wrote: > > > > > > On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote: > > > > > > > > While testing I have found one issue. Basically,

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread Fujii Masao
On 2020/04/01 11:42, movead...@highgo.ca wrote: When I test the patch, I find an issue: I start a stream with 'promote_trigger_file' > GUC valid, and exec pg_wal_replay_pause() during recovery and as below it >> shows success to pause at the first time. I think it use a initialize >> 'S

Re: Add A Glossary

2020-04-01 Thread Jürgen Purtz
On 31.03.20 19:58, Justin Pryzby wrote: On Tue, Mar 31, 2020 at 04:13:00PM +0200, Jürgen Purtz wrote: Please find some minor suggestions in the attachment. They are based on Corey's last patch 0001-glossary-v4.patch. @@ -220,7 +220,7 @@ Records to the file system and creates a special

Re: Add A Glossary

2020-04-01 Thread Jürgen Purtz
On 31.03.20 20:07, Justin Pryzby wrote: On Mon, Mar 30, 2020 at 01:10:19PM -0400, Corey Huinker wrote: + +Aggregating + + + The act of combining a collection of data (input) values into + a single output value, which may not be of the same type as the + input val

Re: Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread movead...@highgo.ca
>But, sorry,,, I failed to understand the issue that you reported, yet... >You mean that the first call of pg_wal_replay_pause() in the step #2 >should check whether the trigger file exists or not? If so, could you >tell me why we should do that? Sorry about my pool english. The 'pg_wal_replay_pa

Re: WAL usage calculation patch

2020-04-01 Thread Julien Rouhaud
So here's a v9, rebased on top of the latest versions of Sawada-san's bug fixes (Amit's v6 for vacuum and Sawada-san's v2 for create index), with all previously mentionned changes. Note that I'm only attaching those patches for convenience and to make sure that cfbot is happy. >From a0fb471f9f498f

Re: Berserk Autovacuum (let's save next Mandrill)

2020-04-01 Thread Dean Rasheed
On Tue, 31 Mar 2020 at 22:16, Tom Lane wrote: > > > Dean Rasheed writes: > >> ... > >> It looks to me as though the problem is that statext_store() needs to > >> take its lock on pg_statistic_ext_data *before* searching for the > >> stats tuple to update. > > > Hmm, yeah, that seems like clearly

Re: pgbench - add \aset to store results of a combined query

2020-04-01 Thread Michael Paquier
On Mon, Mar 30, 2020 at 03:30:58PM +0900, Michael Paquier wrote: > Except for the addition of a test case to skip empty results when > \aset is used, I think that we are pretty good here. While hacking on the patch more by myself, I found that mixing tests for \gset and \aset was rather messy. A

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread Fujii Masao
On 2020/04/01 16:53, movead...@highgo.ca wrote: But, sorry,,, I failed to understand the issue that you reported, yet You mean that the first call of pg_wal_replay_pause() in the step #2 should check whether the trigger file exists or not? If so, could you tell me why we should do that?

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2020-04-01 Thread Alexey Kondratov
On 2020-04-01 05:19, Michael Paquier wrote: On Tue, Mar 31, 2020 at 03:48:21PM +0900, Michael Paquier wrote: Thanks, committed 0001 after fixing the order of the headers. One patch left. And committed now 0002, meaning that we are officially done. Thanks Alexey for your patience. Thanks f

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread movead...@highgo.ca
>Thanks for the explanation again! Maybe I understand your point. Great. >As far as I read the code, in the standby mode, the startup process >periodically checks the trigger file in WaitForWALToBecomeAvailable(). >No? Yes it is. >There can be small delay between the creation of the trigger file

Re: control max length of parameter values logged

2020-04-01 Thread Alexey Bashtanov
Hi, +If greater than zero, bind parameter values reported in non-error +statement-logging messages are trimmed to no more than this many bytes. Can I suggest to say: "Limit bind parameter values reported by non-error statement-logging messages to this many bytes". Or, "The maxi

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-04-01 Thread Fujii Masao
On 2020/04/01 3:42, Julien Rouhaud wrote: On Wed, Apr 01, 2020 at 02:43:10AM +0900, Fujii Masao wrote: On 2020/03/31 16:33, Julien Rouhaud wrote: v12 attached! Thanks for updating the patch! The patch looks good to me. I applied minor and cosmetic changes into the patch. Attached is th

Re: control max length of parameter values logged

2020-04-01 Thread Justin Pryzby
On Wed, Apr 01, 2020 at 10:10:55AM +0100, Alexey Bashtanov wrote: > Hi, > > > +If greater than zero, bind parameter values reported in non-error > > > +statement-logging messages are trimmed to no more than this many > > > bytes. > > Can I suggest to say: > > > > "Limit bind param

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread Fujii Masao
On 2020/04/01 17:58, movead...@highgo.ca wrote: Thanks for the explanation again! Maybe I understand your point. Great. As far as I read the code, in the standby mode, the startup process periodically checks the trigger file in WaitForWALToBecomeAvailable(). No? Yes it is. There can be s

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread movead...@highgo.ca
>This happens because the startup process detects the trigger file >after pg_wal_replay_pause() succeeds, and then make the recovery >get out of the paused state. Yes that is. >It might be problematic to end the paused >state silently? So, to make the situation less confusing, what about >emitti

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread Fujii Masao
On 2020/04/01 18:56, movead...@highgo.ca wrote: >This happens because the startup process detects the trigger file after pg_wal_replay_pause() succeeds, and then make the recovery get out of the paused state. Yes that is. >It might be problematic to end the paused state silently? So, to

Re: WAL usage calculation patch

2020-04-01 Thread Amit Kapila
On Wed, Apr 1, 2020 at 1:32 PM Julien Rouhaud wrote: > > So here's a v9, rebased on top of the latest versions of Sawada-san's bug > fixes > (Amit's v6 for vacuum and Sawada-san's v2 for create index), with all > previously mentionned changes. > Few other comments: v9-0003-Add-infrastructure-to-

Re: WAL usage calculation patch

2020-04-01 Thread Amit Kapila
On Wed, Apr 1, 2020 at 4:29 PM Amit Kapila wrote: > > v9-0005-Keep-track-of-WAL-usage-in-pg_stat_statements > One more comment related to this patch. + + snprintf(buf, sizeof buf, UINT64_FORMAT, tmp.wal_bytes); + + /* Convert to numeric. */ + wal_bytes = DirectFunctionCall3(numeric_in, + CStringG

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread Fujii Masao
On 2020/04/01 19:37, Fujii Masao wrote: On 2020/04/01 18:56, movead...@highgo.ca wrote:  >This happens because the startup process detects the trigger file after pg_wal_replay_pause() succeeds, and then make the recovery get out of the paused state. Yes that is.  >It might be problematic

Re: [PATCH] Opclass parameters

2020-04-01 Thread Alexander Korotkov
On Tue, Mar 31, 2020 at 12:15 PM Alexander Korotkov wrote: > What is XXX supposed to be? > > The rest of patch looks good to me. I've pushed the patch excepts XXX. Thank you. You're welcome to clarify XXX and/or do additional corrections. -- Alexander Korotkov Postgres Professional: http://

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-04-01 Thread Justin Pryzby
On Wed, Apr 01, 2020 at 03:03:34PM +0900, Michael Paquier wrote: > On Tue, Mar 31, 2020 at 01:56:07PM +0300, Alexey Kondratov wrote: > > I am fine with allowing REINDEX (CONCURRENTLY), but then we will have to > > support both syntaxes as we already do for VACUUM. Anyway, if we agree to > > add par

Re: [PATCH] Redudant initilization

2020-04-01 Thread Ranier Vilela
Hi, New patch with yours suggestions. best regards, Ranier Vilela v2_redundant_initialization.patch Description: Binary data

Re: [PATCH] Opclass parameters

2020-04-01 Thread Justin Pryzby
On Wed, Apr 01, 2020 at 02:53:41PM +0300, Alexander Korotkov wrote: > On Tue, Mar 31, 2020 at 12:15 PM Alexander Korotkov > wrote: > > What is XXX supposed to be? > > > > The rest of patch looks good to me. > > I've pushed the patch excepts XXX. Thank you. > You're welcome to clarify XXX and/or

Re: [PATCH] Opclass parameters

2020-04-01 Thread Alexander Korotkov
On Wed, Apr 1, 2020 at 2:59 PM Justin Pryzby wrote: > On Wed, Apr 01, 2020 at 02:53:41PM +0300, Alexander Korotkov wrote: > > On Tue, Mar 31, 2020 at 12:15 PM Alexander Korotkov > > wrote: > > > What is XXX supposed to be? > > > > > > The rest of patch looks good to me. > > > > I've pushed the pa

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 12:01 PM Amit Kapila wrote: > > On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote: > > > > > Agreed. I've attached the updated patch. > > > > > > Thank you for testing, Dilip! > > > > Thanks! One hunk is failing on the latest head. And, I have rebased > > the patch for my

Re: [Proposal] Global temporary tables

2020-04-01 Thread Prabhat Sahu
Hi Wenjing, I hope we need to change the below error message. postgres=# create global temporary table gtt(c1 int) on commit preserve rows; CREATE TABLE postgres=# create materialized view mvw as select * from gtt; ERROR: materialized views must not use global temporary tables* or views* Anyways

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-04-01 Thread Justin Pryzby
On Wed, Apr 01, 2020 at 06:57:18AM -0500, Justin Pryzby wrote: > Alexey suggested that those changes should be done as a separate patch, with > the tablespace options built on top. Which makes sense. I had quite some fun > rebasing these with patches in that order. > > However, I've kept my chan

Re: Some problems of recovery conflict wait events

2020-04-01 Thread Fujii Masao
On 2020/03/30 20:10, Masahiko Sawada wrote: On Fri, 27 Mar 2020 at 17:54, Fujii Masao wrote: On 2020/03/04 14:31, Masahiko Sawada wrote: On Wed, 4 Mar 2020 at 13:48, Fujii Masao wrote: On 2020/03/04 13:27, Michael Paquier wrote: On Wed, Mar 04, 2020 at 01:13:19PM +0900, Masahiko Sa

Re: backend type in log_line_prefix?

2020-04-01 Thread Peter Eisentraut
On 2020-04-01 03:55, Bruce Momjian wrote: Agreed. I ended up moving "wal" as a separate word, since it looks cleaner; patch attached. Tools that look for the backend type in pg_stat_activity would need to be adjusted; it would be an incompatibility. Maybe changing it would cause too much dis

Re: color by default

2020-04-01 Thread Peter Eisentraut
On 2020-03-30 10:03, Michael Paquier wrote: On Sun, Mar 29, 2020 at 02:55:37PM +0200, Juan José Santamaría Flecha wrote: Add it to the tests done when PG_COLOR is "auto". FWIW, I am not sure that it is a good idea to stick into the code knowledge inherent to TERM. That would likely rot depend

Re: WAL usage calculation patch

2020-04-01 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 5:01 PM Amit Kapila wrote: > > On Wed, Apr 1, 2020 at 4:29 PM Amit Kapila wrote: > > > > v9-0005-Keep-track-of-WAL-usage-in-pg_stat_statements > > > > One more comment related to this patch. > + > + snprintf(buf, sizeof buf, UINT64_FORMAT, tmp.wal_bytes); > + > + /* Convert

Re: color by default

2020-04-01 Thread Peter Eisentraut
On 2020-03-30 10:08, Michael Paquier wrote: On Sun, Mar 29, 2020 at 11:56:15AM +0200, Peter Eisentraut wrote: I didn't do this because it would create additional complications in the man pages. But there is now an index entry, so it's possible to find more information. Cannot you add a link t

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Robert Haas
On Wed, Apr 1, 2020 at 2:40 AM Andres Freund wrote: > The problem is that there's no protection again the xids in the > ringbuffer getting old enough to wrap around. Given that practical uses > of old_snapshot_threshold are likely to be several hours to several > days, that's not particularly hard

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-03-31 23:40:08 -0700, Andres Freund wrote: > I added some debug output to print the mapping before/after changes by > MaintainOldSnapshotTimeMapping() (note that I used timestamps relative > to the server start in minutes/seconds to make it easier to interpret). Now attached. Greetin

Re: allow online change primary_conninfo

2020-04-01 Thread Peter Eisentraut
On 2020-03-28 11:49, Sergei Kornilov wrote: I attached updated patch: walreceiver will use configured primary_slot_name as temporary slot name if wal_receiver_create_temp_slot is enabled. The original setup worked consistently with pg_basebackup. In pg_basebackup, if you specify -S/--slot, t

Re: Commitfest 2020-03 Now in Progress

2020-04-01 Thread David Steele
On 3/17/20 8:10 AM, David Steele wrote: On 3/1/20 4:10 PM, David Steele wrote: The last Commitfest for v13 is now in progress! Current stats for the Commitfest are: Needs review: 192 Waiting on Author: 19 Ready for Committer: 4 Total: 215 Halfway through, here's where we stand.  Note that a

Re: potential stuck lock in SaveSlotToPath()

2020-04-01 Thread Peter Eisentraut
On 2020-03-27 08:48, Michael Paquier wrote: On Thu, Mar 26, 2020 at 02:16:05PM +0100, Peter Eisentraut wrote: committed and backpatched The patch committed does that in three places: /* rename to permanent file, fsync file and directory */ if (rename(tmppath, path) != 0) { +

Re: WAL usage calculation patch

2020-04-01 Thread Julien Rouhaud
Hi, I'm replying here to all reviews that have been sent, thanks a lot! On Wed, Apr 01, 2020 at 04:29:16PM +0530, Amit Kapila wrote: > On Wed, Apr 1, 2020 at 1:32 PM Julien Rouhaud wrote: > > > > So here's a v9, rebased on top of the latest versions of Sawada-san's bug > > fixes > > (Amit's v6

Re: Less-silly selectivity for JSONB matching operators

2020-04-01 Thread Tom Lane
Alexey Bashtanov writes: > On 31/03/2020 18:53, Tom Lane wrote: >> Renamed "matchsel" to "matchingsel" etc, added DEFAULT_MATCHING_SEL, >> rebased over commit 911e70207. Since that commit already created >> new versions of the relevant contrib modules, I think we can just >> redefine what those v

Re: control max length of parameter values logged

2020-04-01 Thread Tom Lane
Justin Pryzby writes: > On Wed, Apr 01, 2020 at 10:10:55AM +0100, Alexey Bashtanov wrote: >>> Could you make zero a normal value and -1 the "special" value to disable >>> trimming ? >> I can, but then for the sake of consistency I'll have to do the same for >> log_parameter_max_length. >> Then we

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 10:01:07 -0400, Robert Haas wrote: > On Wed, Apr 1, 2020 at 2:40 AM Andres Freund wrote: > > The problem is that there's no protection again the xids in the > > ringbuffer getting old enough to wrap around. Given that practical uses > > of old_snapshot_threshold are likely to be

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Robert Haas
On Wed, Apr 1, 2020 at 2:40 AM Andres Freund wrote: > I added some debug output to print the mapping before/after changes by > MaintainOldSnapshotTimeMapping() (note that I used timestamps relative > to the server start in minutes/seconds to make it easier to interpret). > > And the output turns o

Re: Verify true root on replicas with amcheck

2020-04-01 Thread David Steele
On 1/9/20 3:55 AM, godjan • wrote: Hi, we have trouble to detect true root corruptions on replicas. I made a patch for resolving it with the locking meta page and potential root page. I heard that amcheck has an invariant about locking no more than 1 page at a moment for avoiding deadlocks. Is

Re: proposal \gcsv

2020-04-01 Thread Tom Lane
Vik Fearing writes: > On 4/1/20 1:53 AM, Tom Lane wrote: >> Consider some syntax along the lines of >> \gpset (pset-option-name [pset-option-value]) ... filename > If parens are going to be required, why don't we just add them to \g? > TABLE blah \g (format csv) filename Yeah, if we're willing t

Re: Online checksums patch - once again

2020-04-01 Thread David Steele
On 1/18/20 6:18 PM, Daniel Gustafsson wrote: Attached is a v16 rebased on top of current master which addresses the above commented points, and which I am basing the concurrency work on. This patch no longer applies cleanly: http://cfbot.cputube.org/patch_27_2260.log The CF entry has been u

Re: Removing unneeded self joins

2020-04-01 Thread David Steele
On 1/27/20 11:10 PM, Andrey Lepikhov wrote: Rebased version v.22. - Added enable_self_join_removal GUC (true is default) - The joinquals of the relation that is being removed, redistributed in accordance with the remove_rel_from_query () machinery. This patch no longer applies cleanly on sr

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 11:15:14 -0400, Robert Haas wrote: > On Wed, Apr 1, 2020 at 2:40 AM Andres Freund wrote: > > I added some debug output to print the mapping before/after changes by > > MaintainOldSnapshotTimeMapping() (note that I used timestamps relative > > to the server start in minutes/secon

Re: BufFileRead() error signalling

2020-04-01 Thread David Steele
Hi Thomas, On 11/29/19 9:46 PM, Thomas Munro wrote: Ok. Here is a first attempt at that. It's been a few CFs since this patch received an update, though there has been plenty of discussion. Perhaps it would be best to mark it RwF until you have a chance to produce an update patch? Rega

Re: proposal \gcsv

2020-04-01 Thread Daniel Verite
Tom Lane wrote: > I could see having a command to copy the current primary formatting > parameters to the alternate area, too. We could have a stack to store parameters before temporary changes, for instance if you want to do one csv export and come back to normal without assuming what "

Re: proposal \gcsv

2020-04-01 Thread Isaac Morland
On Wed, 1 Apr 2020 at 11:52, Daniel Verite wrote: > Tom Lane wrote: > > > I could see having a command to copy the current primary formatting > > parameters to the alternate area, too. > > We could have a stack to store parameters before temporary > changes, for instance if you want to d

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Robert Haas
On Wed, Apr 1, 2020 at 11:09 AM Andres Freund wrote: > That doesn't exist in all the back branches. Think it'd be easier to add > code to explicitly prune it during MaintainOldSnapshotTimeMapping(). That's reasonable. > There's really no reason at all to have bins of one minute. As it's a > PGC_

Re: pgbench - add pseudo-random permutation function

2020-04-01 Thread David Steele
Hi Fabien, On 2/1/20 5:12 AM, Fabien COELHO wrote: Attached is an attempt at improving things. I have added a explicit note and hijacked an existing example to better illustrate the purpose of the function. This patch does not build on Linux due to some unused functions and variables: http

Re: proposal \gcsv

2020-04-01 Thread Pavel Stehule
st 1. 4. 2020 v 17:52 odesílatel Daniel Verite napsal: > Tom Lane wrote: > > > I could see having a command to copy the current primary formatting > > parameters to the alternate area, too. > > We could have a stack to store parameters before temporary > changes, for instance if you want

Re: A bug when use get_bit() function for a long bytea string

2020-04-01 Thread Tom Lane
"movead...@highgo.ca" writes: > [ long_bytea_string_bug_fix_ver5.patch ] I don't think this has really solved the overflow hazards. For example, in binary_encode we've got resultlen = enc->encode_len(VARDATA_ANY(data), datalen); result = palloc(VARHDRSZ + resultlen); and all yo

Re: wraparound dangers in snapshot too old

2020-04-01 Thread Andres Freund
Hi, On 2020-03-31 21:53:04 -0700, Andres Freund wrote: > I am trying to change the snapshot too old infrastructure so it > cooperates with my snapshot scalability patch. While trying to > understand the code sufficiently, I think I found a fairly serious > issue: I accidentally sent this email, I

Re: proposal \gcsv

2020-04-01 Thread Tom Lane
Pavel Stehule writes: > It can work, but it is not user friendly - my proposal was motivated by > using some quick csv exports to gplot's pipe. I kind of liked the stack idea, myself. It's simpler than what I was suggesting and it covers probably 90% of the use-case. However, if we prefer somet

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 9:02 AM Robert Haas wrote: > I complained > when you added those error checks to vacuum in back-branches, and > since that release went out people are regularly tripping those checks > and taking prolonged outages for a problem that wasn't making them > unhappy before. I kno

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2020-04-01 Thread Ashutosh Bapat
On Thu, 26 Mar 2020 at 00:35, Tomas Vondra wrote: > Hi, > > I've started reviewing the patch a couple days ago. I haven't done any > extensive testing, but I do have a bunch of initial comments that I can > share now. > > 1) I wonder if this needs to update src/backend/optimizer/README, which > d

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Robert Haas
On Wed, Apr 1, 2020 at 1:03 PM Peter Geoghegan wrote: > I don't think that it's fair to characterize Andres' actions in that > situation as in any way irresponsible. We had an extremely complicated > data corruption bug that he went to great lengths to fix, following > two other incorrect fixes. H

Re: BUG #16109: Postgres planning time is high across version (Expose buffer usage during planning in EXPLAIN)

2020-04-01 Thread Fujii Masao
On 2020/03/31 10:31, Justin Pryzby wrote: On Wed, Jan 29, 2020 at 12:15:59PM +0100, Julien Rouhaud wrote: Rebase due to conflict with 3ec20c7091e97. This is failing to apply probably since 4a539a25ebfc48329fd656a95f3c1eb2cda38af3. Could you rebase? (Also, not sure if this can be set as RF

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 12:02:18 -0400, Robert Haas wrote: > On Wed, Apr 1, 2020 at 11:09 AM Andres Freund wrote: > > There's really no reason at all to have bins of one minute. As it's a > > PGC_POSTMASTER GUC, it should just have didided time into bins of > > (old_snapshot_threshold * USEC_PER_SEC) /

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 10:28 AM Robert Haas wrote: > Sure, but not all levels of risk are equal. Jumping out of a plane > carries some risk of death whether or not you have a parachute, but > that does not mean that we shouldn't worry about whether you have one > or not before you jump. > > In thi

Re: proposal \gcsv

2020-04-01 Thread Alvaro Herrera
On 2020-Apr-01, Pavel Stehule wrote: > It can work, but it is not user friendly - my proposal was motivated by > using some quick csv exports to gplot's pipe. Can we fix that by adding some syntax to allow command aliases? So you could add to your .psqlrc something like \alias \gcsv \pset push a

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 11:04:43 -0700, Peter Geoghegan wrote: > On Wed, Apr 1, 2020 at 10:28 AM Robert Haas wrote: > > Is there any chance that you're planning to look into the details? > > That would certainly be welcome from my perspective. +1 This definitely needs more eyes. I am not even close t

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 13:27:56 -0400, Robert Haas wrote: > Perhaps "irresponsible" is the wrong word, but it's certainly caused > problems for multiple EnterpriseDB customers, and in my view, those > problems weren't necessary. Either a WARNING or an ERROR would have > shown up in the log, but an ERRO

Re: BUG #16109: Postgres planning time is high across version (Expose buffer usage during planning in EXPLAIN)

2020-04-01 Thread Julien Rouhaud
On Wed, Apr 1, 2020 at 7:51 PM Fujii Masao wrote: > > > On 2020/03/31 10:31, Justin Pryzby wrote: > > On Wed, Jan 29, 2020 at 12:15:59PM +0100, Julien Rouhaud wrote: > >> Rebase due to conflict with 3ec20c7091e97. > > > > This is failing to apply probably since > > 4a539a25ebfc48329fd656a95f3c1eb

Re: [PATCH] Check operator when creating unique index on partition table

2020-04-01 Thread Tom Lane
Guancheng Luo writes: > On Mar 26, 2020, at 01:00, Tom Lane wrote: >> This would reject, for example, a hash index associated with a btree-based >> partition constraint, but I'm not sure we're losing anything much thereby. > There is cases when a BTREE index associated with a HASH partition key,

Re: tweaking perfect hash multipliers

2020-04-01 Thread John Naylor
On Tue, Mar 31, 2020 at 4:05 PM John Naylor wrote: > > On Tue, Mar 31, 2020 at 2:31 AM Andres Freund wrote: > > I think the form of lea generated here is among the ones that can only > > be executed on port 1. Whereas e.g. an register+register/immediate add > > can be executed on four different p

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Robert Haas
On Wed, Apr 1, 2020 at 2:37 PM Andres Freund wrote: > Just continuing is easier said than done. Especially with the background > of knowing that several users had hit the bug that allowed all of the > above to be hit, and that advancing relfrozenxid further would make it > worse. Fair point, but

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Kevin Grittner
On Wed, Apr 1, 2020 at 10:09 AM Andres Freund wrote: First off, many thanks to Andres for investigating this, and apologies for the bugs. Also thanks to Michael for making sure I saw the thread. I must also apologize that for not being able to track the community lists consistently due to healt

Re: error context for vacuum to include block number

2020-04-01 Thread Andres Freund
On 2020-04-01 07:54:45 +0530, Amit Kapila wrote: > Pushed. I think we are done here. The patch is marked as committed in > CF. Thank you! Awesome! Thanks for all your work on this, all. This'll make it a lot easier to debug errors during autovacuum.

Re: error context for vacuum to include block number

2020-04-01 Thread Alvaro Herrera
On 2020-Apr-01, Andres Freund wrote: > On 2020-04-01 07:54:45 +0530, Amit Kapila wrote: > > Pushed. I think we are done here. The patch is marked as committed in > > CF. Thank you! > > Awesome! Thanks for all your work on this, all. This'll make it a lot > easier to debug errors during autovacu

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, Nice to have you back for a bit! Even if the circumstances aren't great... It's very understandable that the lists are past your limits, I barely keep up these days. Without any health issues. On 2020-04-01 14:10:09 -0500, Kevin Grittner wrote: > Perhaps the lack of evidence for usage in th

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Kevin Grittner
On Wed, Apr 1, 2020 at 2:43 PM Andres Freund wrote: > The thing that makes me really worried is that the contents of the time > mapping seem very wrong. I've reproduced query results in a REPEATABLE > READ transaction changing (pruned without triggering an error). That is a very big problem. O

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Robert Haas
On Wed, Apr 1, 2020 at 3:43 PM Andres Freund wrote: > The thing that makes me really worried is that the contents of the time > mapping seem very wrong. I've reproduced query results in a REPEATABLE > READ transaction changing (pruned without triggering an error). And I've > reproduced rows not ge

Re: backup manifests

2020-04-01 Thread Andres Freund
Hi, On 2020-03-31 22:15:04 -0700, Noah Misch wrote: > On Tue, Mar 31, 2020 at 03:50:34PM -0700, Andres Freund wrote: > > On 2020-03-31 14:10:34 -0400, Robert Haas wrote: > > > +/* > > > + * Attempt to parse the WAL files required to restore from backup using > > > + * pg_waldump. > > > + */ > > >

Re: backup manifests

2020-04-01 Thread Andres Freund
Hi, On 2020-03-31 14:56:07 +0530, Amit Kapila wrote: > On Tue, Mar 31, 2020 at 11:10 AM Noah Misch wrote: > > On Mon, Mar 30, 2020 at 12:16:31PM -0700, Andres Freund wrote: > > > On 2020-03-30 15:04:55 -0400, Robert Haas wrote: > > > I'm mildly inclined to name it pg_validate, pg_validate_dbdir o

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 15:11:52 -0500, Kevin Grittner wrote: > On Wed, Apr 1, 2020 at 2:43 PM Andres Freund wrote: > > > The thing that makes me really worried is that the contents of the time > > mapping seem very wrong. I've reproduced query results in a REPEATABLE > > READ transaction changing (pru

Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread James Coleman
Currently there's no good way that I'm aware of for monitoring software to check what the xmin horizon is being blocked at. You can check pg_stat_replication and pg_replication_slots and txid_snapshot_xmin(txid_current_snapshot()) and so on, but that list can grow, and it means monitoring setups ne

Re: backup manifests

2020-04-01 Thread David Steele
On 3/31/20 7:57 AM, Robert Haas wrote: On Mon, Mar 30, 2020 at 7:24 PM David Steele wrote: I'm confused as to why you're not seeing that. What's the exact sequence of steps? $ pg_basebackup -D test/backup5 --manifest-checksums=SHA256 $ vi test/backup5/backup_manifest * Add 'X' to the c

Re: [PATCH] ltree, lquery, and ltxtquery binary protocol support

2020-04-01 Thread Tom Lane
I wrote: > Fortunately for the odds of getting this patch accepted, we just > pushed an ALTER TYPE improvement that will solve your problem [1]. > Please replace ltree--1.2.sql with an upgrade script that uses > that, and resubmit. I decided it would be a shame for this to miss v13, seeing that (a

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-01 Thread Tomas Vondra
On Wed, Apr 01, 2020 at 09:05:27AM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 11:07 PM James Coleman wrote: On Tue, Mar 31, 2020 at 10:44 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 10:12:29PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 9:59 PM Tomas Vondra > > wrot

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread Tom Lane
James Coleman writes: > To my knowledge the current oldest xmin (GetOldestXmin() if I'm not > mistaken) isn't exposed directly in any view or function by Postgres. You could do something like select max(age(backend_xmin)) from pg_stat_activity; though I'm not sure whether that accounts for abso

Re: [PATCH] ltree, lquery, and ltxtquery binary protocol support

2020-04-01 Thread Nino Floris
Hi Tom, Thanks a lot for pushing this through. In complete agreement on fixing mbstrlen, it would clearly have lead to cut off string sends, or worse (does the binary protocol use null terminated strings, or are they length prefixed?). Apologies anyways, it's been a while so I don't know how it m

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread Alvaro Herrera
On 2020-Apr-01, Tom Lane wrote: > James Coleman writes: > > To my knowledge the current oldest xmin (GetOldestXmin() if I'm not > > mistaken) isn't exposed directly in any view or function by Postgres. > > You could do something like > > select max(age(backend_xmin)) from pg_stat_activity; > >

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 1:25 PM Robert Haas wrote: > Maybe that contrib module could even have some functions to simulate > aging without the passage of any real time. Like, say you have a > function or procedure old_snapshot_pretend_time_has_passed(integer), > and it moves oldSnapshotControl->head

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 3:00 PM Peter Geoghegan wrote: > I like that idea. I think that I've spotted what may be an independent > bug, but I have to wait around for a minute or two to reproduce it > each time. Makes it hard to get to a minimal test case. I now have simple steps to reproduce a bug

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread James Coleman
On Wed, Apr 1, 2020 at 5:58 PM Alvaro Herrera wrote: > > On 2020-Apr-01, Tom Lane wrote: > > > James Coleman writes: > > > To my knowledge the current oldest xmin (GetOldestXmin() if I'm not > > > mistaken) isn't exposed directly in any view or function by Postgres. > > > > You could do something

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 14:11:11 -0700, Andres Freund wrote: > As far as I can tell, with a large old_snapshot_threshold, it can take a > very long time to get to a head_timestamp that's old enough for > TransactionIdLimitedForOldSnapshots() to do anything. Look at this > trace of a pgbench run with ol

Re: Ltree syntax improvement

2020-04-01 Thread Tom Lane
Nikita Glukhov writes: > [ latest version of ltree syntax extension ] This is going to need another rebase after all the other ltree hacking that just got done. However, I did include 0001 (use a switch) in the commit I just pushed, so you don't need to worry about that.

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Apr-01, Tom Lane wrote: >> The fact that I had to use max(age(...)) in that sample query >> hints at one reason: it's really hard to do arithmetic correctly >> on raw XIDs. Dealing with wraparound is a problem, and knowing >> what's past or future is even harder.

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 15:30:39 -0700, Peter Geoghegan wrote: > On Wed, Apr 1, 2020 at 3:00 PM Peter Geoghegan wrote: > > I like that idea. I think that I've spotted what may be an independent > > bug, but I have to wait around for a minute or two to reproduce it > > each time. Makes it hard to get to

Re: Add A Glossary

2020-04-01 Thread Alvaro Herrera
On 2020-Apr-01, Jürgen Purtz wrote: > > On 31.03.20 19:58, Justin Pryzby wrote: > > On Tue, Mar 31, 2020 at 04:13:00PM +0200, Jürgen Purtz wrote: > > > Please find some minor suggestions in the attachment. They are based on > > > Corey's last patch 0001-glossary-v4.patch. > > > @@ -220,7 +220,7 @

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 16:59:51 -0700, Andres Freund wrote: > The primary issue here is that there is no TestForOldSnapshot() in > heap_hot_search_buffer(). Therefore index fetches will never even try to > detect that tuples it needs actually have already been pruned away. FWIW, with autovacuum=off th

Re: Allow continuations in "pg_hba.conf" files

2020-04-01 Thread David Zhang
Hi Fabien, Should we consider the case "\ ", i.e. one or more spaces after the backslash? For example, if I replace a user map "mymap /^(.*)@mydomain\.com$ \1" with "mymap /^(.*)@mydomain\.com$ \ " "\1" by adding one extra space after the backslash, then I got the pg_role="\\" but I

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 4:59 PM Andres Freund wrote: > Thanks, that's super helpful. Glad I could help. > I got a bit confused here - you seemed to have switched session 1 and 2 > around? Doesn't seem to matter much though, I was able to reproduce this. Yeah, I switched the session numbers becau

Re: control max length of parameter values logged

2020-04-01 Thread Alexey Bashtanov
Hi, The privilege argument seems irrelevant to me. We already decided that the plan is (a) SUSET for non-error statement logging purposes and (b) USERSET for logging caused by errors, and that would have to apply to length limits as well as enable/disable ability. Otherwise a user could pretty

  1   2   >