Re: making update/delete of inheritance trees scale better

2020-05-13 Thread Amit Langote
On Wed, May 13, 2020 at 8:52 AM David Rowley wrote: > On Wed, 13 May 2020 at 00:54, Ashutosh Bapat > wrote: > > > > On Mon, May 11, 2020 at 8:11 PM Amit Langote > > wrote: > > > > Per row overhead would be incurred for every row whereas the plan time > > > > overhead is one-time or in case of a

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/07 13:47, Fujii Masao wrote: On 2020/05/03 1:59, Tomas Vondra wrote: On Sat, May 02, 2020 at 12:55:00PM +0200, Tomas Vondra wrote: On Sat, May 02, 2020 at 03:56:07PM +0900, Fujii Masao wrote: ... Another thing I found is; pgstat_send_slru() should be called also by other pro

Re: No core file generated after PostgresNode->start

2020-05-13 Thread Michael Paquier
On Tue, May 12, 2020 at 04:15:26PM -0400, Robert Haas wrote: > On Mon, May 11, 2020 at 10:48 PM Tom Lane wrote: >> I have a standing note to check the permissions on /cores after any macOS >> upgrade, because every so often Apple decides that that directory ought to >> be read-only. > > Thanks, t

Re: gcov coverage data not full with immediate stop

2020-05-13 Thread Michael Paquier
On Mon, May 11, 2020 at 05:56:33PM +0530, Ashutosh Bapat wrote: > What happens if a coverage tool other than gcov is used? From that > perspective, it's better to perform a clean shutdown in the TAP tests > instead of immediate if that's possible. Nope, as that's the fastest path we have to shut d

Re: Event trigger code comment duplication

2020-05-13 Thread Michael Paquier
On Tue, May 12, 2020 at 10:26:46PM -0700, David G. Johnston wrote: > On Tuesday, May 12, 2020, Michael Paquier wrote: >> Even with all that, I am not sure that we need to complicate further >> what we have here. An empty currentEventTriggerState gets checks in >> three places, and each one of the

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Kyotaro Horiguchi
At Tue, 12 May 2020 17:12:51 -0400, Tom Lane wrote in > Alvaro Herrera writes: > > Fixed one straggler in contrib, and while testing it I realized why > > ccache doesn't pay attention to the changes I was doing in the file: > > ccache compares the *preprocessed* version of the file and only if t

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 4:13 PM Kyotaro Horiguchi wrote: > At Tue, 12 May 2020 17:12:51 -0400, Tom Lane wrote in > > Alvaro Herrera writes: > > > Fixed one straggler in contrib, and while testing it I realized why > > > ccache doesn't pay attention to the changes I was doing in the file: > > >

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: On 2020/05/07 13:47, Fujii Masao wrote: On 2020/05/03 1:59, Tomas Vondra wrote: On Sat, May 02, 2020 at 12:55:00PM +0200, Tomas Vondra wrote: On Sat, May 02, 2020 at 03:56:07PM +0900, Fujii Masao wrote: ... Another thing I f

Re: A patch for get origin from commit_ts.

2020-05-13 Thread movead...@highgo.ca
>I have not thought about this matter, but it seems to me that you >should add this patch to the upcoming commit fest for evaluation: >https://commitfest.postgresql.org/28/ Thanks. I think about it more detailed, and find it's better to show the 'roident' other than 'roname'. Because an old 'roi

Re: Index Skip Scan

2020-05-13 Thread Dilip Kumar
On Mon, May 11, 2020 at 4:55 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Mon, May 11, 2020 at 04:04:00PM +0530, Dilip Kumar wrote: > > > > > > +static inline bool > > > > +_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key, > > > > + Buffer buf, ScanDirection dir) > > > > +{

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 16:17:50 +0800, Andy Fan wrote in > On Wed, May 13, 2020 at 4:13 PM Kyotaro Horiguchi > wrote: > > > jsonpath_gram.c:1026:16: warning: this statement may fall through > > [-Wimplicit-fallthrough=] > > > if (*++yyp != '\\') > > > ^ > > > jsonpath_

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Amit Kapila
On Wed, May 13, 2020 at 11:35 AM Dilip Kumar wrote: > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > v20-0003-Extend-the-output-plugin-API-with-stream-methods > > > > 1. > > +static void > >

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Julien Rouhaud
On Wed, May 13, 2020 at 12:16 PM Kyotaro Horiguchi wrote: > > At Wed, 13 May 2020 16:17:50 +0800, Andy Fan wrote > in > > On Wed, May 13, 2020 at 4:13 PM Kyotaro Horiguchi > > wrote: > > > > jsonpath_gram.c:1026:16: warning: this statement may fall through > > > [-Wimplicit-fallthrough=] > > >

Re: refactoring basebackup.c

2020-05-13 Thread Sumanta Mukherjee
Hi Suraj, Two points I wanted to mention. 1. The max rate at which the transfer is happening when the tar size is 128 Kb is at most .48 GB/sec. Is there a possibility to understand what is the buffer size which is being used. That could help us explain some part of the puzzle. 2.

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Ashutosh Bapat
On Fri, May 8, 2020 at 7:27 AM Andy Fan wrote: >> +else if (inner_is_onerow) >> +{ >> +/* Since rows in innerrel can't be duplicated AND if >> innerrel is onerow, >> + * the join result will be onerow also as well. Note: >> onerow implies >>

pgstat_read_statsfiles() and reset timestamp

2020-05-13 Thread Fujii Masao
Hi, pgstat_read_statsfiles() sets each stat_reset_timestamp to the current timestamp, at the beginning of the function, just in case we fail to load the existing statsfile. This code is originally introduced by commit 4c468b37a2. But commit ad1b5c842b changed pgstat_read_statsfiles() so that the

Re: Inefficiency in SLRU stats collection

2020-05-13 Thread Fujii Masao
On 2020/05/13 11:26, Kyotaro Horiguchi wrote: At Tue, 12 May 2020 15:50:35 -0400, Tom Lane wrote in I happened to come across this code added by 28cac71bd: static PgStat_MsgSLRU * slru_entry(SlruCtl ctl) { intidx = pgstat_slru_index(ctl->shared->lwlock_tranche_name); Asse

Add explanations which are influenced by track_io_timing

2020-05-13 Thread Atsushi Torikoshi
Hi, When track_io_timing is on, I/O timing information is displayed in pg_stat_database, in the output of EXPLAIN when the BUFFERS option is used, and in pg_stat_statements as documented in [1]. This is also described in the manual for pg_stat_statements [2], however, manuals for pg_stat_database

Re: making update/delete of inheritance trees scale better

2020-05-13 Thread Amit Langote
On Tue, May 12, 2020 at 10:57 PM Tom Lane wrote: > Amit Langote writes: > > On Tue, May 12, 2020 at 5:25 AM Robert Haas wrote: > >> Ah, that makes sense. If we can invent dummy columns on the parent > >> rel, then most of what I was worrying about no longer seems very > >> worrying. > > > IIUC,

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: On 2020/05/07 13:47, Fujii Masao wrote: On 2020/05/03 1:59, Tomas Vondra wrote: On Sat, May 02, 2020 at 12:55:00PM +0200, Tomas Vondra wrote: On Sat, May 02, 2020 at 03:56:07PM +0900,

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-05-13 Thread Amit Kapila
On Thu, May 7, 2020 at 6:51 PM Tom Lane wrote: > > Amit Kapila writes: > > On Thu, May 7, 2020 at 11:57 AM Tom Lane wrote: > >> Monday is a back-branch release wrap day. > > > How can I get the information about release wrap day? The minor > > release dates are mentioned on the website [1], but

Re: No core file generated after PostgresNode->start

2020-05-13 Thread Tom Lane
Michael Paquier writes: > On Tue, May 12, 2020 at 04:15:26PM -0400, Robert Haas wrote: >> On Mon, May 11, 2020 at 10:48 PM Tom Lane wrote: >>> I have a standing note to check the permissions on /cores after any macOS >>> upgrade, because every so often Apple decides that that directory ought to >

Re: COPY, lock release and MVCC

2020-05-13 Thread Amit Kapila
On Wed, May 13, 2020 at 1:20 AM Laurenz Albe wrote: > > On Tue, 2020-05-12 at 11:50 -0400, Tom Lane wrote: > > Laurenz Albe writes: > > > On Mon, 2020-05-11 at 15:43 -0400, Robert Haas wrote: > > > > On Fri, May 8, 2020 at 4:58 AM Laurenz Albe > > > > wrote: > > > > > I happened to notice that

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Tom Lane
Andy Fan writes: >> FWIW, I got a warning for jsonpath_gram.c. Ugh. Confirmed here on Fedora 30 (bison 3.0.5). > I just found this just serval minutes ago. Upgrading your bison to the > latest version (3.6) is ok. I'd like we have a better way to share this > knowledge through. I spend ~30 mi

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-05-13 Thread Tom Lane
Amit Kapila writes: > Now that branches are tagged, I would like to commit and backpatch > this patch tomorrow unless there are any more comments/objections. The "quiet period" is over as soon as the tags appear in git. regards, tom lane

Re: refactoring basebackup.c

2020-05-13 Thread Robert Haas
On Wed, May 13, 2020 at 12:01 AM Suraj Kharage < suraj.khar...@enterprisedb.com> wrote: > 8kb 32kb (default value) 128kB 1024kB > Without refactor patch real 10m22.718s > user 1m23.629s > sys 8m51.410s real 8m36.245s > user 1m8.471s > sys 7m21.520s real 6m54.299s > user 0m55.690s > sys 5m46.502s r

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Fujii Masao writes: > On 2020/05/13 17:21, Tomas Vondra wrote: >> On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: >>> Also I found another minor issue; SLRUStats has not been initialized to 0 >>> and which could update the counters unexpectedly. Attached patch fixes >>> this issue. >

Re: Add explanations which are influenced by track_io_timing

2020-05-13 Thread Fujii Masao
On 2020/05/13 21:54, Atsushi Torikoshi wrote: Hi, When track_io_timing is on, I/O timing information is displayed in pg_stat_database, in the output of EXPLAIN when the BUFFERS option is used, and in pg_stat_statements as documented in [1]. This is also described in the manual for pg_stat_sta

Ideas about moving live rows to the top of the table

2020-05-13 Thread Антон Пацев
I read about the utility – https://github.com/dataegret/pgcompacttable . It is more careful about resources, because it works on slightly different principles. The main point of pgcompacttable is that it moves all live rows to the beginning of the table with updates in the table. And then it starts

Re: Inefficiency in SLRU stats collection

2020-05-13 Thread Tom Lane
Kyotaro Horiguchi writes: > AFAICS it is right and the change suggested looks reasonable to me. > One arguable point might be whether it is right that SlruData holds > pgstats internal index from the standpoint of modularity. (It is one > of the reasons I didn't propose a patch for that..) Yeah,

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/13 23:26, Tom Lane wrote: Fujii Masao writes: On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: Also I found another minor issue; SLRUStats has not been initialized to 0 and which could update the counters unexpectedly. Attache

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-13 Thread Jehan-Guillaume de Rorthais
(too bad the history has been removed to keep context) On Fri, 8 May 2020 15:02:26 +0500 godjan • wrote: > I got it, thank you. > Can you recommend what to use to determine which quorum standby should be > promoted in such case? We planned to use pg_last_wal_receive_lsn() to > determine which

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 10:26:39AM -0400, Tom Lane wrote: Fujii Masao writes: On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: Also I found another minor issue; SLRUStats has not been initialized to 0 and which could update the counters une

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Tomas Vondra writes: > On Wed, May 13, 2020 at 10:26:39AM -0400, Tom Lane wrote: >> Why is that necessary? A static variable is defined by C to start off >> as zeroes. > But is it a static variable? It's not declared as 'static' but maybe we > can assume it inits to zeroes anyway? I see we do th

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-13 Thread Jehan-Guillaume de Rorthais
On Mon, 11 May 2020 15:54:02 +0900 Michael Paquier wrote: [...] > There are several HA solutions floating around in the community, and I > got to wonder as well if some of them don't just scan the local > pg_wal/ of each standby in this case, even if that's more simple to > let the nodes start and

Re: PG 13 release notes, first draft

2020-05-13 Thread Bruce Momjian
On Wed, May 13, 2020 at 11:56:33AM +0900, Kyotaro Horiguchi wrote: > > > > Allow skipping of WAL for new tables and indexes if wal_level is > > 'minimal' (Kyotaro Horiguchi) > > > > Relations larger than wal_skip_threshold will have their files > > fsync'ed rather than writing the

Re: PG 13 release notes, first draft

2020-05-13 Thread Bruce Momjian
On Wed, May 13, 2020 at 07:18:38AM +0200, Fabien COELHO wrote: > > Hello Bruce, > > > > * e1ff780485 > > > > I was told in this email thread to not include that one. > > Ok. > > > > * 34a0a81bfb > > > > We already have: > > > > Reformat tables containing function information for better

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Tom Lane wrote: > Andy Fan writes: > >> FWIW, I got a warning for jsonpath_gram.c. > > Ugh. Confirmed here on Fedora 30 (bison 3.0.5). Ugh. > > I just found this just serval minutes ago. Upgrading your bison to the > > latest version (3.6) is ok. I'd like we have a better way

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 11:01:47AM -0400, Tom Lane wrote: Tomas Vondra writes: On Wed, May 13, 2020 at 10:26:39AM -0400, Tom Lane wrote: Why is that necessary? A static variable is defined by C to start off as zeroes. But is it a static variable? It's not declared as 'static' but maybe we

Re: SLRU statistics

2020-05-13 Thread Tomas Vondra
On Wed, May 13, 2020 at 11:46:30PM +0900, Fujii Masao wrote: On 2020/05/13 23:26, Tom Lane wrote: Fujii Masao writes: On 2020/05/13 17:21, Tomas Vondra wrote: On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: Also I found another minor issue; SLRUStats has not been initialized t

Re: Do I understand commit timestamps correctly?

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Chapman Flack wrote: > Getting around to trying it out, simply changing the setting in > postgresql.conf before starting the server does not seem sufficient: > once it comes online, it has track_commit_timestamp on, but has not > populated the cache from transactions it applied dur

Re: Is it possible to find out write_lsn on standby?

2020-05-13 Thread Jehan-Guillaume de Rorthais
Hello, On Fri, 8 May 2020 18:25:02 +0500 godjan • wrote: > On primary I can execute ’SELECT write_lsn FROM pg_stat_replication;’ and get > write_lsn of standby. I didn’t find function like "pg_last_write_lsn()” to > get write_lsn on standby. Is it possible? No, there's no admin function exposin

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Tom Lane
Alvaro Herrera writes: >> I fear that is going to mean that we revert this patch. > Or we can filter-out the -Wimplicit-fallthrough, or change to level 3, > for bison-emitted files. Let's just go to level 3 overall (and revert the changes you made for level 4 compliance --- they're more likely t

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Tomas Vondra writes: > I think it counts as a variable with "static storage duration" per 6.7.8 > (para 10), see [1]. I wasn't aware of this either, but it probably means > the memset is unnecessary. > Also, it seems a bit strange/confusing to handle this differently from > BgWriterStats. And that

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-13 Thread Dilip Kumar
On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > On Wed, May 13, 2020 at 11:35 AM Dilip Kumar wrote: > > > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > > > > > v20-0003-Extend-the-output-plugin-API-with-stream-methods > > >

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat wrote: > On Fri, May 8, 2020 at 7:27 AM Andy Fan wrote: > > >> +else if (inner_is_onerow) > >> +{ > >> +/* Since rows in innerrel can't be duplicated AND if > >> innerrel is onerow, > >> + * th

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
Hi Ashutosh: All your suggestions are followed except the UNION ALL one. I replied the reason below. For the suggestions about partitioned table, looks lot of cases to handle, so I summarize/enrich your idea in README and email thread, we can continue to talk about that. >> + >> +foreach

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Tom Lane wrote: > Alvaro Herrera writes: > >> I fear that is going to mean that we revert this patch. > > > Or we can filter-out the -Wimplicit-fallthrough, or change to level 3, > > for bison-emitted files. > > Let's just go to level 3 overall (and revert the changes you made f

ldap tls test fails in some environments

2020-05-13 Thread Christoph Berg
> I just added the extra tests to the postgresql-13 package and got > this: Some other problem emerged here in the ldap test: 17:28:58 17:28:58 # Failed test 'StartTLS' 17:28:58 # at t/001_auth.pl line 169. 17:28:58 # got: '2' 17:28:58 # expected: '0' 17:28:58 17:28:58 # Failed

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/14 0:38, Tom Lane wrote: Tomas Vondra writes: I think it counts as a variable with "static storage duration" per 6.7.8 (para 10), see [1]. I wasn't aware of this either, but it probably means the memset is unnecessary. Also, it seems a bit strange/confusing to handle this differen

Re: SLRU statistics

2020-05-13 Thread Alvaro Herrera
On 2020-May-14, Fujii Masao wrote: > So I tried the similar test again and found that postmaster seems to be > able to increment the counters unless I'm missing something. > For example, > > frame #2: 0x00010d93845f > postgres`pgstat_count_slru_page_zeroed(ctl=0x00010de27320) at > p

Re: pg13: xlogreader API adjust

2020-05-13 Thread Alvaro Herrera
Pushed. Thanks for the help! -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/14 1:14, Alvaro Herrera wrote: On 2020-May-14, Fujii Masao wrote: So I tried the similar test again and found that postmaster seems to be able to increment the counters unless I'm missing something. For example, frame #2: 0x00010d93845f postgres`pgstat_count_slru_page_z

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Fujii Masao writes: > But since the counter that postmaster incremented is propagated to > child processes via fork, it should be zeroed at postmaster or the > beginning of child process? Otherwise that counter always starts > with non-zero in child process. Yes, if the postmaster is incrementing

Re: Is it useful to record whether plans are generic or custom?

2020-05-13 Thread legrand legrand
Hello, yes this can be usefull, under the condition of differentiating all the counters for a queryid using a generic plan and the one using a custom one. For me one way to do that is adding a generic_plan column to pg_stat_statements key, someting like: - userid, - dbid, - queryid, - generic_pl

Re: SLRU statistics

2020-05-13 Thread Tom Lane
I wrote: >>> (IIRC only the Async module is doing that.) > Hm, maybe we can fix that. Yeah, it's quite easy to make async.c postpone its first write to the async SLRU. This seems like a win all around, because many installations don't use NOTIFY and so will never need to do that work at all. In

Re: MultiXact\SLRU buffers configuration

2020-05-13 Thread Andrey M. Borodin
> 11 мая 2020 г., в 16:17, Andrey M. Borodin написал(а): > > I've went ahead and created 3 patches: > 1. Configurable SLRU buffer sizes for MultiXacOffsets and MultiXactMembers > 2. Reduce locking level to shared on read of MultiXactId members > 3. Configurable cache size I'm looking more at

Re: Parallel copy

2020-05-13 Thread Robert Haas
On Tue, May 12, 2020 at 1:01 AM Amit Kapila wrote: > I don't understand why we need to do something special for combo CIDs > if they are not generated during this operation? Hmm. Well I guess if they're not being generated then we don't need to do anything about them, but I still think we should

Re: tablespace_map code cleanup

2020-05-13 Thread Robert Haas
On Tue, May 12, 2020 at 10:26 PM Amit Kapila wrote: > I was trying to say that tablespace listing will happen under > PROGRESS_BASEBACKUP_PHASE_WAIT_CHECKPOINT phase which could be a > problem if it is a costly operation but as you said it is pretty cheap > so I think we don't need to bother about

Re: new heapcheck contrib module

2020-05-13 Thread Robert Haas
On Tue, May 12, 2020 at 11:06 PM Peter Geoghegan wrote: > try/catch seems like the way to do it. Not all amcheck errors come > from amcheck -- some are things that the backend code does, that are > known to appear in amcheck from time to time. I'm thinking in > particular of the > table_index_buil

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Alvaro Herrera wrote: > On 2020-May-13, Tom Lane wrote: > > Let's just go to level 3 overall (and revert the changes you made for > > level 4 compliance --- they're more likely to cause back-patching > > pain than do anything useful). > > Ok. And done. -- Álvaro Herrera

Re: JSON output from psql

2020-05-13 Thread Robert Haas
On Mon, May 11, 2020 at 4:42 PM Gurjeet Singh wrote: > That's a good point! It might still be desirable, perhaps for performance > trade-off of JSON conversion on the client-side instead of on the server-side. If there's a performance problem with the server's code here, we should probably try t

Re: SLRU statistics

2020-05-13 Thread Ranier Vilela
Em qua., 13 de mai. de 2020 às 11:46, Fujii Masao < masao.fu...@oss.nttdata.com> escreveu: > > > On 2020/05/13 23:26, Tom Lane wrote: > > Fujii Masao writes: > >> On 2020/05/13 17:21, Tomas Vondra wrote: > >>> On Wed, May 13, 2020 at 04:10:30PM +0900, Fujii Masao wrote: > Also I found anothe

Re: JSON output from psql

2020-05-13 Thread Gurjeet Singh
On Wed, May 13, 2020 at 12:50 PM Robert Haas wrote: > > On Mon, May 11, 2020 at 4:42 PM Gurjeet Singh wrote: > > That's a good point! It might still be desirable, perhaps for performance > > trade-off of JSON conversion on the client-side instead of on the > > server-side. > > If there's a perf

Re: JSON output from psql

2020-05-13 Thread Gurjeet Singh
On Wed, May 13, 2020 at 1:14 PM Gurjeet Singh wrote: > > Arguably, delivering JSON (with its repeating attribute names in every > element of the array, dquotes and commas) is more network intensive > than converting the resultset to JSON on network side. s/network side/client side/ Best regards,

Re: COPY, lock release and MVCC

2020-05-13 Thread Laurenz Albe
On Wed, 2020-05-13 at 19:29 +0530, Amit Kapila wrote: > > > > > On Fri, May 8, 2020 at 4:58 AM Laurenz Albe > > > > > wrote: > > > > > > I happened to notice that COPY TO releases the ACCESS SHARE lock > > > > > > on the table right when the command ends rather than holding it > > > > > > until t

Re: JSON output from psql

2020-05-13 Thread Chapman Flack
On 05/13/20 16:16, Gurjeet Singh wrote: > On Wed, May 13, 2020 at 1:14 PM Gurjeet Singh wrote: >> >> Arguably, delivering JSON (with its repeating attribute names in every >> element of the array, dquotes and commas) is more network intensive >> than converting the resultset to JSON on [client] si

document? Re: Do I understand commit timestamps correctly?

2020-05-13 Thread Chapman Flack
On 03/23/18 11:29, Chapman Flack wrote: > Can somebody confirm or correct what I (think I)'ve gleaned from > the code? > ... > - The extra machinery turned on by track_commit_timestamp maintains > a cache of recent ones so they can be efficiently queried from SQL > in normal operation. I guess

Re: Our naming of wait events is a disaster.

2020-05-13 Thread Tom Lane
I looked through the naming situation for LWLocks, and SLRUs which turn out to be intimately connected to that, because most of the dubious LWLock names we're currently exposing actually are derived from SLRUs. Here's some ideas (not quite a full proposal yet) for changing that. Note that because

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 12:22 PM Robert Haas wrote: > I think this is a good summary of the problems in this area. On the > one hand, I think it's hideous that we sanity check user input to > death, but blindly trust the bytes on disk to the point of seg > faulting if they're wrong. The idea that

Re: new heapcheck contrib module

2020-05-13 Thread Alvaro Herrera
On 2020-May-12, Peter Geoghegan wrote: > > The point is that when checking the table for corruption I avoid > > calling anything that might assert (or segfault, or whatever). > > I don't think that you can expect to avoid assertion failures in > general. Hmm. I think we should (try to?) write c

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread David Rowley
On Thu, 14 May 2020 at 03:48, Andy Fan wrote: > On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat > wrote: >> My impression about the one row stuff, is that there is too much >> special casing around it. We should somehow structure the UniqueKey >> data so that one row unique keys come naturally ra

Re: pgstat_read_statsfiles() and reset timestamp

2020-05-13 Thread Tom Lane
Fujii Masao writes: > pgstat_read_statsfiles() sets each stat_reset_timestamp to > the current timestamp, at the beginning of the function, > just in case we fail to load the existing statsfile. This code is > originally introduced by commit 4c468b37a2. > But commit ad1b5c842b changed pgstat_read

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 3:10 PM Alvaro Herrera wrote: > Hmm. I think we should (try to?) write code that avoids all crashes > with production builds, but not extend that to assertion failures. Assertions are only a problem at all because Mark would like to write tests that involve a selection of

Re: Index Skip Scan

2020-05-13 Thread Peter Geoghegan
On Mon, Jan 20, 2020 at 5:05 PM Peter Geoghegan wrote: > You can add another assertion that calls a new utility function in > bufmgr.c. That can use the same logic as this existing assertion in > FlushOneBuffer(): > > Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr))); > > We haven't n

Re: making update/delete of inheritance trees scale better

2020-05-13 Thread David Rowley
On Wed, 13 May 2020 at 19:02, Amit Langote wrote: > > As for which ResultRelInfos to initialize, couldn't we just have the > > planner generate an OidList of all the ones that we could need. > > Basically, all the non-pruned partitions. > > Why would replacing list of RT indexes by OIDs be better?

Re: gcov coverage data not full with immediate stop

2020-05-13 Thread Peter Geoghegan
On Tue, May 12, 2020 at 10:10 AM Tom Lane wrote: > > Can that easily be accommodated? > > There's no real lead time needed AFAICS: when we are ready to branch, > we can just do it. So sure, let's wait till the end of May to decide. > If things look bad then, we could reconsider again mid-June. G

Re: Concurrency bug in amcheck

2020-05-13 Thread Peter Geoghegan
On Mon, May 11, 2020 at 5:56 AM Alexander Korotkov wrote: > Thank you. I've worked a bit on comments and commit message. I would > appreciate you review. This looks good to me. > > I like your idea of making the primary consistent with the REDO > > routine on the master branch only. I wonder i

Re: pg13: xlogreader API adjust

2020-05-13 Thread Tom Lane
Alvaro Herrera writes: > Pushed. Thanks for the help! This seems to have fixed bowerbird. Were you expecting that? regards, tom lane

Re: new heapcheck contrib module

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Peter Geoghegan wrote: > On Wed, May 13, 2020 at 3:10 PM Alvaro Herrera > wrote: > > Hmm. I think we should (try to?) write code that avoids all crashes > > with production builds, but not extend that to assertion failures. > > Assertions are only a problem at all because Mark

Re: pg13: xlogreader API adjust

2020-05-13 Thread Alvaro Herrera
On 2020-May-13, Tom Lane wrote: > Alvaro Herrera writes: > > Pushed. Thanks for the help! > > This seems to have fixed bowerbird. Were you expecting that? Hm, not really. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 4:32 PM Alvaro Herrera wrote: > I think my initial approach for this would be to use a fuzzing tool that > generates data blocks semi-randomly, then uses them as Postgres data > pages somehow, and see what happens -- examine any resulting crashes and > make individual judge

Re: new heapcheck contrib module

2020-05-13 Thread Mark Dilger
> On May 13, 2020, at 3:29 PM, Peter Geoghegan wrote: > > On Wed, May 13, 2020 at 3:10 PM Alvaro Herrera > wrote: >> Hmm. I think we should (try to?) write code that avoids all crashes >> with production builds, but not extend that to assertion failures. > > Assertions are only a problem a

Re: new heapcheck contrib module

2020-05-13 Thread Peter Geoghegan
On Wed, May 13, 2020 at 5:18 PM Mark Dilger wrote: > I am not removing any assertions. I do not propose to remove any assertions. > When I talk about "hardening against assertions", that is not in any way a > proposal to remove assertions from the code. I'm sorry if I seemed to suggest that yo

Re: PG 13 release notes, first draft

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 11:15:18 -0400, Bruce Momjian wrote in > On Wed, May 13, 2020 at 11:56:33AM +0900, Kyotaro Horiguchi wrote: > > > > > > Allow skipping of WAL for new tables and indexes if wal_level is > > > 'minimal' (Kyotaro Horiguchi) > > > > > > Relations larger than wal_skip_thres

Re: pg13: xlogreader API adjust

2020-05-13 Thread Bossart, Nathan
I think I've discovered a problem with 850196b6. The following steps can be used to trigger a segfault: # wal_level = logical psql postgres -c "create database testdb;" psql testdb -c "select pg_create_logical_replication_slot('slot', 'test_decoding');" psql "dbna

Re: JSON output from psql

2020-05-13 Thread Gurjeet Singh
On Wed, May 13, 2020 at 2:01 PM Chapman Flack wrote: > > Seems like a lot of work just to get json-shaped query results from psql, +1. If we look at the amount of work needed for the hybrid approach you describe, compared to running CSV result through something like csv2json, there's a 100% chanc

Re: MultiXact\SLRU buffers configuration

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 23:08:37 +0500, "Andrey M. Borodin" wrote in > > > > 11 мая 2020 г., в 16:17, Andrey M. Borodin > > написал(а): > > > > I've went ahead and created 3 patches: > > 1. Configurable SLRU buffer sizes for MultiXacOffsets and MultiXactMembers > > 2. Reduce locking level to sh

Re: Our naming of wait events is a disaster.

2020-05-13 Thread Tom Lane
The attached patch doesn't actually change any LWLock names, but it is a useful start on that project. What it does is to get rid of the current scheme of dynamically registering the names of built-in LWLocks in favor of having a constant array of those names. It's completely foolish to expend pr

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-13 Thread godjan •
-> Why do you kill -9 your standby? Hi, it’s Jepsen test for our HA solution. It checks that we don’t lose data in such situation. So, now we update logic as Michael said. All ha alive standbys now waiting for replaying all WAL that they have and after we use pg_last_replay_lsn() to choose whi

Re: SLRU statistics

2020-05-13 Thread Fujii Masao
On 2020/05/14 2:44, Tom Lane wrote: I wrote: (IIRC only the Async module is doing that.) Hm, maybe we can fix that. Yeah, it's quite easy to make async.c postpone its first write to the async SLRU. This seems like a win all around, because many installations don't use NOTIFY and so will

Re: refactoring basebackup.c

2020-05-13 Thread Suraj Kharage
Hi, On Wed, May 13, 2020 at 7:49 PM Robert Haas wrote: > > So the patch came out slightly faster at 8kB and slightly slower in the > other tests. That's kinda strange. I wonder if it's just noise. How much do > the results vary run to run? > It is not varying much except for 8kB run. Please see

Re: SLRU statistics

2020-05-13 Thread Tom Lane
Fujii Masao writes: > On 2020/05/14 2:44, Tom Lane wrote: >> I got through check-world with the assertion shown that we are not >> counting any SLRU operations in the postmaster. Don't know if we >> want to commit that or not --- any thoughts? > +1 to add this assertion because basically it's no

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-13 Thread Jonathan S. Katz
On 5/10/20 2:03 PM, Jonathan S. Katz wrote: > On 5/10/20 12:27 PM, Tom Lane wrote: >> Just FTR, here's a complete patch for this. > > Cool. I'll play around with it tonight once I clear out release work. > Per upthread reference, I believe you've become a CSS maven yourself. Time slipped a bit (

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Thu, May 14, 2020 at 6:20 AM David Rowley wrote: > On Thu, 14 May 2020 at 03:48, Andy Fan wrote: > > On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat < > ashutosh.bapat@gmail.com> wrote: > >> My impression about the one row stuff, is that there is too much > >> special casing around it. We

Re: PG 13 release notes, first draft

2020-05-13 Thread Bruce Momjian
On Thu, May 14, 2020 at 09:51:41AM +0900, Kyotaro Horiguchi wrote: > At Wed, 13 May 2020 11:15:18 -0400, Bruce Momjian wrote in > > On Wed, May 13, 2020 at 11:56:33AM +0900, Kyotaro Horiguchi wrote: > It is just an more accurate (not an detailed) version of the > previously proposed description.

Re: tablespace_map code cleanup

2020-05-13 Thread Kyotaro Horiguchi
At Wed, 13 May 2020 15:10:30 -0400, Robert Haas wrote in > On Tue, May 12, 2020 at 10:26 PM Amit Kapila wrote: > > I was trying to say that tablespace listing will happen under > > PROGRESS_BASEBACKUP_PHASE_WAIT_CHECKPOINT phase which could be a > > problem if it is a costly operation but as yo

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 11:48 PM Andy Fan wrote: > >> My impression about the one row stuff, is that there is too much >> special casing around it. We should somehow structure the UniqueKey >> data so that one row unique keys come naturally rather than special >> cased. E.g every column in such a

Re: Another modest proposal for docs formatting: catalog descriptions

2020-05-13 Thread Tom Lane
"Jonathan S. Katz" writes: > Time slipped a bit (sorry!), but while prepping for the release I > reviewed this. Visually, it looks WAY better. The code checks out too. I > think any tweaks would be primarily around personal preference on the > UI, so it may be better just to commit and get it out

Re: ldap tls test fails in some environments

2020-05-13 Thread Thomas Munro
On Thu, May 14, 2020 at 4:05 AM Christoph Berg wrote: > Some other problem emerged here in the ldap test: Hi Christoph, > 17:28:59 Data directory: > /<>/build/src/test/ldap/tmp_check/t_001_auth_node_data/pgdata I know nothing about Debian package building so I could be missing something about

  1   2   >