Re: fdatasync performance problem with large number of DB files

2021-03-19 Thread Thomas Munro
Thanks Justin and David. Replies to two emails inline: On Sat, Mar 20, 2021 at 12:01 AM Justin Pryzby wrote: > On Fri, Mar 19, 2021 at 06:28:46PM +1300, Thomas Munro wrote: > > +++ b/doc/src/sgml/config.sgml > > > +PostgreSQL will recursively open and > > fsync > > +all files in

shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Andres Freund
Hi, I am working on Kyotaro Horiguchi's shared memory stats patch [1] with the goal of getting it into a shape that I'd be happy to commit. That thread is quite long and most are probably skipping over new messages in it. There are two high-level design decisions / questions that I think warrant

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Alvaro Herrera
On 2021-Mar-19, Robert Haas wrote: > On Fri, Mar 19, 2021 at 6:22 PM Alvaro Herrera > wrote: > > (At least, for binary upgrade surely you must make sure to apply the > > correct setting regardless of defaults on either system). > > It's not critical from a system integrity point of view; the c

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-19 Thread Bruce Momjian
On Sat, Mar 20, 2021 at 01:03:16AM +0100, Hannu Krosing wrote: > It would be really convenient if user-visible serialisations of the query id > had something that identifies the computation method. > > maybe prefix 'N' for internal, 'S' for pg_stat_statements etc. > > This would immediately show

Re: [HACKERS] Custom compression methods

2021-03-19 Thread David Steele
On 3/19/21 8:00 PM, Andres Freund wrote: On 2021-03-19 15:44:34 -0400, Robert Haas wrote: I committed the core patch (0003) with a bit more editing. Let's see what the buildfarm thinks. Congrats Dilip, Robert, All. The slow toast compression has been a significant issue for a long time. Yes

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Andres Freund
Hi, On 2021-03-20 01:16:31 +0100, Hannu Krosing wrote: > > But now we could instead schedule stats to be removed at commit > time. That's not trivial of course, as we'd need to handle cases where > the commit fails after the commit record, but before processing the > dropped stats. > > We likely

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Mar-19, Robert Haas wrote: >> Well, I really do hope that some day in the bright future, pglz will >> no longer be the thing we're shipping as the postgresql.conf default. >> So we'd just be postponing the noise until then. I think we need a >> better idea than tha

Re: fdatasync performance problem with large number of DB files

2021-03-19 Thread David Steele
On 3/19/21 7:16 PM, Thomas Munro wrote: Thanks Justin and David. Replies to two emails inline: Fair point. Here's what I went with: When set to fsync, which is the default, PostgreSQL will recursively open and synchronize all files in the data directory before crash

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Hannu Krosing
On Sat, Mar 20, 2021 at 1:21 AM Andres Freund wrote: > > Hi, > > On 2021-03-20 01:16:31 +0100, Hannu Krosing wrote: > > > But now we could instead schedule stats to be removed at commit > > time. That's not trivial of course, as we'd need to handle cases where > > the commit fails after the commit

Re: [PATCH] Identify LWLocks in tracepoints

2021-03-19 Thread Craig Ringer
On Sat, 20 Mar 2021, 04:21 Peter Eisentraut, < peter.eisentr...@enterprisedb.com> wrote: > > On 18.03.21 07:34, Craig Ringer wrote: > > The main reason I didn't want to add more tracepoints than were strictly > > necessary is that Pg doesn't enable the systemtap semaphores feature, so > > right no

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-19 Thread Ajin Cherian
On Sat, Mar 20, 2021 at 1:35 AM Amit Kapila wrote: > On Fri, Mar 19, 2021 at 5:03 AM Ajin Cherian wrote: > > > > Missed the patch - 0001, resending. > > > > > @@ -538,10 +550,21 @@ CreateDecodingContext(XLogRecPtr start_lsn, > .. > + /* Set two_phase_at LSN only if it hasn't already been set. */

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Andrew Dunstan
On 3/19/21 8:25 PM, Tom Lane wrote: > Alvaro Herrera writes: >> On 2021-Mar-19, Robert Haas wrote: >>> Well, I really do hope that some day in the bright future, pglz will >>> no longer be the thing we're shipping as the postgresql.conf default. >>> So we'd just be postponing the noise until the

RE: invalid data in file backup_label problem on windows

2021-03-19 Thread wangsh.f...@fujitsu.com
David Steele wrote: > It's not clear to me what text editors have to do with this? Are you > editing the file manually? When I execute SELECT * FROM pg_stop_backup(false, true) in psql. The results will be shown like: lsn| labelfile

Re: Logical Replication vs. 2PC

2021-03-19 Thread Amit Kapila
On Fri, Mar 19, 2021 at 9:22 PM Markus Wanner wrote: > > On 18.03.21 10:45, Amit Kapila wrote: > > While reviewing/testing subscriber-side work for $SUBJECT [1], I > > noticed a problem that seems to need a broader discussion, so started > > this thread. We can get prepare for the same GID more th

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-19 Thread Amit Kapila
On Sat, Mar 20, 2021 at 7:07 AM Ajin Cherian wrote: > > On Sat, Mar 20, 2021 at 1:35 AM Amit Kapila wrote: >> >> On Fri, Mar 19, 2021 at 5:03 AM Ajin Cherian wrote: >> > >> > Missed the patch - 0001, resending. >> > >> >> >> @@ -538,10 +550,21 @@ CreateDecodingContext(XLogRecPtr start_lsn, >> ..

Re: Maintaining a list of pgindent commits for "git blame" to ignore

2021-03-19 Thread Peter Geoghegan
On Thu, Mar 18, 2021 at 5:07 PM Tom Lane wrote: > Doubt that it matters. The workflow would have to be "commit and push > the mechanical updates, then edit the tracking file, commit and push > that". You don't have the commit hash nailed down till you've pushed. Okay. I have made a personal TOD

recovery_init_sync_method=wal

2021-03-19 Thread Thomas Munro
Hello hackers, I'm starting a new thread and CF entry for the material for r15 from the earlier thread[1] that introduced the recovery_init_sync_method GUC for r14. I wrote a summary of this topic as I see it, while it's still fresh on my mind from working on commit 61752afb, starting from what p

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Robert Haas
On Fri, Mar 19, 2021 at 8:25 PM Tom Lane wrote: > Extrapolating from the way we've dealt with similar issues > in the past, I think the structure of pg_dump's output ought to be: > > 1. SET default_toast_compression = 'source system's value' > in among the existing passel of SETs at the top. Does

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Robert Haas
On Fri, Mar 19, 2021 at 6:38 PM Alvaro Herrera wrote: > I suggest to add comments to this effect, > perhaps as the attached (feel free to reword, I think mine is awkward.) It's not bad, although "the decompressed version of the full datum" might be a little better. I'd probably say instead: "This

pspg pager is finished

2021-03-19 Thread Pavel Stehule
Hi I finished work on pspg. https://github.com/okbob/pspg Now it has special features like rows or block selection by mouse, and export related data to file or to clipboard in csv or tsv or insert formats. Some basic features like sorting data per selected columns are possible too. I hope this

Re: pspg pager is finished

2021-03-19 Thread Julien Rouhaud
On Sat, Mar 20, 2021 at 04:34:30AM +0100, Pavel Stehule wrote: > Hi > > I finished work on pspg. > > https://github.com/okbob/pspg > > Now it has special features like rows or block selection by mouse, and > export related data to file or to clipboard in csv or tsv or insert > formats. Some basi

Re: Replication slot stats misgivings

2021-03-19 Thread Amit Kapila
On Sat, Mar 20, 2021 at 12:22 AM Andres Freund wrote: > > And then more generally about the feature: > - If a slot was used to stream out a large amount of changes (say an > initial data load), but then replication is interrupted before the > transaction is committed/aborted, stream_bytes will

Re: a verbose option for autovacuum

2021-03-19 Thread Masahiko Sawada
On Fri, Mar 19, 2021 at 3:14 PM Michael Paquier wrote: > > On Thu, Mar 18, 2021 at 11:30:46PM +0900, Masahiko Sawada wrote: > > Sorry, I attached the wrong version patch. So attached the right one. > > Thanks. I have been hacking aon that, and I think that we could do > more in terms of integrati

Re: pg_upgrade failing for 200+ million Large Objects

2021-03-19 Thread Jan Wieck
On 3/8/21 11:58 AM, Tom Lane wrote: The answer up to now has been "raise max_locks_per_transaction enough so you don't see the failure". Having now consumed a little more caffeine, I remember that that works in pg_upgrade scenarios too, since the user can fiddle with the target cluster's postgre

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-19 Thread Masahiro Ikeda
When only the stats collector exits by SIGQUIT, with the patch FreeWaitEventSet() is also skipped. Is this ok? Thanks, I fixed it. I'm still not sure if FreeWaitEventSet() is actually necessary in that exit case. Could you tell me why you thought FreeWaitEventSet() is necessary in the case?

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-19 Thread Thomas Munro
On Sat, Mar 6, 2021 at 12:10 PM Daniel Gustafsson wrote: > > On 3 Mar 2021, at 23:19, Thomas Munro wrote: > > That's why I release and reacquire that LWLock. But does that break some > > logic? > > One clear change to current behavior is naturally that a concurrent > TablespaceCreateDbspace can

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-19 Thread Ajin Cherian
On Sat, Mar 20, 2021 at 1:35 AM Amit Kapila wrote: > On Fri, Mar 19, 2021 at 5:03 AM Ajin Cherian wrote: > > > > Missed the patch - 0001, resending. > > > > I have made miscellaneous changes in the patch which includes > improving comments, error messages, and miscellaneous coding > improvements

Re: Replication slot stats misgivings

2021-03-19 Thread Amit Kapila
On Sat, Mar 20, 2021 at 9:25 AM Amit Kapila wrote: > > On Sat, Mar 20, 2021 at 12:22 AM Andres Freund wrote: > > > > And then more generally about the feature: > > - If a slot was used to stream out a large amount of changes (say an > > initial data load), but then replication is interrupted be

Re: pglz compression performance, take two

2021-03-19 Thread Justin Pryzby
On Fri, Mar 19, 2021 at 01:29:14PM -0700, Mark Dilger wrote: > Robert Haas just committed Dilip Kumar's LZ4 compression, > bbe0a81db69bd10bd166907c3701492a29aca294. > > Is this pglz compression patch still relevant? How does the LZ4 compression > compare on your hardware? I think it's still re

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-19 Thread Julien Rouhaud
On Fri, Mar 19, 2021 at 08:10:54PM -0400, Bruce Momjian wrote: > On Sat, Mar 20, 2021 at 01:03:16AM +0100, Hannu Krosing wrote: > > It would be really convenient if user-visible serialisations of the query id > > had something that identifies the computation method. > > > > maybe prefix 'N' for in

Re: Asynchronous Append on postgres_fdw nodes.

2021-03-19 Thread Etsuro Fujita
On Fri, Mar 19, 2021 at 9:57 PM Justin Pryzby wrote: > is_async_capable_path() should probably have a "break" for case T_ForeignPath. Good catch! Will fix. > little typos: > aready > sigle > givne > a event Lots of typos. :-( Will fix. Thank you for the review! Best regards, Etsuro Fujita

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-19 Thread Julien Rouhaud
On Fri, Mar 19, 2021 at 12:53:18PM -0400, Bruce Momjian wrote: > > Well, given we don't really want to support multiple query id types > being generated or displayed, the "error out" above should fix it. > > Let's do this --- tell extensions to error out if the query id is > already set, either

Re: pspg pager is finished

2021-03-19 Thread Pavel Stehule
so 20. 3. 2021 v 4:45 odesílatel Julien Rouhaud napsal: > On Sat, Mar 20, 2021 at 04:34:30AM +0100, Pavel Stehule wrote: > > Hi > > > > I finished work on pspg. > > > > https://github.com/okbob/pspg > > > > Now it has special features like rows or block selection by mouse, and > > export related

Re: a verbose option for autovacuum

2021-03-19 Thread Michael Paquier
On Sat, Mar 20, 2021 at 01:06:51PM +0900, Masahiko Sawada wrote: > It's not bad but it seems redundant a bit to me. We pass the idx in > spite of passing also Irel[idx] and &(vacrelstats->indstats[idx]). I > think your first idea that is done in v4 patch (saving index names at > the beginning of he

Re: pspg pager is finished

2021-03-19 Thread Vik Fearing
On 3/20/21 4:34 AM, Pavel Stehule wrote: > Hi > > I finished work on pspg. > > https://github.com/okbob/pspg > > Now it has special features like rows or block selection by mouse, and > export related data to file or to clipboard in csv or tsv or insert > formats. Some basic features like sortin

<    1   2