Re: WIP/PoC for parallel backup

On Mon, Mar 16, 2020 at 11:52 AM Asif Rehman wrote: > > > On Mon, Mar 16, 2020 at 11:08 AM Rajkumar Raghuwanshi < > rajkumar.raghuwan...@enterprisedb.com> wrote: > >> Thanks for the patches. >> >> I have verified reported issues with new patches, issues are fixed now. >> >> I got another observat

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Mon, Mar 16, 2020 at 9:43 AM Dilip Kumar wrote: > On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada > wrote: > > IsRelationExtensionLockHeld and IsPageLockHeld are used only when > > assertion is enabled. So how about making CheckAndSetLockHeld work > > only if USE_ASSERT_CHECKING to avoid overh

Re: [Proposal] Global temporary tables

Hi Wenjing, Please check the below scenario, where the Foreign table on GTT not showing records. postgres=# create extension postgres_fdw; CREATE EXTENSION postgres=# do $d$ begin execute $$create server fdw foreign data wrapper postgres_fdw options (host 'localhost',dbname 'postgres',

Re: WIP/PoC for parallel backup

On Mon, Mar 16, 2020 at 11:08 AM Rajkumar Raghuwanshi < rajkumar.raghuwan...@enterprisedb.com> wrote: > Thanks for the patches. > > I have verified reported issues with new patches, issues are fixed now. > > I got another observation where If a new slot name given without -C > option, it leads to

Re: error context for vacuum to include block number

On Thu, Mar 5, 2020 at 3:22 AM Justin Pryzby wrote: > > On Wed, Mar 04, 2020 at 04:21:06PM +0900, Masahiko Sawada wrote: > > Thank you for updating the patch. But we have two more places where we > > do fsm vacuum. > > Oops, thanks. > > I realized that vacuum_page is called not only from lazy_vacu

Re: WIP/PoC for parallel backup

Thanks for the patches. I have verified reported issues with new patches, issues are fixed now. I got another observation where If a new slot name given without -C option, it leads to server crash error. [edb@localhost bin]$ ./pg_basebackup -p 5432 -j 4 -D /tmp/bkp --slot test_bkp_slot pg_baseba

Re: backup manifests

One more suggestion, recent commit (1933ae62) has added the PostgreSQL home page to --help output. e.g: *PostgreSQL home page: >* We might need to consider this change for pg_validatebackup binary. On Mon, Mar 16, 2020 at 10:37 AM Suraj K

Re: Expose lock group leader pid in pg_stat_activity

On Mon, Mar 16, 2020 at 12:43:41AM -0500, Justin Pryzby wrote: > I think I see. Julien's v3 patch did this: > https://www.postgresql.org/message-id/attachment/106429/pgsa_leader_pid-v3.diff > + if (proc->lockGroupLeader) > + values[29

Re: Expose lock group leader pid in pg_stat_activity

On Sun, Mar 15, 2020 at 11:27:52PM -0500, Justin Pryzby wrote: > On Tue, Jan 28, 2020 at 12:36:41PM +0100, Julien Rouhaud wrote: > > So, AFAICT the LockHashPartitionLockByProc is required when > > iterating/modifying lockGroupMembers or lockGroupLink, but just > > getting the leader pid should be s

Re: Refactor compile-time assertion checks for C/C++

On Fri, Mar 13, 2020 at 11:00:33AM -0400, Tom Lane wrote: > Michael Paquier writes: >> Hmm. v3 actually broke the C++ fallback of StaticAssertExpr() and >> StaticAssertStmt() (v1 did not), a simple fix being something like >> the attached. > > The buildfarm seems happy, so why do you think it's

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Mon, Mar 16, 2020 at 8:15 AM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 9:17 PM Dilip Kumar wrote: > > > > On Sun, Mar 15, 2020 at 5:58 PM Amit Kapila wrote: > > > > > > > > > 1. Group members wait for page locks. If you test that the leader > > > acquires the page lock and then member a

Re: backup manifests

Thank you, Robert. Getting below warning while compiling the v11-0003-pg_validatebackup-Validate-a-backup-against-the-.patch. *pg_validatebackup.c: In function ‘report_manifest_error’:pg_validatebackup.c:356:2: warning: function might be possible candidate for ‘gnu_printf’ format attribute [-Ws

Re: Online checksums verification in the backend

On Wed, 11 Mar 2020 at 16:18, Julien Rouhaud wrote: > > On Tue, Dec 10, 2019 at 11:12:34AM +0100, Julien Rouhaud wrote: > > On Tue, Dec 10, 2019 at 3:26 AM Michael Paquier wrote: > > > > > > On Mon, Dec 09, 2019 at 07:02:43PM +0100, Julien Rouhaud wrote: > > > > On Mon, Dec 9, 2019 at 5:21 PM Rob

Re: Berserk Autovacuum (let's save next Mandrill)

On Fri, Mar 13, 2020 at 10:48:27PM +0100, Laurenz Albe wrote: > On Fri, 2020-03-13 at 13:44 -0500, Justin Pryzby wrote: > > Possible it would be better to run VACUUM *without* freeze_min_age=0 ? (I > > get > > confused and have to spend 20min re-reading the vacuum GUC docs every time I > > deal w

Re: Expose lock group leader pid in pg_stat_activity

On Tue, Jan 28, 2020 at 12:36:41PM +0100, Julien Rouhaud wrote: > So, AFAICT the LockHashPartitionLockByProc is required when > iterating/modifying lockGroupMembers or lockGroupLink, but just > getting the leader pid should be safe. This still seems unsafe: git show -U11 -w --patience b025f32e0b

Re: effective_io_concurrency's steampunk spindle maths

On Tue, Mar 10, 2020 at 12:20 PM Thomas Munro wrote: > Here's a patch set to remove the spindle stuff, add a maintenance > variant, and use the maintenance one in > heap_compute_xid_horizon_for_tuples(). Pushed.

Re: allow online change primary_conninfo

On Fri, Mar 13, 2020 at 11:17:38AM -0300, Alvaro Herrera wrote: > ... oh, that's exactly what 0002 does. So patch 0001 is only about > making a temporary change to the create_temp_slot to be consistent with > existing policy, before changing the policy. Yes. In my opinion, patch 0002 should not

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada wrote: > > On Mon, 16 Mar 2020 at 00:54, Dilip Kumar wrote: > > > > On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > > > > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > > > > > I have modified 0001 and 0002 slightly, Basicall

Re: Berserk Autovacuum (let's save next Mandrill)

On Fri, 13 Mar 2020 at 05:11, David Rowley wrote: > > On Fri, 13 Mar 2020 at 01:43, Masahiko Sawada > wrote: > > > > On Thu, 12 Mar 2020 at 16:28, David Rowley wrote: > > > Laurenz highlighted a seemingly very valid reason that the current > > > GUCs cannot be reused. Namely, say the table has 1

Re: Online checksums verification in the backend

On Wed, Mar 11, 2020 at 08:18:23AM +0100, Julien Rouhaud wrote: > The cfbot reported a build failure, so here's a rebased v2 which also contains > the pg_stat_report_failure() call and extra log info. + * - if a block is not found in shared_buffers, the LWLock is relased and the + * block is rea

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Mon, 16 Mar 2020 at 00:54, Dilip Kumar wrote: > > On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > > > I have modified 0001 and 0002 slightly, Basically, instead of two > > > function CheckAndSetLockHeld and CheckAndReSetLo

Re: add types to index storage params on doc

On Sun, Mar 15, 2020 at 7:10 PM Atsushi Torikoshi wrote: > I think it'll be better to add types to storage parameters > on CREATE INDEX for the consistency. Seems reasonable to me. -- Peter Geoghegan

Re: backend type in log_line_prefix?

On 2020/03/15 19:32, Peter Eisentraut wrote: On 2020-03-13 22:24, Peter Eisentraut wrote: On 2020-03-10 19:07, Alvaro Herrera wrote: I like these patches; the first two are nice cleanup. My only gripe is that pgstat_get_backend_desc() is not really a pgstat function; I think it should have

Re: add types to index storage params on doc

On 2020/03/16 11:09, Atsushi Torikoshi wrote: Hi, The current manual on CREATE TABLE[1] describes storage parameters with their types. But manual on CREATE INDEX[2] describes storage parameters WITHOUT their types. I think it'll be better to add types to storage parameters on CREATE INDEX fo

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sun, Mar 15, 2020 at 9:17 PM Dilip Kumar wrote: > > On Sun, Mar 15, 2020 at 5:58 PM Amit Kapila wrote: > > > > > > 1. Group members wait for page locks. If you test that the leader > > acquires the page lock and then member also tries to acquire the same > > lock on the same index, it wouldn'

Re: ssl passphrase callback

On 2/18/20 11:39 PM, Andrew Dunstan wrote: This should fix the issue, it happened when I switched to using a pre-generated cert/key. # Review The patch still applies and passes the test suite, both with openssl enabled and with it disabled. As for the feature I agree that it is nice to expo

add types to index storage params on doc

Hi, The current manual on CREATE TABLE[1] describes storage parameters with their types. But manual on CREATE INDEX[2] describes storage parameters WITHOUT their types. I think it'll be better to add types to storage parameters on CREATE INDEX for the consistency. Attached a patch. Any thought?

Re: comments on elements of xlogctldata

On 2020/03/16 10:57, Atsushi Torikoshi wrote: Hi, It seems the comments on SharedHotStandbyActive and SharedRecoveryInProgress are the same in XLogCtlData. How about modifying the comment on SharedHotStandbyActive? Attached a patch. Thanks for the report and patch! The patch looks good t

type of some table storage params on doc

Hi, As far as I read the reloptions.c, autovacuum_vacuum_cost_delay, autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor are the members of relopt_real, so their type seems the same, real. But the manual about storage parameters[1] says two of their type are float4 and the other is

comments on elements of xlogctldata

Hi, It seems the comments on SharedHotStandbyActive and SharedRecoveryInProgress are the same in XLogCtlData. How about modifying the comment on SharedHotStandbyActive? Attached a patch. Regards, -- Torikoshi Atsushi fix_comments_on_SharedHotStandbyActive_v1.patch Description: Binary data

RE: Planning counters in pg_stat_statements (using pgss_store)

On Sat, Mar 14, 2020 at 5:28 PM, Julien Rouhaud wrote: > On Sat, Mar 14, 2020 at 03:04:00AM -0700, legrand legrand wrote: > > imai.yoshik...@fujitsu.com wrote > > > On Thu, Mar 12, 2020 at 6:37 PM, Julien Rouhaud wrote: > > >> That's very interesting feedback, thanks! I'm not a fan of giving a way

Re: proposal: new polymorphic types - commontype and commontypearray

Pavel Stehule writes: > ne 15. 3. 2020 v 17:48 odesílatel Tom Lane napsal: >> Well, here's a version that does it like that, but personally I find these >> messages too verbose and not an improvement on what I had before. > There was a problem just with anyrange type. This last version looks > p

Re: control max length of parameter values logged

On 2020-Mar-14, Tom Lane wrote: > Bruce Momjian writes: > > I am sorry --- I am confused. Why are we truncating or allowing control > > of truncation of BIND parameter values, but have no such facility for > > queries. Do we assume queries are shorter than BIND parameters, or is > > it just tha

More weird stuff in polymorphic type resolution

While poking at Pavel's "anycompatible" patch, I found a couple more pre-existing issues having to do with special cases for actual input type "anyarray". Ordinarily that would be impossible since we should have resolved "anyarray" to some specific array type earlier; but you can make it happen by

Re: Memory-Bounded Hash Aggregation

On Thu, 2020-03-12 at 16:01 -0500, Justin Pryzby wrote: > I don't understand what's meant by "the chosen plan". > Should it say, "at execution ..." instead of "execution time" ? I removed that wording; hopefully it's more clear without it? > Either remove "plan types" for consistency with > enabl

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

On Sun, Mar 15, 2020 at 06:15:02PM +0100, Fabien COELHO wrote: > Some feedback on v10: Thanks for looking. I'm hoping to hear from Alvaro what he thinks of this approach (all functions to show isdir, rather than one function which lists recursively). > All patches apply cleanly, one on top of th

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

On Fri, Mar 13, 2020 at 4:22 PM Tom Lane wrote: > > Alvaro Herrera writes: > > Also, I wonder if it would be better to modify our policies so that we > > update typedefs.list more frequently. Some people include additions > > with their commits, but it's far from SOP. > > Perhaps. My own workfl

Re: WAL usage calculation patch

> > > On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik > > > wrote: > > > > I wanted to keep the patch small and simple, and fit to practical > > > > needs. This patch is supposed to provide tuning assistance, catching > > > > an io heavy query in commit-bound situation. > > > > Total WAL usage per D

Re: proposal: new polymorphic types - commontype and commontypearray

ne 15. 3. 2020 v 17:48 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > Tom Lane napsal: > >> Yeah, that's what I said. But does it really add anything beyond the > >> proposed text "A function returning a polymorphic type must have at > least > >> one matching polymorphic argument"? I

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

On Sat, Mar 14, 2020 at 10:55 PM James Coleman wrote: > > On Fri, Mar 13, 2020 at 1:06 PM James Coleman wrote: > > > > On Thu, Mar 12, 2020 at 5:53 PM Alvaro Herrera > > wrote: > > > > > > I gave this a very quick look; I don't claim to understand it or > > > anything, but I thought these trivi

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

Hello Justin, Some feedback on v10: All patches apply cleanly, one on top of the previous. I really wish there would be less than 9 patches… * v10.1 doc change: ok * v10.2 doc change: ok, not sure why it is not merged with previous * v10.3 test add: could be merge with both previous Tests

Re: [PATCH] Erase the distinctClause if the result is unique by definition

Hi All: I have re-implemented the patch based on David's suggestion/code, Looks it works well. The updated patch mainly includes: 1. Maintain the not_null_colno in RelOptInfo, which includes the not null from catalog and the not null from vars. 2. Add the restictinfo check at populate_base

Re: proposal: new polymorphic types - commontype and commontypearray

Pavel Stehule writes: > Tom Lane napsal: >> Yeah, that's what I said. But does it really add anything beyond the >> proposed text "A function returning a polymorphic type must have at least >> one matching polymorphic argument"? I don't think it'd be terribly >> helpful to say "A function retur

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > I have modified 0001 and 0002 slightly, Basically, instead of two > > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created > > a one function. > > > > +CheckAndSetL

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sun, Mar 15, 2020 at 5:58 PM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar wrote: > > > > On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > > > > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > > > > > Apart from that, I have also extended the solution

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > I have modified 0001 and 0002 slightly, Basically, instead of two > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created > a one function. > +CheckAndSetLockHeld(LOCALLOCK *locallock, bool value) Can we rename the paramete

Re: Additional improvements to extended statistics

On Sun, 15 Mar 2020 at 00:08, Tomas Vondra wrote: > > On Sat, Mar 14, 2020 at 05:56:10PM +0100, Tomas Vondra wrote: > > > >Attached is a patch series rebased on top of the current master, after > >committing the ScalarArrayOpExpr enhancements. I've updated the OR patch > >to get rid of the code du

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar wrote: > > On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > > > Apart from that, I have also extended the solution for the page lock. > > > And, I have also broken down the 3rd patch

Re: expose parallel leader in CSV and log_line_prefix

On Sun, Mar 15, 2020 at 06:18:31AM -0500, Justin Pryzby wrote: > See also: > https://commitfest.postgresql.org/27/2390/ > https://www.postgresql.org/message-id/flat/caobau_yy5bt0vtpz2_lum6cucgeqmynoj8-rgto+c2+w3de...@mail.gmail.com > b025f32e0b Add leader_pid to pg_stat_activity FTR this is a fol

expose parallel leader in CSV and log_line_prefix

See also: https://commitfest.postgresql.org/27/2390/ https://www.postgresql.org/message-id/flat/caobau_yy5bt0vtpz2_lum6cucgeqmynoj8-rgto+c2+w3de...@mail.gmail.com b025f32e0b Add leader_pid to pg_stat_activity -- Justin >From 5268e89fb32fbb639cb39796729dfe0dfcf14705 Mon Sep 17 00:00:00 2001 From:

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar wrote: > > On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > > > Apart from that, I have also extended the solution for the page lock. > > > And, I have also broken down the 3rd patch

Re: backend type in log_line_prefix?

On 2020-03-15 10:57, Justin Pryzby wrote: I suggest the CSV/log should also have the leader_pid, corresponding to | b025f32e0b Add leader_pid to pg_stat_activity I haven't followed those developments. It sounds interesting, but I suggest you start a new thread or continue in the thread that a

Re: backend type in log_line_prefix?

On 2020-03-13 22:24, Peter Eisentraut wrote: On 2020-03-10 19:07, Alvaro Herrera wrote: I like these patches; the first two are nice cleanup. My only gripe is that pgstat_get_backend_desc() is not really a pgstat function; I think it should have a different name with a prototype in miscadmin.h

Re: Berserk Autovacuum (let's save next Mandrill)

On Fri, Mar 13, 2020 at 02:38:51PM -0700, Andres Freund wrote: > > Having now played with the patch, I'll suggest that 1000 is too high a > > threshold. If autovacuum runs without FREEZE, I don't see why it couldn't > > be > > much lower (10?) or use (0.2 * n_ins + 50) like the other auto

Re: backend type in log_line_prefix?

On Fri, Mar 13, 2020 at 10:22:52PM +0100, Peter Eisentraut wrote: > >Can I suggest: > > > >- appendCSVLiteral(&buf, MyBgworkerEntry->bgw_type); > >+ appendCSVLiteral(&buf, MyBgworkerEntry->bgw_name); > > The difference is intentional. bgw_type is so that you can filter

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > Apart from that, I have also extended the solution for the page lock. > > And, I have also broken down the 3rd patch in two parts for relation > > extension and for the page lock. > >