Re: subscriptionCheck failures

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 9:00 AM Amit Kapila wrote: > > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro wrote: > > > > Hi, > > > > This seems to be a new low frequency failure, I didn't see it mentioned > > already: > > > > Thanks for reporting, I'll look into it. > By looking at the logs [1] in th

Re: subscriptionCheck failures

2021-03-16 Thread vignesh C
On Tue, Mar 16, 2021 at 12:29 PM Amit Kapila wrote: > > On Tue, Mar 16, 2021 at 9:00 AM Amit Kapila wrote: > > > > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro wrote: > > > > > > Hi, > > > > > > This seems to be a new low frequency failure, I didn't see it mentioned > > > already: > > > > > > >

Permission failures with WAL files in 13~ on Windows

2021-03-16 Thread Michael Paquier
Hi all, There has been for the last couple of weeks a collection of reports complaining that the renaming of WAL segments is broken: https://www.postgresql.org/message-id/3861ff1e-0923-7838-e826-094cc9bef...@hot.ee https://www.postgresql.org/message-id/16874-c3eecd319e36a...@postgresql.org https:/

RE: libpq debug log

2021-03-16 Thread tsunakawa.ta...@fujitsu.com
I've finished the review. Here are the last set of comments. (16) (15) is also true for the processing of 'H' message. (17) pqTraceOutputD + for (i = 0; i < nfields; i++) + { + len = pqTraceOutputInt32(message + cursor, f); +

Re: shared-memory based stats collector

2021-03-16 Thread Kyotaro Horiguchi
At Mon, 15 Mar 2021 19:04:29 -0700, Andres Freund wrote in > I have a few questions about the patch: > > - Why was collect_oids() changed to a different hashtable as part of > this change? Seems fairly independent? Right. It is changed at the time I used simplehash for some other stuff. > -

comment fix in postmaster.c

2021-03-16 Thread Kyotaro Horiguchi
While I worked on a patch, I noticed a comment that is inconsistent with the fact. > * SIGQUIT is the special signal that says exit without proc_exit > * and let the user know what's going on. But if SendStop is set > * (-s on command line), then we send SIGSTOP instead, so that we > * can get cor

Re: fdatasync performance problem with large number of DB files

2021-03-16 Thread Fujii Masao
On 2021/03/15 8:33, Thomas Munro wrote: On Mon, Mar 15, 2021 at 11:52 AM Thomas Munro wrote: Time being of the essence, here is the patch I posted last year, this time with a GUC and some docs. You can set sync_after_crash to "fsync" (default) or "syncfs" if you have it. Cfbot told me to

Re: Allow batched insert during cross-partition updates

2021-03-16 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, March 16, 2021 6:13 AM, Amit Langote wrote: > Hi Georgios, > > On Fri, Mar 12, 2021 at 7:59 PM gkokola...@pm.me wrote: > > > On Friday, March 12, 2021 3:45 AM, Amit Langote amitlangot...@gmail.com > > wrote: > > > > > On Thu, Mar 11, 2021 at 8:

Re: shared-memory based stats collector

2021-03-16 Thread Kyotaro Horiguchi
At Tue, 16 Mar 2021 16:44:38 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 15 Mar 2021 19:04:29 -0700, Andres Freund wrote > in > > Do you have some benchmarks that you used to verify performance? > > https://www.postgresql.org/message-id/20201008.160326.2246946707652981235.horikyota.ntt

Re: pg_subscription - substream column?

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't > have any mention of the substream column. > > Accidental omission by commit 4648243 [2] from last year? > Right. I'll fix this unless you or someone else is interested i

Re: GROUP BY DISTINCT

2021-03-16 Thread Vik Fearing
On 3/13/21 12:33 AM, Tomas Vondra wrote: > Hi Vik, > > The patch seems quite ready, I have just two comments. Thanks for taking a look. > 1) Shouldn't this add another for DISTINCT, somewhere in the > documentation? Now the index points just to the SELECT DISTINCT part. Good idea; I never thin

Re: fdatasync performance problem with large number of DB files

2021-03-16 Thread Fujii Masao
On 2021/03/16 8:15, Thomas Munro wrote: On Tue, Mar 16, 2021 at 3:30 AM Paul Guo wrote: By the way, there is a usual case that we could skip fsync: A fsync-ed already standby generated by pg_rewind/pg_basebackup. The state of those standbys are surely not DB_SHUTDOWNED/DB_SHUTDOWNED_IN_REC

Re: HotStandbyActive() issue in postgres

2021-03-16 Thread Hao Wu
Yes, I have an extension/UDF that needs to know if the server is currently running as hot standby. For example, a UDF foo() wants to run on both the primary and secondary and runs different behaviors for different roles. Promoted secondary looks the same as the primary since it's no longer real hot

Re: "has_column_privilege()" issue with attnums and non-existent columns

2021-03-16 Thread Chengxi Sun
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested I tested the patch and it works well. And according to the comment,

Re: HotStandbyActive() issue in postgres

2021-03-16 Thread Fujii Masao
On 2021/03/16 12:24, Hao Wu wrote: Yes, I have an extension/UDF that needs to know if the server is currently running as hot standby. For example, a UDF foo() wants to run on both the primary and secondary and runs different behaviors for different roles. Promoted secondary looks the same as t

Re: Allow batched insert during cross-partition updates

2021-03-16 Thread Amit Langote
Hi Georgios, On Tue, Mar 16, 2021 at 5:12 PM wrote: > On Tuesday, March 16, 2021 6:13 AM, Amit Langote > wrote: > > On Fri, Mar 12, 2021 at 7:59 PM gkokola...@pm.me wrote: > > > On Friday, March 12, 2021 3:45 AM, Amit Langote amitlangot...@gmail.com > > > wrote: > > > > By the way, the test ca

Re: Permission failures with WAL files in 13~ on Windows

2021-03-16 Thread Magnus Hagander
On Tue, Mar 16, 2021 at 8:20 AM Michael Paquier wrote: > > Hi all, > > There has been for the last couple of weeks a collection of reports > complaining that the renaming of WAL segments is broken: > https://www.postgresql.org/message-id/3861ff1e-0923-7838-e826-094cc9bef...@hot.ee > https://www.po

Re: Allow batched insert during cross-partition updates

2021-03-16 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, March 16, 2021 9:59 AM, Amit Langote wrote: > Hi Georgios, > > On Tue, Mar 16, 2021 at 5:12 PM gkokola...@pm.me wrote: > > > On Tuesday, March 16, 2021 6:13 AM, Amit Langote amitlangot...@gmail.com > > wrote: > > > > > On Fri, Mar 12, 2021 at 7

Re: comment fix in postmaster.c

2021-03-16 Thread Fujii Masao
On 2021/03/16 16:51, Kyotaro Horiguchi wrote: While I worked on a patch, I noticed a comment that is inconsistent with the fact. * SIGQUIT is the special signal that says exit without proc_exit * and let the user know what's going on. But if SendStop is set * (-s on command line), then we se

Re: A new function to wait for the backend exit after termination

2021-03-16 Thread Bharath Rupireddy
On Mon, Mar 15, 2021 at 10:38 AM Fujii Masao wrote: > On 2021/03/15 12:27, Bharath Rupireddy wrote: > > On Sun, Mar 7, 2021 at 2:39 PM Bharath Rupireddy > > wrote: > >> Attaching v7 patch for further review. > > > > Attaching v8 patch after rebasing on to the latest master. > > Thanks for rebasin

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-16 Thread houzj.f...@fujitsu.com
> To support parallel insert into FK relation. > There are two scenarios need attention. > 1) foreign key and primary key are on the same table(INSERT's target table). > (referenced and referencing are the same table) > 2) referenced and referencing table are both partition of INSERT's target >

Re: fdatasync performance problem with large number of DB files

2021-03-16 Thread Paul Guo
On Tue, Mar 16, 2021 at 4:29 PM Fujii Masao wrote: > > > > On 2021/03/16 8:15, Thomas Munro wrote: > > On Tue, Mar 16, 2021 at 3:30 AM Paul Guo wrote: > >> By the way, there is a usual case that we could skip fsync: A fsync-ed > >> already standby generated by pg_rewind/pg_basebackup. > >> The s

Re: Permission failures with WAL files in 13~ on Windows

2021-03-16 Thread Michael Paquier
On Tue, Mar 16, 2021 at 10:02:25AM +0100, Magnus Hagander wrote: > If you back out that patch, does the problem you can reproduce with > archive_command go away? That's the first thing I did after seeing the failure, and I saw nothing after 2~3 hours of pgbench :) The second thing I did was to re

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread vignesh C
On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: >> >> I think something on these lines should be much >> easier than the spool-file implementation unless we see any problem >> with this idea. >> > > Here's a new patch-set that implements

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Dilip Kumar
On Tue, Mar 16, 2021 at 12:59 AM Robert Haas wrote: > > On Mon, Mar 15, 2021 at 8:14 AM Dilip Kumar wrote: > > In the attached patches I have changed this, ... > > OK, so just looking over this patch series, here's what I think: > > Regarding 0003: > > The biggest thing that jumps out at me while

Re: Permission failures with WAL files in 13~ on Windows

2021-03-16 Thread Magnus Hagander
On Tue, Mar 16, 2021 at 11:22 AM Michael Paquier wrote: > > On Tue, Mar 16, 2021 at 10:02:25AM +0100, Magnus Hagander wrote: > > If you back out that patch, does the problem you can reproduce with > > archive_command go away? > > That's the first thing I did after seeing the failure, and I saw > n

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Masahiko Sawada
On Sun, Mar 14, 2021 at 12:23 PM Peter Geoghegan wrote: > > On Fri, Mar 12, 2021 at 9:34 PM Masahiko Sawada wrote: > > I agreed that when we're close to overrunning the > > maintnenance_work_mem space, the situation changes. If we skip it in > > even that case, the next vacuum will be likely to u

Re: Permission failures with WAL files in 13~ on Windows

2021-03-16 Thread Ranier Vilela
>Yeah, it'd definitely be good to figure out exactly what it is that >triggers the issue. I think that this issue is the same at 1. And IMHO the patch solves, but nobody is interested. I think that "MOVEFILE_COPY_ALLOWED" that's what is missing. At least on my machine, Postgres can rename statis

Re[2]: On login trigger: take three

2021-03-16 Thread Ivan Panchenko
Hi, Thank you, Konstantin, for this very good feature with numerous use cases. Please find the modified patch attached. I’ve added the ‘enable_client_connection_trigger’ GUC to the sample config file and also an additional example page to the docs. Check world has passed and it is ready for com

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: >> >> I think something on these lines should be much >> easier than the spool-file implementation unless we see any problem >> with this idea. >> > > Here's a new patch-set that implements

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-03-16 Thread Bharath Rupireddy
On Mon, Mar 15, 2021 at 11:23 AM torikoshia wrote: > > On 2021-03-07 19:16, Bharath Rupireddy wrote: > > On Fri, Feb 5, 2021 at 5:15 PM Bharath Rupireddy > > wrote: > >> > >> pg_terminate_backend and pg_cancel_backend with postmaster PID produce > >> "PID is not a PostgresSQL server process"

Re: pg_stat_statements oddity with track = all

2021-03-16 Thread Magnus Hagander
On Tue, Mar 9, 2021 at 3:39 AM Julien Rouhaud wrote: > > On Mon, Mar 08, 2021 at 06:03:59PM +0100, Magnus Hagander wrote: > > On Sun, Mar 7, 2021 at 8:39 AM Julien Rouhaud wrote: > > > > > > Yes I was a bit worried about the possible extra text entry. I kept > > > things > > > simple until now

Re: pg_subscription - substream column?

2021-03-16 Thread Peter Smith
On Tue, Mar 16, 2021 at 7:20 PM Amit Kapila wrote: > > On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't > > have any mention of the substream column. > > > > Accidental omission by commit 4648243 [2] from last year?

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
On Mon, Mar 15, 2021 at 3:21 PM Avinash Kumar wrote: > Hi, > > On Mon, Mar 15, 2021 at 1:18 PM Peter Geoghegan wrote: > >> On Mon, Mar 15, 2021 at 6:56 AM Avinash Kumar >> wrote: >> > psql:amchecksql.sql:17: DEBUG: leaf block 1043751 of index >> "idx_id_mtime" has no first data item >> >> That

Re: EXPLAIN/EXPLAIN ANALYZE REFRESH MATERIALIZED VIEW

2021-03-16 Thread Bharath Rupireddy
On Tue, Mar 16, 2021 at 1:15 AM Tom Lane wrote: > > [ Sorry for not looking at this thread sooner ] > > Bharath Rupireddy writes: > > Currently, $subject is not allowed. We do plan the mat view query > > before every refresh. I propose to show the explain/explain analyze of > > the select part of

Re: dynamic result sets support in extended query protocol

2021-03-16 Thread Peter Eisentraut
On 15.03.21 14:56, David Steele wrote: Hi Peter, On 12/30/20 9:33 AM, Peter Eisentraut wrote: On 2020-10-09 20:46, Andres Freund wrote: Is there really a good reason for forcing the client to issue NextResult, Describe, Execute for each of the dynamic result sets? It's not like there's really

RE: subscriptionCheck failures

2021-03-16 Thread osumi.takami...@fujitsu.com
Hi On Tuesday, March 16, 2021 4:15 PM vignesh C wrote: > On Tue, Mar 16, 2021 at 12:29 PM Amit Kapila > wrote: > > > > On Tue, Mar 16, 2021 at 9:00 AM Amit Kapila > wrote: > > > > > > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro > wrote: > > > > > > > > Hi, > > > > > > > > This seems to be a

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread vignesh C
On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: >> >> I think something on these lines should be much >> easier than the spool-file implementation unless we see any problem >> with this idea. >> > > Here's a new patch-set that implements

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Dilip Kumar
On Tue, Mar 16, 2021 at 4:07 PM Dilip Kumar wrote: > > INSERT TIME > Head: 17418.299 ms Patch: 20956.231 ms > > CTAS TIME: > Head: 12837.872 ms Patch: 16775.739 ms On quick analysis with perf it appeared that the performance is degrading because of deforming - 16.19% 3.54% postgres postg

Re: simplifying foreign key/RI checks

2021-03-16 Thread Amit Langote
On Mon, Mar 8, 2021 at 11:41 PM Amit Langote wrote: > On Thu, Mar 4, 2021 at 5:15 AM Tom Lane wrote: > > Lastly, ri_PerformCheck is pretty careful about not only which > > snapshot it uses, but which *pair* of snapshots it uses, because > > sometimes it needs to worry about data changes since the

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Masahiko Sawada
On Mon, Mar 15, 2021 at 11:04 AM Peter Geoghegan wrote: > > On Thu, Mar 11, 2021 at 8:31 AM Robert Haas wrote: > > But even if not, I'm not sure this > > helps much with the situation you're concerned about, which involves > > non-HOT tuples. > > Attached is a POC-quality revision of Masahiko's >

Re: pg_subscription - substream column?

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 5:27 PM Peter Smith wrote: > > On Tue, Mar 16, 2021 at 7:20 PM Amit Kapila wrote: > > > > On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > > > > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't > > > have any mention of the substream column.

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 6:22 PM vignesh C wrote: > > On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: > >> > > 2) table_states_not_ready global variable is used immediately after > call to FetchTableStates, we can make FetchTableState

crash during cascaded foreign key update

2021-03-16 Thread Amit Langote
With HEAD (I think v12 and greater), I see $subject when trying out the following scenario: -- in backend 1 create table p (a int primary key); create table f (a int references p on update cascade deferrable initially deferred); insert into p values (1); begin isolation level serializable; insert

Re: Nondeterministic collations and the value returned by GROUP BY x

2021-03-16 Thread Tom Lane
Jim Finnerty writes: > right. It doesn't matter which of the values is returned; however, a > plausible-sounding implementation would case-fold the value, like GROUP BY > LOWER(x), but the case-folded value isn't necessarily one of the original > values and so that could be subtly wrong in the ca

Re: crash during cascaded foreign key update

2021-03-16 Thread Tom Lane
Amit Langote writes: > With HEAD (I think v12 and greater), I see $subject when trying out > the following scenario: I wonder if this is related to https://www.postgresql.org/message-id/flat/89429.1584443208%40antos which we've still not done anything about. regards, to

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
On Mon, Mar 15, 2021 at 6:58 PM Andres Freund wrote: > I don't particularly like PG_RETURN_HEAPTUPLEHEADER_RAW(). What is "raw" > about it? It also seems to me like there needs to at least be a > sentence or two explaining when to use which of the functions. It seemed like the natural name to me

re: crash during cascaded foreign key update

2021-03-16 Thread Ranier Vilela
>0 0x7f747e6e2387 in raise () from /lib64/libc.so.6 >#1 0x7f747e6e3a78 in abort () from /lib64/libc.so.6 >#2 0x00ae056a in ExceptionalCondition ( >conditionName=0xb67c10 "!ItemPointerEquals(&oldtup.t_self, >&oldtup.t_data->t_ctid)", >errorType=0xb66d89 "FailedAssertion", fileName=0x

Re: GROUP BY DISTINCT

2021-03-16 Thread Tomas Vondra
On 3/16/21 9:21 AM, Vik Fearing wrote: > On 3/13/21 12:33 AM, Tomas Vondra wrote: >> Hi Vik, >> >> The patch seems quite ready, I have just two comments. > > Thanks for taking a look. > >> 1) Shouldn't this add another for DISTINCT, somewhere in the >> documentation? Now the index points just

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-03-16 Thread Tom Lane
Justin Pryzby writes: > I think it's somewhat confusing that there's two callbacks. > The first one applies only during typinput/typreceive. > I guess the 2nd one should say that they're printed "in *future errors". I adjusted the comments to make this a bit clearer, and pushed it.

Re: libpq debug log

2021-03-16 Thread Alvaro Herrera
On 2021-Mar-15, iwata@fujitsu.com wrote: > I create protocol message reading function for each protocol message type. > (Ex. pqTraceOutputB() read Bind message) > This makes the nesting shallower and makes the code easier to read. I'm not sure I agree with this structural change. Yes, it is

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Dilip Kumar
On Tue, Mar 16, 2021 at 7:57 PM Robert Haas wrote: > > That behavior feels unacceptable to me from a user expectations point > of view. I think there's an argument that if I update a tuple that > contains a compressed datum, and I don't update that particular > column, I think it would be OK to n

Re: pg_stat_statements oddity with track = all

2021-03-16 Thread Julien Rouhaud
On Tue, Mar 16, 2021 at 12:55:45PM +0100, Magnus Hagander wrote: > On Tue, Mar 9, 2021 at 3:39 AM Julien Rouhaud wrote: > > > > I think that we might be able to handle that without a flag. The only thing > > that would need to be done is when creating an entry, look for an existing > > entry with

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Masahiko Sawada
On Tue, Mar 16, 2021 at 10:39 PM Masahiko Sawada wrote: > > On Mon, Mar 15, 2021 at 11:04 AM Peter Geoghegan wrote: > > > > One consequence of my approach is that we now call > > lazy_cleanup_all_indexes(), even when we've skipped index vacuuming > > itself. We should at least "check-in" with the

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 15, 2021, at 11:09 PM, Noah Misch wrote: > >> Not sure that I believe the theory that this is from bad luck of >> concurrent autovacuum timing, though. > > With autovacuum_naptime=1s, on hornet, the failure reproduced twice in twelve > runs. With v6-0001-Turning-off-autovacuum-durin

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Mark Dilger writes: > I think autovacuum simply triggers the bug, and is not the cause of the bug. > If I turn autovacuum off and instead do an ANALYZE in each test database > rather than performing the corruptions, I get reports about problems in > pg_statistic. This is on my mac laptop. Th

Re: A new function to wait for the backend exit after termination

2021-03-16 Thread Magnus Hagander
On Tue, Mar 16, 2021 at 10:38 AM Bharath Rupireddy wrote: > > On Mon, Mar 15, 2021 at 10:38 AM Fujii Masao > wrote: > > On 2021/03/15 12:27, Bharath Rupireddy wrote: > > > On Sun, Mar 7, 2021 at 2:39 PM Bharath Rupireddy > > > wrote: > > >> Attaching v7 patch for further review. > > > > > > Atta

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 9:07 AM, Tom Lane wrote: > > Mark Dilger writes: >> I think autovacuum simply triggers the bug, and is not the cause of the bug. >> If I turn autovacuum off and instead do an ANALYZE in each test database >> rather than performing the corruptions, I get reports about

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 5:01 AM Avinash Kumar wrote: > I am afraid that it looks to me like a deduplication bug but not sure how > this can be pin-pointed. If there is something I could do to determine that, > I would be more than happy. That cannot be ruled out, but I don't consider it to be t

Re: [PATCH] pgbench: improve \sleep meta command

2021-03-16 Thread Fujii Masao
On 2021/03/09 0:54, Fujii Masao wrote: On 2021/03/08 23:10, Alvaro Herrera wrote: On 2021-Mar-08, kuroda.hay...@fujitsu.com wrote: Dear Fujii-san, Miyake-san Isn't it better to accept even negative sleep time like currently pgbench does? Otherwise we always need to check the variable is

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
Hi, On Tue, Mar 16, 2021 at 1:44 PM Peter Geoghegan wrote: > On Tue, Mar 16, 2021 at 5:01 AM Avinash Kumar > wrote: > > I am afraid that it looks to me like a deduplication bug but not sure > how this can be pin-pointed. If there is something I could do to determine > that, I would be more tha

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 9:30 AM, Mark Dilger wrote: > > > >> On Mar 16, 2021, at 9:07 AM, Tom Lane wrote: >> >> Mark Dilger writes: >>> I think autovacuum simply triggers the bug, and is not the cause of the >>> bug. If I turn autovacuum off and instead do an ANALYZE in each test >>> data

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger wrote: > Yeah, that looks related: > > regression=# select attname, attlen, attnum, attalign from pg_attribute where > attrelid = 2619 and attname like 'stavalue%'; > attname | attlen | attnum | attalign > +++--

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 9:50 AM Avinash Kumar wrote: > Yes, it was on the failover-over server where the issue is currently seen. > Took a snapshot of the data directory so that the issue can be analyzed. I would be very cautious when using LVM snapshots with a Postgres data directory, or VM-bas

Re: Calendar support in localization

2021-03-16 Thread Surafel Temesgen
Hi Thomas On Mon, Mar 15, 2021 at 2:58 PM Thomas Munro wrote: > > One key question here is whether you need a different date type or > just different operations (functions, operators etc) on the existing > types. > > I am thinking of having a converter to a specific calendar after each operation

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger > wrote: >> It shows them all has having attalign = 'd', but for some array types the >> alignment will be 'i', not 'd'. So it's lying a bit about the contents. >> But I'm now confused why this caused problems on the two hosts

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
... btw, I now see that tern and hornet are passing this test at least as much as they're failing, proving that there's some timing or random chance involved. That doesn't completely eliminate the idea that there may be an architecture component to the issue, but it sure reduces its credibility.

Re: [Proposal] Global temporary tables

2021-03-16 Thread Pavel Stehule
Hi this patch is broken now. Please, can you check it? Regards Pavel st 25. 11. 2020 v 14:08 odesílatel 曾文旌 napsal: > > > 2020年11月25日 14:19,Pavel Stehule 写道: > > > > po 23. 11. 2020 v 10:27 odesílatel 曾文旌 > napsal: > >> >> >> 2020年11月21日 02:28,Pavel Stehule 写道: >> >> Hi >> >> pá 11. 9. 20

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 10:48 AM, Tom Lane wrote: > > Robert Haas writes: >> On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger >> wrote: >>> It shows them all has having attalign = 'd', but for some array types the >>> alignment will be 'i', not 'd'. So it's lying a bit about the contents. >>>

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Tom Lane
Peter Geoghegan writes: > ... It's hard to believe that the problem is > squarely with _bt_swap_posting(). IIUC, the problem is seen on a replica server but not the primary? In that case, my thoughts would run towards a bug in WAL log creation or replay, causing the index contents to be different

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
On Tue, Mar 16, 2021 at 3:08 PM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would run towards a bug in WAL log c

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Mark Dilger writes: > On Mar 16, 2021, at 10:48 AM, Tom Lane wrote: >> (Too bad the message doesn't report the >> TOAST OID it probed for, so we can see if that's sane or not.) > I've added that and now get the toast pointer's va_valueid in the message: > heap table "postgres"."pg_catalog"."pg_

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:08 AM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would run towards a bug in WAL log c

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 1:48 PM Tom Lane wrote: > No. What should be happening there is that some arrays in the column > get larger alignment than they actually need, but that shouldn't cause > a problem (and has not done so, AFAIK, in the decades that it's been > like this). As you say, deformi

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 10:48 AM, Tom Lane wrote: > > I'm not entirely sure what's going on, but I think coming at this > with the mindset that "amcheck has detected some corruption" is > just going to lead you astray. Almost certainly, it's "amcheck > is incorrectly claiming corruption". That

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:20 AM Avinash Kumar wrote: > I can share any detail that would help here. I would like to know what you see when you run a slightly modified version of the same amcheck query. The same query as before, but with the call to bt_index_parent_check() replaced with a call to

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 2:22 PM Tom Lane wrote: > I'm circling back around to the idea that amcheck is trying to > validate TOAST references that are already dead, and it's getting > burnt because something-or-other has already removed the toast > rows, though not the referencing datums. That's l

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 16, 2021 at 1:48 PM Tom Lane wrote: >> No. What should be happening there is that some arrays in the column >> get larger alignment than they actually need, but that shouldn't cause >> a problem (and has not done so, AFAIK, in the decades that it's been >> like

Re: "has_column_privilege()" issue with attnums and non-existent columns

2021-03-16 Thread Joe Conway
On 3/16/21 1:42 AM, Chengxi Sun wrote: The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested I tested the patch and it wo

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Mark Dilger writes: > CopyStatistics seems to just copy Form_pg_statistic without regard for > who owns the toast. Is this safe? No less so than a ton of other places that insert values that might've come from on-disk storage. heap_toast_insert_or_update() is responsible for dealing with the pr

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 10:27:12 -0400, Robert Haas wrote: > > I'm don't like that after 0002 ExecEvalRow(), ExecEvalFieldStoreForm() > > contain a nearly identical copy of the same code. And > > make_tuple_from_row() also is similar. It seem that there should be a > > heap_form_tuple() version doing t

Re: shared-memory based stats collector

2021-03-16 Thread Robert Haas
On Mon, Mar 15, 2021 at 10:56 PM Andres Freund wrote: > I did roughly the first steps of the split as I had outlined. I moved: > > 1) wait event related functions into utils/activity/wait_event.c / >wait_event.h > > 2) "backend status" functionality (PgBackendStatus stuff) into >utils/acti

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 2:45 PM Tom Lane wrote: > > In what context is it OK to just add extra alignment padding? > > It's *not* extra, according to pg_statistic's tuple descriptor. > Both forming and deforming of pg_statistic tuples should honor > that and locate stavaluesX values on d-aligned bo

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 11:21 AM Dilip Kumar wrote: > If that is only the argument then it's possible today as well. I mean > you can INSERT INTO .. SELECT FROM where source attribute as > compressed data but the target attribute as external storage then also > we will move the compressed data as

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-16 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > > >> I think e.g. prefetch_targblock could be moved to the next #ifdef, which > > >> will eliminate the one-line ifdef. > > > > > > Sure, done in the attached. > > > > > > Thanks for the review! Unless there's other comments, I'll plan to

Re: Calendar support in localization

2021-03-16 Thread Thomas Munro
On Wed, Mar 17, 2021 at 6:31 AM Surafel Temesgen wrote: > Ethiopice calendar have 13 months so it can not be stored as date and > timestamp type and you approach seems more complicated and i suggest to have > this feature on the purpose of PostgreSQL popularities too not only for my > need I k

Re: pg_amcheck contrib application

2021-03-16 Thread Andrew Dunstan
On 3/13/21 1:30 AM, Andres Freund wrote: > Hi, > > On 2021-03-13 01:22:54 -0500, Tom Lane wrote: >> Mark Dilger writes: >>> On Mar 12, 2021, at 10:16 PM, Noah Misch wrote: hoverfly does configure with PERL=perl64. /usr/bin/prove is from the 32-bit Perl, so I suspect the TAP sui

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 2:54 PM Andres Freund wrote: > Oh, I guess it would make sense to do it that way. However, I was just > thinking of doing the iteration over the tuples that ExecEvalRow() etc > do inside heap_form_flattened_tuple() (or whatever). That'd not be any > worse than what the patc

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 11:40 AM, Robert Haas wrote: > > On Tue, Mar 16, 2021 at 2:22 PM Tom Lane wrote: >> I'm circling back around to the idea that amcheck is trying to >> validate TOAST references that are already dead, and it's getting >> burnt because something-or-other has already removed

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Mon, Mar 15, 2021 at 10:10 PM Mark Dilger wrote: > It is unfortunate that the failing test only runs pg_amcheck after creating > numerous corruptions, as we can't know if pg_amcheck would have complained > about pg_statistic before the corruptions were created in other tables, or if > it onl

Re: shared-memory based stats collector

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 15:08:39 -0400, Robert Haas wrote: > On Mon, Mar 15, 2021 at 10:56 PM Andres Freund wrote: > > I did roughly the first steps of the split as I had outlined. I moved: > > > > 1) wait event related functions into utils/activity/wait_event.c / > >wait_event.h > > > > 2) "backen

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-16 Thread Andres Freund
Hi, On 2021-03-05 21:35:59 -0300, Alvaro Herrera wrote: > I'll take the weekend to think about the issue with conn->last_query and > conn->queryclass that I mentioned yesterday; other than that detail my > feeling is that this is committable, so I'll be looking at getting this > pushed early next

Re: Boundary value check in lazy_tid_reaped()

2021-03-16 Thread Peter Geoghegan
On Sun, Mar 14, 2021 at 4:22 PM Thomas Munro wrote: > BTW I got around to trying this idea out for a specialised > bsearch_itemptr() using a wide comparator, over here: Cool! I have another thing that should be considered when we revisit this area in the future: maybe we should structure the bin

Re: Boundary value check in lazy_tid_reaped()

2021-03-16 Thread Hannu Krosing
We could also go parallel in another direction - I have been mulling about writing a "vectorized" bsearch which would use AVX2, where you look up 64 (or more likely 32, so tids also fit in 256bit vector) tids at a time. The trickiest part is that the search can complete at different iteration for

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 6:40 AM Masahiko Sawada wrote: > > Note that I've merged multiple existing functions in vacuumlazy.c into > > one: the patch merges lazy_vacuum_all_indexes() and lazy_vacuum_heap() > > into a single function named vacuum_indexes_mark_unused() > I agree to create a function

RE: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Masahiro Ikeda
On 2021-03-16 13:44, kuroda.hay...@fujitsu.com wrote: Dear Ikeda-san I think the idea is good. I read the patch and other sources, and I found process_startup_packet_die also execute _exit(1). I think they can be combined into one function and moved to interrupt.c, but some important comments

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Zhihong Yu
Hi, + * Simple signal handler for processes HAVE NOT yet touched or DO NOT I think there should be a 'which' between processes and HAVE. It seems the words in Capital letters should be in lower case. + * Simple signal handler for processes have touched shared memory to + * exit quickly. Add 'wh

Re: pg_subscription - substream column?

2021-03-16 Thread Peter Smith
On Wed, Mar 17, 2021 at 12:45 AM Amit Kapila wrote: > > > Attached, please find the patch to update the description of substream > in pg_subscription. > I applied your patch and regenerated the PG docs to check the result. LGTM. Kind Regards, Peter Smith. Fujitsu Australia

Getting better results from valgrind leak tracking

2021-03-16 Thread Tom Lane
[ starting a new thread for this ] Andres Freund writes: > I wonder if it'd be worth starting to explicitly annotate all the places > that do allocations and are fine with leaking them. E.g. by introducing > malloc_permanently() or such. Right now it's hard to use valgrind et al > to detect leaks

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Masahiro Ikeda
On 2021-03-17 08:25, Zhihong Yu wrote: Hi, Thanks for your comments! + * Simple signal handler for processes HAVE NOT yet touched or DO NOT I think there should be a 'which' between processes and HAVE. It seems the words in Capital letters should be in lower case. + * Simple signal handler

  1   2   >