Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Peter Smith
On Fri, Mar 12, 2021 at 8:38 PM vignesh C wrote: > ... > 1) I felt twophase_given can be a local variable, it need not be added > as a function parameter as it is not used outside the function. > --- a/src/backend/commands/subscriptioncmds.c > +++ b/src/backend/commands/subscriptioncmds.c > @@ -67

Re: Type of wait events WalReceiverWaitStart and WalSenderWaitForWAL

2021-03-16 Thread Kyotaro Horiguchi
At Wed, 17 Mar 2021 15:31:37 +0900 (JST), Kyotaro Horiguchi wrote in > WAIT_EVENT_WAL_RECEIVER_MAIN(WalReceiverMain) is waiting for new data > to arrive. This looks like an activity to me. > > WAIT_EVENT_WAL_RECEIVER_WAIT_START is waiting for waiting for starup > process to kick me. So it may

Re: Type of wait events WalReceiverWaitStart and WalSenderWaitForWAL

2021-03-16 Thread Kyotaro Horiguchi
At Tue, 16 Mar 2021 15:42:27 +0900, Fujii Masao wrote in > > > On 2021/03/16 11:59, Kyotaro Horiguchi wrote: > > At Tue, 16 Mar 2021 03:12:54 +0900, Fujii Masao > > wrote in > >> The wait event WalReceiverWaitStart has been categorized in the type > >> Client. > >> But why? Walreceiver is wai

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-16 Thread Julien Rouhaud
On Wed, Mar 17, 2021 at 05:30:30AM +0100, Pavel Stehule wrote: > st 17. 3. 2021 v 4:52 odesílatel Michael Paquier > > > I am wondering whether it would be better to allow multiple aliases > > though, and if it would bring more readability to the routines written > > if these are treated equal to

Re: Assertion failure with barriers in parallel hash join

2021-03-16 Thread Thomas Munro
On Wed, Mar 17, 2021 at 6:17 PM Thomas Munro wrote: > On Sat, Mar 6, 2021 at 9:56 AM Thomas Munro wrote: > > While working on Melanie's Parallel Hash Full Join patch I remembered > > that this (apparently extremely rare) race still needs fixing. Here > > is a slightly tidied version, which I'm a

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 5:03 PM Amit Kapila wrote: > > On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > > > Here's a new patch-set that implements this new solution proposed by Amit. > > Patchset-v60 implements: > > > > I have reviewed the latest patch and below are my comments, some of >

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > What I meant was that I didn't understand how there's not a leak > danger when compilation fails halfway through, given that the context > in question is below TopMemoryContext and that I didn't see a relevant > TRY block. But that probably

Re: PATCH: Attempt to make dbsize a bit more consistent

2021-03-16 Thread Michael Paquier
On Mon, Mar 15, 2021 at 03:10:59PM +0900, Michael Paquier wrote: > Anyway, as mentioned by other people upthread, I am not really > convinced either that we should have more flavors of size functions, > particularly depending on the relkind as this would be confusing for > the end-user. pg_relatio

Re: libpq debug log

2021-03-16 Thread Kyotaro Horiguchi
At Wed, 17 Mar 2021 02:09:32 +, "tsunakawa.ta...@fujitsu.com" wrote in > Alvaro-san, > > > Thank you for taking your time to take a look at an incomplete patch. I > thought I would ask you for final check for commit after Iwata-san has > reflected my review comments. > > I discussed wi

Re: Assertion failure with barriers in parallel hash join

2021-03-16 Thread Thomas Munro
On Sat, Mar 6, 2021 at 9:56 AM Thomas Munro wrote: > While working on Melanie's Parallel Hash Full Join patch I remembered > that this (apparently extremely rare) race still needs fixing. Here > is a slightly tidied version, which I'm adding to the next CF for CI > coverage. Pushed and back-patc

Re: Regression tests vs SERIALIZABLE

2021-03-16 Thread Tom Lane
Thomas Munro writes: > On Tue, Mar 16, 2021 at 3:28 AM Tom Lane wrote: >> Usually, if we issue a SET in the regression tests, we explicitly RESET >> as soon thereafter as practical, so as to have a well-defined scope >> where the script is running under unusual conditions. > Oh, of course. Than

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-16 Thread Pavel Stehule
st 17. 3. 2021 v 4:52 odesílatel Michael Paquier napsal: > On Mon, Mar 15, 2021 at 05:31:18AM +0100, Pavel Stehule wrote: > > Thank you very much > > I am not much a fan of the implementation done here. Based on my > understanding of the PL code, the namespace lookup is organized as a > list of

Re: Regression tests vs SERIALIZABLE

2021-03-16 Thread Thomas Munro
On Tue, Mar 16, 2021 at 3:28 AM Tom Lane wrote: > Thomas Munro writes: > > On Mon, Mar 15, 2021 at 5:24 PM Thomas Munro wrote: > >> However, since commit 862ef372d6b, there *is* one test that fails if > >> you run make installcheck against a cluster running with -c > >> default_transaction_isola

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Tom Lane
"Andres Freund" writes: > On Tue, Mar 16, 2021, at 20:01, Tom Lane wrote: >> That seems both unnecessary and impractical. We have to consider that >> everything-still-reachable is an OK final state. > I don't consider "still reachable" a leak. Just definitely unreachable. OK, we're in violent a

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-16 Thread Michael Paquier
On Mon, Mar 15, 2021 at 05:31:18AM +0100, Pavel Stehule wrote: > Thank you very much I am not much a fan of the implementation done here. Based on my understanding of the PL code, the namespace lookup is organized as a list of items, with the namespace associated to the routine name being the fir

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Andres Freund
Hi, On Tue, Mar 16, 2021, at 20:01, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-16 19:36:10 -0400, Tom Lane wrote: > >> It doesn't appear to me that we leak much at all, at least not if you > >> are willing to take "still reachable" blocks as not-leaked. > > > Well, I think for any so

Re: fdatasync performance problem with large number of DB files

2021-03-16 Thread Thomas Munro
On Tue, Mar 16, 2021 at 9:29 PM Fujii Masao wrote: > On 2021/03/16 8:15, Thomas Munro wrote: > > I don't want to add a hypothetical sync_after_crash=none, because it > > seems like generally a bad idea. We already have a > > running-with-scissors mode you could use for that: fsync=off. > > I hear

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Wed, Mar 17, 2021 at 8:07 AM vignesh C wrote: > > On Tue, Mar 16, 2021 at 7:22 PM Amit Kapila wrote: > > > > On Tue, Mar 16, 2021 at 6:22 PM vignesh C wrote: > > > > > > On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > > > > > > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila > > > >

Re: subscriptionCheck failures

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 6:22 PM osumi.takami...@fujitsu.com wrote: > > > To me, this correctly works because > the timing I put the while loop and stops the walsender > makes the DROP SUBSCRIPTION affects two slots. Any comments ? > No, your testing looks fine. I have also done the similar test.

Re: pg_subscription - substream column?

2021-03-16 Thread Amit Kapila
On Wed, Mar 17, 2021 at 4:56 AM Peter Smith wrote: > > On Wed, Mar 17, 2021 at 12:45 AM Amit Kapila wrote: > > > > > > Attached, please find the patch to update the description of substream > > in pg_subscription. > > > > I applied your patch and regenerated the PG docs to check the result. > > L

Re: fdatasync performance problem with large number of DB files

2021-03-16 Thread Thomas Munro
On Tue, Mar 16, 2021 at 9:10 PM Fujii Masao wrote: > Thanks for the patch! > > +When set to fsync, which is the default, > +PostgreSQL will recursively open and fsync > +all files in the data directory before crash recovery begins. > > Isn't this a bit misleading? This may

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Tom Lane
Andres Freund writes: > On 2021-03-16 19:36:10 -0400, Tom Lane wrote: >> It doesn't appear to me that we leak much at all, at least not if you >> are willing to take "still reachable" blocks as not-leaked. > Well, I think for any sort of automated testing - which I think would be > useful - we'd

Re: A new function to wait for the backend exit after termination

2021-03-16 Thread Kyotaro Horiguchi
At Wed, 17 Mar 2021 07:01:39 +0530, Bharath Rupireddy wrote in > Attaching v10 patch for further review. The time-out mechanism doesn't count remainingtime as expected, concretely it does the following. do { kill(); WaitLatch(WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, waittime); Re

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Zhihong Yu
Hi, For the second last trace involving SearchCatCacheList (catcache.c:1691), the ctlist's members are stored in cl->members array where cl is returned at the end of SearchCatCacheList. Maybe this was not accounted for by valgrind ? Cheers On Tue, Mar 16, 2021 at 7:31 PM Andres Freund wrote: >

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread vignesh C
On Tue, Mar 16, 2021 at 7:22 PM Amit Kapila wrote: > > On Tue, Mar 16, 2021 at 6:22 PM vignesh C wrote: > > > > On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > > > > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila > > > wrote: > > >> > > > > 2) table_states_not_ready global variable is u

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-03-16 Thread torikoshia
On 2021-03-16 20:51, Bharath Rupireddy wrote: On Mon, Mar 15, 2021 at 11:23 AM torikoshia wrote: On 2021-03-07 19:16, Bharath Rupireddy wrote: > On Fri, Feb 5, 2021 at 5:15 PM Bharath Rupireddy > wrote: >> >> pg_terminate_backend and pg_cancel_backend with postmaster PID produce >> "PID

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Andres Freund
Hi, David, there's a question about a commit of yours below, hence adding you. On 2021-03-16 19:36:10 -0400, Tom Lane wrote: > Out of curiosity I poked at this for a little while. Cool. > It doesn't appear to me that we leak much at all, at least not if you > are willing to take "still reachab

RE: libpq debug log

2021-03-16 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Thank you for taking your time to take a look at an incomplete patch. I thought I would ask you for final check for commit after Iwata-san has reflected my review comments. I discussed with Iwata-sn your below comments. Let me convey her opinions. (She is now focusing on fixing

Re: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2021-03-16 Thread Thomas Munro
On Tue, Mar 16, 2021 at 2:41 PM Thomas Munro wrote: > On Mon, Mar 15, 2021 at 8:25 PM Bharath Rupireddy > wrote: > > > > The problem with a case like REFRESH MATERIALIZED VIEW is that there's > > > > nothing to prevent something that gets run in the course of the query > > > > from trying to acce

Re: crash during cascaded foreign key update

2021-03-16 Thread Amit Langote
On Tue, Mar 16, 2021 at 11:17 PM Tom Lane wrote: > Amit Langote writes: > > With HEAD (I think v12 and greater), I see $subject when trying out > > the following scenario: > > I wonder if this is related to > > https://www.postgresql.org/message-id/flat/89429.1584443208%40antos > > which we've st

Re: cleanup temporary files after crash

2021-03-16 Thread Euler Taveira
On Sun, Mar 14, 2021, at 11:01 PM, Thomas Munro wrote: > On Wed, Mar 10, 2021 at 1:31 AM Michael Paquier wrote: > > On Tue, Mar 09, 2021 at 02:28:43AM +0100, Tomas Vondra wrote: > > > Let's move this patch forward. Based on the responses, I agree the > > > default behavior should be to remove the

Re: A new function to wait for the backend exit after termination

2021-03-16 Thread Bharath Rupireddy
On Tue, Mar 16, 2021 at 9:48 PM Magnus Hagander wrote: > Does it really make sense that pg_wait_for_backend_termination() > defaults to waiting *100 milliseconds*, and then logs a warning? That > seems extremely short if I'm explicitly asking it to wait. I increased the default wait timeout to 5s

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Masahiro Ikeda
On 2021-03-17 08:25, Zhihong Yu wrote: Hi, Thanks for your comments! + * Simple signal handler for processes HAVE NOT yet touched or DO NOT I think there should be a 'which' between processes and HAVE. It seems the words in Capital letters should be in lower case. + * Simple signal handler

Getting better results from valgrind leak tracking

2021-03-16 Thread Tom Lane
[ starting a new thread for this ] Andres Freund writes: > I wonder if it'd be worth starting to explicitly annotate all the places > that do allocations and are fine with leaking them. E.g. by introducing > malloc_permanently() or such. Right now it's hard to use valgrind et al > to detect leaks

Re: pg_subscription - substream column?

2021-03-16 Thread Peter Smith
On Wed, Mar 17, 2021 at 12:45 AM Amit Kapila wrote: > > > Attached, please find the patch to update the description of substream > in pg_subscription. > I applied your patch and regenerated the PG docs to check the result. LGTM. Kind Regards, Peter Smith. Fujitsu Australia

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Zhihong Yu
Hi, + * Simple signal handler for processes HAVE NOT yet touched or DO NOT I think there should be a 'which' between processes and HAVE. It seems the words in Capital letters should be in lower case. + * Simple signal handler for processes have touched shared memory to + * exit quickly. Add 'wh

RE: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-16 Thread Masahiro Ikeda
On 2021-03-16 13:44, kuroda.hay...@fujitsu.com wrote: Dear Ikeda-san I think the idea is good. I read the patch and other sources, and I found process_startup_packet_die also execute _exit(1). I think they can be combined into one function and moved to interrupt.c, but some important comments

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 6:40 AM Masahiko Sawada wrote: > > Note that I've merged multiple existing functions in vacuumlazy.c into > > one: the patch merges lazy_vacuum_all_indexes() and lazy_vacuum_heap() > > into a single function named vacuum_indexes_mark_unused() > I agree to create a function

Re: Boundary value check in lazy_tid_reaped()

2021-03-16 Thread Hannu Krosing
We could also go parallel in another direction - I have been mulling about writing a "vectorized" bsearch which would use AVX2, where you look up 64 (or more likely 32, so tids also fit in 256bit vector) tids at a time. The trickiest part is that the search can complete at different iteration for

Re: Boundary value check in lazy_tid_reaped()

2021-03-16 Thread Peter Geoghegan
On Sun, Mar 14, 2021 at 4:22 PM Thomas Munro wrote: > BTW I got around to trying this idea out for a specialised > bsearch_itemptr() using a wide comparator, over here: Cool! I have another thing that should be considered when we revisit this area in the future: maybe we should structure the bin

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-16 Thread Andres Freund
Hi, On 2021-03-05 21:35:59 -0300, Alvaro Herrera wrote: > I'll take the weekend to think about the issue with conn->last_query and > conn->queryclass that I mentioned yesterday; other than that detail my > feeling is that this is committable, so I'll be looking at getting this > pushed early next

Re: shared-memory based stats collector

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 15:08:39 -0400, Robert Haas wrote: > On Mon, Mar 15, 2021 at 10:56 PM Andres Freund wrote: > > I did roughly the first steps of the split as I had outlined. I moved: > > > > 1) wait event related functions into utils/activity/wait_event.c / > >wait_event.h > > > > 2) "backen

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Mon, Mar 15, 2021 at 10:10 PM Mark Dilger wrote: > It is unfortunate that the failing test only runs pg_amcheck after creating > numerous corruptions, as we can't know if pg_amcheck would have complained > about pg_statistic before the corruptions were created in other tables, or if > it onl

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 11:40 AM, Robert Haas wrote: > > On Tue, Mar 16, 2021 at 2:22 PM Tom Lane wrote: >> I'm circling back around to the idea that amcheck is trying to >> validate TOAST references that are already dead, and it's getting >> burnt because something-or-other has already removed

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 2:54 PM Andres Freund wrote: > Oh, I guess it would make sense to do it that way. However, I was just > thinking of doing the iteration over the tuples that ExecEvalRow() etc > do inside heap_form_flattened_tuple() (or whatever). That'd not be any > worse than what the patc

Re: pg_amcheck contrib application

2021-03-16 Thread Andrew Dunstan
On 3/13/21 1:30 AM, Andres Freund wrote: > Hi, > > On 2021-03-13 01:22:54 -0500, Tom Lane wrote: >> Mark Dilger writes: >>> On Mar 12, 2021, at 10:16 PM, Noah Misch wrote: hoverfly does configure with PERL=perl64. /usr/bin/prove is from the 32-bit Perl, so I suspect the TAP sui

Re: Calendar support in localization

2021-03-16 Thread Thomas Munro
On Wed, Mar 17, 2021 at 6:31 AM Surafel Temesgen wrote: > Ethiopice calendar have 13 months so it can not be stored as date and > timestamp type and you approach seems more complicated and i suggest to have > this feature on the purpose of PostgreSQL popularities too not only for my > need I k

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-16 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > > >> I think e.g. prefetch_targblock could be moved to the next #ifdef, which > > >> will eliminate the one-line ifdef. > > > > > > Sure, done in the attached. > > > > > > Thanks for the review! Unless there's other comments, I'll plan to

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 11:21 AM Dilip Kumar wrote: > If that is only the argument then it's possible today as well. I mean > you can INSERT INTO .. SELECT FROM where source attribute as > compressed data but the target attribute as external storage then also > we will move the compressed data as

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 2:45 PM Tom Lane wrote: > > In what context is it OK to just add extra alignment padding? > > It's *not* extra, according to pg_statistic's tuple descriptor. > Both forming and deforming of pg_statistic tuples should honor > that and locate stavaluesX values on d-aligned bo

Re: shared-memory based stats collector

2021-03-16 Thread Robert Haas
On Mon, Mar 15, 2021 at 10:56 PM Andres Freund wrote: > I did roughly the first steps of the split as I had outlined. I moved: > > 1) wait event related functions into utils/activity/wait_event.c / >wait_event.h > > 2) "backend status" functionality (PgBackendStatus stuff) into >utils/acti

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 10:27:12 -0400, Robert Haas wrote: > > I'm don't like that after 0002 ExecEvalRow(), ExecEvalFieldStoreForm() > > contain a nearly identical copy of the same code. And > > make_tuple_from_row() also is similar. It seem that there should be a > > heap_form_tuple() version doing t

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Mark Dilger writes: > CopyStatistics seems to just copy Form_pg_statistic without regard for > who owns the toast. Is this safe? No less so than a ton of other places that insert values that might've come from on-disk storage. heap_toast_insert_or_update() is responsible for dealing with the pr

Re: "has_column_privilege()" issue with attnums and non-existent columns

2021-03-16 Thread Joe Conway
On 3/16/21 1:42 AM, Chengxi Sun wrote: The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested I tested the patch and it wo

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 16, 2021 at 1:48 PM Tom Lane wrote: >> No. What should be happening there is that some arrays in the column >> get larger alignment than they actually need, but that shouldn't cause >> a problem (and has not done so, AFAIK, in the decades that it's been >> like

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 2:22 PM Tom Lane wrote: > I'm circling back around to the idea that amcheck is trying to > validate TOAST references that are already dead, and it's getting > burnt because something-or-other has already removed the toast > rows, though not the referencing datums. That's l

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:20 AM Avinash Kumar wrote: > I can share any detail that would help here. I would like to know what you see when you run a slightly modified version of the same amcheck query. The same query as before, but with the call to bt_index_parent_check() replaced with a call to

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 10:48 AM, Tom Lane wrote: > > I'm not entirely sure what's going on, but I think coming at this > with the mindset that "amcheck has detected some corruption" is > just going to lead you astray. Almost certainly, it's "amcheck > is incorrectly claiming corruption". That

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 1:48 PM Tom Lane wrote: > No. What should be happening there is that some arrays in the column > get larger alignment than they actually need, but that shouldn't cause > a problem (and has not done so, AFAIK, in the decades that it's been > like this). As you say, deformi

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 11:08 AM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would run towards a bug in WAL log c

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Mark Dilger writes: > On Mar 16, 2021, at 10:48 AM, Tom Lane wrote: >> (Too bad the message doesn't report the >> TOAST OID it probed for, so we can see if that's sane or not.) > I've added that and now get the toast pointer's va_valueid in the message: > heap table "postgres"."pg_catalog"."pg_

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
On Tue, Mar 16, 2021 at 3:08 PM Tom Lane wrote: > Peter Geoghegan writes: > > ... It's hard to believe that the problem is > > squarely with _bt_swap_posting(). > > IIUC, the problem is seen on a replica server but not the primary? > In that case, my thoughts would run towards a bug in WAL log c

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Tom Lane
Peter Geoghegan writes: > ... It's hard to believe that the problem is > squarely with _bt_swap_posting(). IIUC, the problem is seen on a replica server but not the primary? In that case, my thoughts would run towards a bug in WAL log creation or replay, causing the index contents to be different

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 10:48 AM, Tom Lane wrote: > > Robert Haas writes: >> On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger >> wrote: >>> It shows them all has having attalign = 'd', but for some array types the >>> alignment will be 'i', not 'd'. So it's lying a bit about the contents. >>>

Re: [Proposal] Global temporary tables

2021-03-16 Thread Pavel Stehule
Hi this patch is broken now. Please, can you check it? Regards Pavel st 25. 11. 2020 v 14:08 odesílatel 曾文旌 napsal: > > > 2020年11月25日 14:19,Pavel Stehule 写道: > > > > po 23. 11. 2020 v 10:27 odesílatel 曾文旌 > napsal: > >> >> >> 2020年11月21日 02:28,Pavel Stehule 写道: >> >> Hi >> >> pá 11. 9. 20

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
... btw, I now see that tern and hornet are passing this test at least as much as they're failing, proving that there's some timing or random chance involved. That doesn't completely eliminate the idea that there may be an architecture component to the issue, but it sure reduces its credibility.

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger > wrote: >> It shows them all has having attalign = 'd', but for some array types the >> alignment will be 'i', not 'd'. So it's lying a bit about the contents. >> But I'm now confused why this caused problems on the two hosts

Re: Calendar support in localization

2021-03-16 Thread Surafel Temesgen
Hi Thomas On Mon, Mar 15, 2021 at 2:58 PM Thomas Munro wrote: > > One key question here is whether you need a different date type or > just different operations (functions, operators etc) on the existing > types. > > I am thinking of having a converter to a specific calendar after each operation

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 9:50 AM Avinash Kumar wrote: > Yes, it was on the failover-over server where the issue is currently seen. > Took a snapshot of the data directory so that the issue can be analyzed. I would be very cautious when using LVM snapshots with a Postgres data directory, or VM-bas

Re: pg_amcheck contrib application

2021-03-16 Thread Robert Haas
On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger wrote: > Yeah, that looks related: > > regression=# select attname, attlen, attnum, attalign from pg_attribute where > attrelid = 2619 and attname like 'stavalue%'; > attname | attlen | attnum | attalign > +++--

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 9:30 AM, Mark Dilger wrote: > > > >> On Mar 16, 2021, at 9:07 AM, Tom Lane wrote: >> >> Mark Dilger writes: >>> I think autovacuum simply triggers the bug, and is not the cause of the >>> bug. If I turn autovacuum off and instead do an ANALYZE in each test >>> data

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
Hi, On Tue, Mar 16, 2021 at 1:44 PM Peter Geoghegan wrote: > On Tue, Mar 16, 2021 at 5:01 AM Avinash Kumar > wrote: > > I am afraid that it looks to me like a deduplication bug but not sure > how this can be pin-pointed. If there is something I could do to determine > that, I would be more tha

Re: [PATCH] pgbench: improve \sleep meta command

2021-03-16 Thread Fujii Masao
On 2021/03/09 0:54, Fujii Masao wrote: On 2021/03/08 23:10, Alvaro Herrera wrote: On 2021-Mar-08, kuroda.hay...@fujitsu.com wrote: Dear Fujii-san, Miyake-san Isn't it better to accept even negative sleep time like currently pgbench does? Otherwise we always need to check the variable is

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Peter Geoghegan
On Tue, Mar 16, 2021 at 5:01 AM Avinash Kumar wrote: > I am afraid that it looks to me like a deduplication bug but not sure how > this can be pin-pointed. If there is something I could do to determine that, > I would be more than happy. That cannot be ruled out, but I don't consider it to be t

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 16, 2021, at 9:07 AM, Tom Lane wrote: > > Mark Dilger writes: >> I think autovacuum simply triggers the bug, and is not the cause of the bug. >> If I turn autovacuum off and instead do an ANALYZE in each test database >> rather than performing the corruptions, I get reports about

Re: A new function to wait for the backend exit after termination

2021-03-16 Thread Magnus Hagander
On Tue, Mar 16, 2021 at 10:38 AM Bharath Rupireddy wrote: > > On Mon, Mar 15, 2021 at 10:38 AM Fujii Masao > wrote: > > On 2021/03/15 12:27, Bharath Rupireddy wrote: > > > On Sun, Mar 7, 2021 at 2:39 PM Bharath Rupireddy > > > wrote: > > >> Attaching v7 patch for further review. > > > > > > Atta

Re: pg_amcheck contrib application

2021-03-16 Thread Tom Lane
Mark Dilger writes: > I think autovacuum simply triggers the bug, and is not the cause of the bug. > If I turn autovacuum off and instead do an ANALYZE in each test database > rather than performing the corruptions, I get reports about problems in > pg_statistic. This is on my mac laptop. Th

Re: pg_amcheck contrib application

2021-03-16 Thread Mark Dilger
> On Mar 15, 2021, at 11:09 PM, Noah Misch wrote: > >> Not sure that I believe the theory that this is from bad luck of >> concurrent autovacuum timing, though. > > With autovacuum_naptime=1s, on hornet, the failure reproduced twice in twelve > runs. With v6-0001-Turning-off-autovacuum-durin

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Masahiko Sawada
On Tue, Mar 16, 2021 at 10:39 PM Masahiko Sawada wrote: > > On Mon, Mar 15, 2021 at 11:04 AM Peter Geoghegan wrote: > > > > One consequence of my approach is that we now call > > lazy_cleanup_all_indexes(), even when we've skipped index vacuuming > > itself. We should at least "check-in" with the

Re: pg_stat_statements oddity with track = all

2021-03-16 Thread Julien Rouhaud
On Tue, Mar 16, 2021 at 12:55:45PM +0100, Magnus Hagander wrote: > On Tue, Mar 9, 2021 at 3:39 AM Julien Rouhaud wrote: > > > > I think that we might be able to handle that without a flag. The only thing > > that would need to be done is when creating an entry, look for an existing > > entry with

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Dilip Kumar
On Tue, Mar 16, 2021 at 7:57 PM Robert Haas wrote: > > That behavior feels unacceptable to me from a user expectations point > of view. I think there's an argument that if I update a tuple that > contains a compressed datum, and I don't update that particular > column, I think it would be OK to n

Re: libpq debug log

2021-03-16 Thread Alvaro Herrera
On 2021-Mar-15, iwata@fujitsu.com wrote: > I create protocol message reading function for each protocol message type. > (Ex. pqTraceOutputB() read Bind message) > This makes the nesting shallower and makes the code easier to read. I'm not sure I agree with this structural change. Yes, it is

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-03-16 Thread Tom Lane
Justin Pryzby writes: > I think it's somewhat confusing that there's two callbacks. > The first one applies only during typinput/typreceive. > I guess the 2nd one should say that they're printed "in *future errors". I adjusted the comments to make this a bit clearer, and pushed it.

Re: GROUP BY DISTINCT

2021-03-16 Thread Tomas Vondra
On 3/16/21 9:21 AM, Vik Fearing wrote: > On 3/13/21 12:33 AM, Tomas Vondra wrote: >> Hi Vik, >> >> The patch seems quite ready, I have just two comments. > > Thanks for taking a look. > >> 1) Shouldn't this add another for DISTINCT, somewhere in the >> documentation? Now the index points just

re: crash during cascaded foreign key update

2021-03-16 Thread Ranier Vilela
>0 0x7f747e6e2387 in raise () from /lib64/libc.so.6 >#1 0x7f747e6e3a78 in abort () from /lib64/libc.so.6 >#2 0x00ae056a in ExceptionalCondition ( >conditionName=0xb67c10 "!ItemPointerEquals(&oldtup.t_self, >&oldtup.t_data->t_ctid)", >errorType=0xb66d89 "FailedAssertion", fileName=0x

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Robert Haas
On Mon, Mar 15, 2021 at 6:58 PM Andres Freund wrote: > I don't particularly like PG_RETURN_HEAPTUPLEHEADER_RAW(). What is "raw" > about it? It also seems to me like there needs to at least be a > sentence or two explaining when to use which of the functions. It seemed like the natural name to me

Re: crash during cascaded foreign key update

2021-03-16 Thread Tom Lane
Amit Langote writes: > With HEAD (I think v12 and greater), I see $subject when trying out > the following scenario: I wonder if this is related to https://www.postgresql.org/message-id/flat/89429.1584443208%40antos which we've still not done anything about. regards, to

Re: Nondeterministic collations and the value returned by GROUP BY x

2021-03-16 Thread Tom Lane
Jim Finnerty writes: > right. It doesn't matter which of the values is returned; however, a > plausible-sounding implementation would case-fold the value, like GROUP BY > LOWER(x), but the case-folded value isn't necessarily one of the original > values and so that could be subtly wrong in the ca

crash during cascaded foreign key update

2021-03-16 Thread Amit Langote
With HEAD (I think v12 and greater), I see $subject when trying out the following scenario: -- in backend 1 create table p (a int primary key); create table f (a int references p on update cascade deferrable initially deferred); insert into p values (1); begin isolation level serializable; insert

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 6:22 PM vignesh C wrote: > > On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: > >> > > 2) table_states_not_ready global variable is used immediately after > call to FetchTableStates, we can make FetchTableState

Re: pg_subscription - substream column?

2021-03-16 Thread Amit Kapila
On Tue, Mar 16, 2021 at 5:27 PM Peter Smith wrote: > > On Tue, Mar 16, 2021 at 7:20 PM Amit Kapila wrote: > > > > On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > > > > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't > > > have any mention of the substream column.

Re: New IndexAM API controlling index vacuum strategies

2021-03-16 Thread Masahiko Sawada
On Mon, Mar 15, 2021 at 11:04 AM Peter Geoghegan wrote: > > On Thu, Mar 11, 2021 at 8:31 AM Robert Haas wrote: > > But even if not, I'm not sure this > > helps much with the situation you're concerned about, which involves > > non-HOT tuples. > > Attached is a POC-quality revision of Masahiko's >

Re: simplifying foreign key/RI checks

2021-03-16 Thread Amit Langote
On Mon, Mar 8, 2021 at 11:41 PM Amit Langote wrote: > On Thu, Mar 4, 2021 at 5:15 AM Tom Lane wrote: > > Lastly, ri_PerformCheck is pretty careful about not only which > > snapshot it uses, but which *pair* of snapshots it uses, because > > sometimes it needs to worry about data changes since the

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Dilip Kumar
On Tue, Mar 16, 2021 at 4:07 PM Dilip Kumar wrote: > > INSERT TIME > Head: 17418.299 ms Patch: 20956.231 ms > > CTAS TIME: > Head: 12837.872 ms Patch: 16775.739 ms On quick analysis with perf it appeared that the performance is degrading because of deforming - 16.19% 3.54% postgres postg

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-16 Thread vignesh C
On Mon, Mar 15, 2021 at 6:14 PM Ajin Cherian wrote: > > On Mon, Mar 15, 2021 at 2:04 PM Amit Kapila wrote: >> >> I think something on these lines should be much >> easier than the spool-file implementation unless we see any problem >> with this idea. >> > > Here's a new patch-set that implements

RE: subscriptionCheck failures

2021-03-16 Thread osumi.takami...@fujitsu.com
Hi On Tuesday, March 16, 2021 4:15 PM vignesh C wrote: > On Tue, Mar 16, 2021 at 12:29 PM Amit Kapila > wrote: > > > > On Tue, Mar 16, 2021 at 9:00 AM Amit Kapila > wrote: > > > > > > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro > wrote: > > > > > > > > Hi, > > > > > > > > This seems to be a

Re: dynamic result sets support in extended query protocol

2021-03-16 Thread Peter Eisentraut
On 15.03.21 14:56, David Steele wrote: Hi Peter, On 12/30/20 9:33 AM, Peter Eisentraut wrote: On 2020-10-09 20:46, Andres Freund wrote: Is there really a good reason for forcing the client to issue NextResult, Describe, Execute for each of the dynamic result sets? It's not like there's really

Re: EXPLAIN/EXPLAIN ANALYZE REFRESH MATERIALIZED VIEW

2021-03-16 Thread Bharath Rupireddy
On Tue, Mar 16, 2021 at 1:15 AM Tom Lane wrote: > > [ Sorry for not looking at this thread sooner ] > > Bharath Rupireddy writes: > > Currently, $subject is not allowed. We do plan the mat view query > > before every refresh. I propose to show the explain/explain analyze of > > the select part of

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-16 Thread Avinash Kumar
On Mon, Mar 15, 2021 at 3:21 PM Avinash Kumar wrote: > Hi, > > On Mon, Mar 15, 2021 at 1:18 PM Peter Geoghegan wrote: > >> On Mon, Mar 15, 2021 at 6:56 AM Avinash Kumar >> wrote: >> > psql:amchecksql.sql:17: DEBUG: leaf block 1043751 of index >> "idx_id_mtime" has no first data item >> >> That

Re: pg_subscription - substream column?

2021-03-16 Thread Peter Smith
On Tue, Mar 16, 2021 at 7:20 PM Amit Kapila wrote: > > On Tue, Mar 16, 2021 at 3:35 AM Peter Smith wrote: > > > > I noticed that the PG docs [1] for the catalog pg_subscription doesn't > > have any mention of the substream column. > > > > Accidental omission by commit 4648243 [2] from last year?

  1   2   >