Re: [BUG] Error in BRIN summarization

2020-08-12 Thread Alvaro Herrera
On 2020-Jul-28, Peter Geoghegan wrote: > On Mon, Jul 27, 2020 at 10:25 AM Alvaro Herrera > wrote: > > (I was also considering whether it needs to be a loop to reobtain root > > tuples, in case a concurrent transaction can create a new item while > > we're checking

Re: [BUG] Error in BRIN summarization

2020-08-12 Thread Alvaro Herrera
On 2020-Aug-11, Alvaro Herrera wrote: > I think this is more complicated than necessary. It seems easier to > solve this problem by just checking whether the given root pointer is > set to InvalidOffsetNumber, which is already done in the existing coding > of heap_get_root_tuple

Re: [BUG] Error in BRIN summarization

2020-08-12 Thread Alvaro Herrera
On 2020-Aug-11, Alvaro Herrera wrote: > A much more troubling thought is what happens if the range is > desummarized, then the index item is used for the summary of a different > range. Then the index might end up returning corrupt results. Actually, this is not a concern because

Re: [BUG] Error in BRIN summarization

2020-08-12 Thread Alvaro Herrera
On 2020-Aug-12, Alvaro Herrera wrote: > 'anyvisible' mode is not required AFAICS; reading the code, I think this > could also hit REINDEX CONCURRENTLY and CREATE INDEX CONCURRENTLY, which > do not use that flag. I didn't try to reproduce it there, though. > Anywa

Re: Dependencies for partitioned indexes are still a mess

2020-08-12 Thread Alvaro Herrera
e effect is the same. > Issue #2: parallel restore does not work Looking. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 8334445705c53bb0abff407ebb92ac67975a5898 Mon Sep 17 00:00:00 2001 From: Alvar

Re: Dependencies for partitioned indexes are still a mess

2020-08-12 Thread Alvaro Herrera
On 2020-Jul-15, Tom Lane wrote: > Issue #2: parallel restore does not work > > 1. dropdb r2; createdb r2 > 2. pg_restore -j8 -d r2 regression.dump > > This is fairly timing-dependent, but some attempts fail with messages > like > > pg_restore: while PROCESSING TOC: > pg_restore: from TOC entry

Re: run pgindent on a regular basis / scripted manner

2020-08-12 Thread Alvaro Herrera
On 2020-Aug-12, Andres Freund wrote: > Is there any reason we don't just automatically run pgindent regularly? > Like once a week? And also update typedefs.list automatically, while > we're at it? Seconded. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24

Re: Switch to multi-inserts for pg_depend

2020-08-12 Thread Alvaro Herrera
On 2020-Aug-11, Robert Haas wrote: > On Tue, Aug 11, 2020 at 1:59 AM Michael Paquier wrote: > > On Mon, Aug 10, 2020 at 05:32:21PM -0700, Andres Freund wrote: > > > Do we really want to end up with several separate defines for different > > > type of catalog batch inserts? That doesn't seem like

Re: Switch to multi-inserts for pg_depend

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Michael Paquier wrote: > Okay. Would src/include/catalog/catalog.h be a suited location for > this flag or somebody has a better idea? Next to the API definition I guess, is that dependency.h? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Developmen

Re: Switch to multi-inserts for pg_depend

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Michael Paquier wrote: > On Thu, Aug 13, 2020 at 05:35:14AM -0400, Alvaro Herrera wrote: > > Next to the API definition I guess, is that dependency.h? > > We need something more central, see also MAX_PGATTRIBUTE_INSERT_BYTES > and MAX_PGSHDEPEND_INSERT_BYTES.

Re: [BUG] Error in BRIN summarization

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Anastasia Lubennikova wrote: > Cool. > This version looks much simpler than mine and passes the tests fine. Thanks, pushed it to all branches. Thanks for diagnosing this problem! -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: pg_dump from v13 is slow

2020-08-13 Thread Alvaro Herrera
On 2020-Aug-13, Justin Pryzby wrote: > I'm trying to narrow this down, but I'd be very happy for suggestions. Maybe you can time "pg_dump --binary-upgrade" to see if the slowness comes from there. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: pg_dump from v13 is slow

2020-08-13 Thread Alvaro Herrera
Hmm, I wonder if you're comparing an assert-enabled pg13 build to a non-assert-enabled pg12 build, or something like that. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Dependencies for partitioned indexes are still a mess

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-12, Alvaro Herrera wrote: > Hmm, we do make the FK constraint depend on the ATTACH for the direct > children; what I think we're lacking is dependencies on descendants > twice-removed (?) or higher. This mock patch seems to fix this problem > by adding dependencies r

Re: Switch to multi-inserts for pg_depend

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-14, Michael Paquier wrote: > Regarding the maximum number of slots allocated. Do people like the > current approach taken by the patch to do a single loop of the > dependency entries at the cost of more allocating perhaps too much for > the array holding the set of TupleTableSlots (th

Re: run pgindent on a regular basis / scripted manner

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-13, Stephen Frost wrote: > For my 2c, anyway, I like the idea of having folks update the typedefs > themselves when they've got a patch that needs a new typedef to be > indented correctly. Well, let's for starters encourage committers to update typedefs. Personally I've stayed away fr

Re: run pgindent on a regular basis / scripted manner

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-13, Magnus Hagander wrote: > That is: > 1. Whenever a patch is pushed on master on the main repo a process kicked > off (or maybe wait 5 minutes to coalesce multiple patches if there are) > 2. This process checks out master, and runs pgindent on it > 3. When done, this gets committed t

Re: Dependencies for partitioned indexes are still a mess

2020-08-14 Thread Alvaro Herrera
On 2020-Aug-14, Alvaro Herrera wrote: > On 2020-Aug-12, Alvaro Herrera wrote: > > > Hmm, we do make the FK constraint depend on the ATTACH for the direct > > children; what I think we're lacking is dependencies on descendants > > twice-removed (?) or higher. This

Re: Add information to rm_redo_error_callback()

2020-08-17 Thread Alvaro Herrera
On 2020-Aug-17, Drouvot, Bertrand wrote: > Having this "pg_waldump" kind of format in this place > (rm_redo_error_callback()) ensures that we'll always see the same piece of > information during rm_redo. > > I think it's good to guarantee that we'll always see the same piece of > information (sho

Re: [BUG] Error in BRIN summarization

2020-08-17 Thread Alvaro Herrera
On 2020-Aug-15, Tom Lane wrote: > hyrax's latest report suggests that this patch has issues under > CLOBBER_CACHE_ALWAYS: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hyrax&dt=2020-08-13%2005%3A09%3A58 > > Hard to tell whether there's an actual bug there or just test instability,

Re: Improving connection scalability: GetSnapshotData()

2020-08-17 Thread Alvaro Herrera
On 2020-Aug-16, Peter Geoghegan wrote: > On Sun, Aug 16, 2020 at 2:11 PM Andres Freund wrote: > > For the first, one issue is that there's no obviously good candidate for > > an uninitialized xid. We could use something like FrozenTransactionId, > > which may never be in the procarray. But it's n

Re: [BUG] Error in BRIN summarization

2020-08-17 Thread Alvaro Herrera
On 2020-Aug-17, Alvaro Herrera wrote: > Hmm, the only explanation I can see for this is that autovacuum managed > to summarize the range before the test script did it. So the solution > would simply be to disable autovacuum for the table across the whole > script. > > I

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2020-08-17 Thread Alvaro Herrera
On 2020-Aug-14, Ibrar Ahmed wrote: > The table used for the test contains three columns (integer, text, > varchar). > The total number of rows is 1000 in total. > > Unpatched (Master: 92c12e46d5f1e25fc85608a6d6a19b8f5ea02600) > COPY: 9069.432 ms vacuum; 2567.961ms > COPY: 9004.533 ms vacuum:

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-18 Thread Alvaro Herrera
On 2020-Aug-17, Ashutosh Sharma wrote: > > + if (heap_force_opt == HEAP_FORCE_KILL) > > + ItemIdSetDead(itemid); > > > > I think that if the page is an all-visible page, we should clear an > > all-visible bit on the visibility map corresponding to the page and > > PD_ALL_VI

Re: Print logical WAL message content

2020-08-18 Thread Alvaro Herrera
On 2020-Aug-18, Ashutosh Bapat wrote: > Right now pg_waldump just prints whether the message is transactional > or not and its size. That doesn't help much to understand the message > itself. If it prints the contents of a logical WAL message, it helps > debugging logical replication related probl

Re: Creating foreign key on partitioned table is too slow

2020-08-19 Thread Alvaro Herrera
On 2020-Aug-19, Amit Langote wrote: Hello > On Thu, Aug 6, 2020 at 4:25 PM kato-...@fujitsu.com > wrote: > > On Wednesday, August 5, 2020 9:43 AM I wrote: > > > I'll report the result before the end of August . > > > > I test v2-0001-build-partdesc-memcxt.patch at 9a9db08ae4 and it is ok. > > I

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-08-19 Thread Alvaro Herrera
tps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 2a5088dfa35cbc800a87dc2154b6ebfa22837a66 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Tue, 4 Aug 2020 22:04:57 -0400 Subject: [PATCH v2] Avoid spurious CREATE INDEX CONCURRENTLY waits --- src/back

"ccold" left by reindex concurrently are droppable?

2020-08-19 Thread Alvaro Herrera
Hello The REINDEX CONCURRENTLY documentation states that if a transient index used lingers, the fix is to drop the invalid index and perform RC again; and that this is to be done for "ccnew" indexes and also for "ccold" indexes: The recommended recovery method in such cases is to drop the inv

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-19 Thread Alvaro Herrera
On 2020-Aug-19, David Rowley wrote: > Andres' suggestion: > > regression=# explain (analyze, costs off, timing off, summary off) > select count(*) from tenk1 t1 inner join tenk1 t2 on > t1.twenty=t2.unique1; > QUERY PLAN >

Re: "ccold" left by reindex concurrently are droppable?

2020-08-20 Thread Alvaro Herrera
Thanks, Michael and Julien! Pushed to 12-master, with a slight rewording to use the passive voice, hopefully matching the surrounding text. I also changed "temporary" to "transient" in another line, for consistency. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Develo

Re: [PATCH] Fix Uninitialized scalar variable (UNINIT) (src/backend/access/heap/heapam_handler.c)

2020-08-25 Thread Alvaro Herrera
On 2020-Aug-25, Ranier Vilela wrote: > The variable root_offsets is read at line 1641, but, at this point, > the content is unknown, so it is impossible to test works well. Surely it is set by heap_get_root_tuples() in line 1347? The root_blkno variable is used exclusively to know whether root_o

Re: [PATCH] Fix Uninitialized scalar variable (UNINIT) (src/backend/access/heap/heapam_handler.c)

2020-08-25 Thread Alvaro Herrera
On 2020-Aug-25, Ranier Vilela wrote: > If the variable hscan->rs_cblock is InvalidBlockNumber the test can > protect root_offsets fail. When does that happen? > The var root_blkno only is checked at line 1853. That's a different function. -- Álvaro Herrerahttps://www.2ndQuadra

Re: [PATCH] Fix Uninitialized scalar variable (UNINIT) (src/backend/access/heap/heapam_handler.c)

2020-08-25 Thread Alvaro Herrera
On 2020-Aug-25, Ranier Vilela wrote: > And it is guaranteed that, rs_cblock is not InvalidBlockNumber when the > page is read? It could be InvalidBlockNumber if sufficient neutrinos hit the memory bank and happen to set all the bits in the block number. -- Álvaro Herrerahttps://

Re: [PATCH] Fix Uninitialized scalar variable (UNINIT) (src/backend/access/heap/heapam_handler.c)

2020-08-25 Thread Alvaro Herrera
On 2020-Aug-25, Ranier Vilela wrote: > kkk, I think it's enough for me. > I believe that PostgreSQL will not run on the ISS yet. Actually, I believe there are some satellites that run Postgres -- not 100% sure about this. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL

Re: [PATCH] Fix Uninitialized scalar variable (UNINIT) (src/backend/access/heap/heapam_handler.c)

2020-08-25 Thread Alvaro Herrera
On 2020-Aug-25, Ranier Vilela wrote: > 1. Even heap_get_root_tuples at line 1347, be called. > Does it fill all roots_offsets? Yes -- read the comments there. > 2. hscan->rs_cbuf is constant? > if (hscan->rs_cblock != root_blkno) It is the buffer that contains the given block. Those two things

Re: [PATCH] Fix Uninitialized scalar variable (UNINIT) (src/backend/access/heap/heapam_handler.c)

2020-08-25 Thread Alvaro Herrera
On 2020-Aug-25, Ranier Vilela wrote: > Em ter., 25 de ago. de 2020 às 19:45, Alvaro Herrera < > alvhe...@2ndquadrant.com> escreveu: > > > On 2020-Aug-25, Ranier Vilela wrote: > > > > > If the variable hscan->rs_cblock is InvalidBlockNumber the t

Re: LWLockAcquire and LockBuffer mode argument

2020-08-26 Thread Alvaro Herrera
On 2020-Aug-26, Robert Haas wrote: > On Wed, Aug 26, 2020 at 7:47 AM Dagfinn Ilmari Mannsåker > wrote: > > Would it be possible to make the compat versions only available when > > building extensions, but not to core code? > > I think that would be good if we can do it. We could even have it > i

Re: [POC]Enable tuple change partition caused by BEFORE TRIGGER

2020-08-26 Thread Alvaro Herrera
On 2020-Aug-21, Ashutosh Bapat wrote: > On Fri, Aug 21, 2020 at 1:28 PM movead...@highgo.ca > wrote: > > In current BEFORE TRIGGER implementation, it reports an error once a > > trigger result out of current partition, but I think it should check > > it after finish all triggers call, and you c

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-26 Thread Alvaro Herrera
On 2020-Aug-25, Andres Freund wrote: > Hi, > > On 2020-08-26 15:58:14 +1200, Thomas Munro wrote: > > > --12.51%--compactify_tuples > > > PageRepairFragmentation > > > heap2_redo > > > StartupXLOG > > >

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-26 Thread Alvaro Herrera
On 2020-Aug-25, Jakub Wartak wrote: > Turning on/off the defer SLRU patch and/or fsync doesn't seem to make > any difference, so if anyone is curious the next sets of append-only > bottlenecks is like below: > > 14.69% postgres postgres[.] hash_search_with_hash_value >

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2020-08-26 Thread Alvaro Herrera
On 2020-Aug-20, Jeremy Schneider wrote: > While working with Nathan Bossart on an extension, we came across an > interesting quirk and possible inconsistency in the PostgreSQL code > around infomask flags. I'd like to know if there's something I'm > misunderstanding here or if this really is a co

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-08-26 Thread Alvaro Herrera
On 2020-Aug-04, Robert Haas wrote: > On Mon, Aug 3, 2020 at 7:49 PM Alvaro Herrera > wrote: > > Why two transactions? The reason is that in order for this to work, we > > make a catalog change (mark it detached), and commit so that all > > concurrent transactions can s

Re: [POC]Enable tuple change partition caused by BEFORE TRIGGER

2020-08-27 Thread Alvaro Herrera
On 2020-Aug-27, Ashutosh Bapat wrote: > On Wed, 26 Aug 2020 at 22:47, Alvaro Herrera > wrote: > > But I'm not 100% about running the BEFORE triggers. Maybe > > one way to address this is to check whether the BEFORE triggers in the > > new target partition are

Re: Clang UndefinedBehaviorSanitize (Postgres14) Detected undefined-behavior

2020-08-27 Thread Alvaro Herrera
On 2020-Aug-27, Ranier Vilela wrote: > indexcmds.c (1162): > memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); Looks legit, and at least per commit 13bba02271dc we do fix such things, even if it's useless in practice. Given that no buildfarm member has ever complained, this exercise seems

Re: Clang UndefinedBehaviorSanitize (Postgres14) Detected undefined-behavior

2020-08-27 Thread Alvaro Herrera
On 2020-Aug-27, Ranier Vilela wrote: > If we are passing a null pointer in these places and it should not be done, > it is a sign that perhaps these calls should not or should not be made, and > they can be avoided. Feel free to send a patch. -- Álvaro Herrerahttps://www.2ndQuad

Re: list of extended statistics on psql

2020-08-27 Thread Alvaro Herrera
+1 for the general idea, and +1 for \dX being the syntax to use IMO the per-type columns should show both the type being enabled as well as it being built. (How many more stat types do we expect -- Tomas? I wonder if having one column per type is going to scale in the long run.) Also, the stat

Re: list of extended statistics on psql

2020-08-27 Thread Alvaro Herrera
On 2020-Aug-28, Tatsuro Yamada wrote: > > IMO the per-type columns should show both the type being enabled as > > well as it being built. > > Hmm. I'm not sure how to get the status (enabled or disabled) of > extended stats. :( > Could you explain it more? pg_statistic_ext_data.stxdndistinct is

Re: list of extended statistics on psql

2020-08-29 Thread Alvaro Herrera
On 2020-Aug-29, Tomas Vondra wrote: > But if we want the > output to show both what was requested and which types were actually > built, that'll effectively double the number of columns needed :-( I was thinking it would be one column per type showing either disabled or enabled or built. But an

Re: list of extended statistics on psql

2020-08-30 Thread Alvaro Herrera
On 2020-Aug-30, Tomas Vondra wrote: > On Sat, Aug 29, 2020 at 06:43:47PM -0400, Alvaro Herrera wrote: > > On 2020-Aug-29, Tomas Vondra wrote: > > > Also, it might be useful to show the size of the statistics built, just > > > like we show for \d+ etc. > > >

Re: list of extended statistics on psql

2020-08-31 Thread Alvaro Herrera
On 2020-Aug-30, Tomas Vondra wrote: > On Sun, Aug 30, 2020 at 12:33:29PM -0400, Alvaro Herrera wrote: > > I wonder how to report that. Knowing that psql \-commands are not meant > > for anything other than human consumption, maybe we can use a format() > > string that

Re: list of extended statistics on psql

2020-08-31 Thread Alvaro Herrera
On 2020-Aug-31, Tomas Vondra wrote: > I wonder if trying to list info about all stats from the statistics > object in a single line is necessary. Maybe we should split the info > into one line per statistics, so for example > > CREATE STATISTICS s (mcv, ndistinct, dependencies) ON ... > > wo

Re: Clang UndefinedBehaviorSanitize (Postgres14) Detected undefined-behavior

2020-08-31 Thread Alvaro Herrera
On 2020-Aug-31, Ranier Vilela wrote: > More troubles with undefined-behavior. > > This type of code can leaves overflow: > var = (cast) (expression); > diff = (int32) (id1 - id2); > > See: > diff64 = ((long int) d1 - (long int) d2); > diff64=-4294901760 Did you compile this with gcc -f

LogwrtResult contended spinlock

2020-08-31 Thread Alvaro Herrera
So I decided to make it cover both and call it a day. We did verify that the patch solves the reported problem, at any rate. -- Álvaro HerreraPostgreSQL Expert, https://www.2ndQuadrant.com/ >From 3eb871f235c1b6005ff5dc88561173e4e92c1314 Mon Sep 17 00:00:00 2001 From: Alvaro

Re: LogwrtResult contended spinlock

2020-08-31 Thread Alvaro Herrera
On 2020-Aug-31, Andres Freund wrote: > Hi, > > On August 31, 2020 11:21:56 AM PDT, Alvaro Herrera > wrote: > >At first I wanted to make the new LWLock cover only LogwrtResult > >proper, > >and leave LogwrtRqst alone. However on doing it, it seemed that th

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-11, Justin Pryzby wrote: > On Tue, Aug 11, 2020 at 02:39:45PM +0900, Michael Paquier wrote: > > The grammar that has been committed was the one that for the most > > support, so we need to live with that. I wonder if we should simplify > > ReindexStmt and move the "concurrent" flag to

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alvaro Herrera
On 2020-Sep-01, Justin Pryzby wrote: > On Tue, Sep 01, 2020 at 11:40:18AM -0400, Alvaro Herrera wrote: > > The advantage of using a parenthesized option list is that you can add > > *further* options without making the new keywords reserved. Of course, > > we already res

Re: Reloptions for table access methods

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-31, Jeff Davis wrote: > fillRelOptions() validates when filling in a struct to make sure there > aren't "leftover" options. It does this using a hard-coded parsing > table that is not extensible. Hmm, I think that if we're going to do this, we should do it for all AMs, not just table

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-27, Robert Haas wrote: > On Wed, Aug 26, 2020 at 7:40 PM Alvaro Herrera > wrote: > > To mark it detached means to set pg_inherits.inhdetached=true. That > > column name is a bit of a misnomer, since that column really means "is > > in the process of bei

Re: v13: show extended stats target in \d

2020-09-01 Thread Alvaro Herrera
+1 on fixing this, since the ability to change stats target is new in pg13. On 2020-Aug-31, Justin Pryzby wrote: > Maybe it should have a comma, like ", STATISTICS %s"? It does need some separator. Maybe a comma is sufficient, but I'm not sure: that will fail when we add cross-relation stats, b

Re: logtape.c stats don't account for unused "prefetched" block numbers

2020-09-01 Thread Alvaro Herrera
On 2020-Jul-30, Peter Geoghegan wrote: > Commit 896ddf9b added prefetching to logtape.c to avoid excessive > fragmentation in the context of hash aggs that spill and have many > batches/tapes. Apparently the preallocation doesn't actually perform > any filesystem operations, so the new mechanism s

Re: Dependencies for partitioned indexes are still a mess

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-12, Alvaro Herrera wrote: > On 2020-Jul-15, Tom Lane wrote: > > (There seem to be some other problems as well, but most of the 54 complaints > > are related to partitioned indexes/constraints.) > > In my run of it there's a good dozen remaining problems, al

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alvaro Herrera
On 2020-Sep-02, Michael Paquier wrote: > Yeah, I am all for removing "concurrent" from ReindexStmt, but I don't > think that the proposed 0002 is that, because it is based on the > assumption that we'd want more than just boolean-based options in > those statements, and this case is not justified

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-09-02 Thread Alvaro Herrera
On 2020-Sep-02, David Rowley wrote: > v7 (Separate Result Cache node) > Query 1: > Execution Time: 894.003 ms > > Query 2: > Execution Time: 854.950 ms > v7 + hacks_V3 (caching done in Nested Loop) > Query 1: > Execution Time: 770.470 ms > > Query 2 > Execution Time: 779.181 ms Wow, this is

Re: builtin functions, parameter names and psql's \df

2020-09-02 Thread Alvaro Herrera
On 2020-Sep-02, Tom Lane wrote: > I don't think we should go overboard on this, but +1 for labeling all the > cases where the usage isn't obvious. +1 -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Fix for configure error in 9.5/9.6 on macOS 11.0 Big Sur

2020-09-02 Thread Alvaro Herrera
On 2020-Sep-02, Jesse Zhang wrote: > Hi Peter, > > Yeah it's funny I got this immediately after upgrading to Big Sur (beta > 5). I found commit 1c0cf52b39ca3 but couldn't quite find the mailing > list thread on it from 4 years ago (it lists Heikki and Thomas Munro as > reviewers). Was it prompted

Re: Switch to multi-inserts for pg_depend

2020-09-03 Thread Alvaro Herrera
I agree, this version looks much better, thanks. Two very minor things: On 2020-Sep-03, Michael Paquier wrote: > @@ -76,11 +77,23 @@ recordMultipleDependencies(const ObjectAddress *depender, > > dependDesc = table_open(DependRelationId, RowExclusiveLock); > > + /* > + * Alloca

Re: LogwrtResult contended spinlock

2020-09-03 Thread Alvaro Herrera
Looking at patterns like this if (XLogCtl->LogwrtRqst.Write < EndPos) XLogCtl->LogwrtRqst.Write = EndPos; It seems possible to implement with do { XLogRecPtr currwrite; currwrite = pg_atomic_read_u64(LogwrtRqst.Write); if (currwrite > End

Re: proposal: possibility to read dumped table's name from file

2020-09-03 Thread Alvaro Herrera
On 2020-Jul-27, Pavel Stehule wrote: > +/* > + * getline is originally GNU function, and should not be everywhere still. > + * Use own reduced implementation. > + */ > +static size_t > +pg_getline(char **lineptr, size_t *n, FILE *fp) > +{ So, Tom added a coding pattern for doing this in commit 8f

Re: default partition and concurrent attach partition

2020-09-03 Thread Alvaro Herrera
Thanks for this fix! Looking into it now. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c index 4d34734a45..fe42670e0a 100

Re: default partition and concurrent attach partition

2020-09-03 Thread Alvaro Herrera
On 2020-Sep-03, Alvaro Herrera wrote: > + /* > + * If setting up a PartitionDispatch for a sub-partitioned table, we may > + * also need a fake ResultRelInfo for checking the partition constraint > + * later; set that up now. > + */ > +

Re: Parallel worker hangs while handling errors.

2020-09-03 Thread Alvaro Herrera
On 2020-Sep-03, Tom Lane wrote: > As for the question of SIGQUIT handling, I see that postgres.c > does "PG_SETMASK(&BlockSig)" immediately after applying the sigdelset > change, so there probably isn't any harm in having the background > processes do likewise. I wonder though why bgworkers are n

report expected contrecord size

2020-09-03 Thread Alvaro Herrera
A pretty minor issue: when reporting that WAL appears invalid because contrecord length doesn't match, we may as well print to the server log the value that we're expecting. Patch attached. -- Álvaro Herrera http://www.flickr.com/photos/alvherre/ diff --git a/src/backend/

Re: default partition and concurrent attach partition

2020-09-03 Thread Alvaro Herrera
Also, I should have pointed out that ExecInsert doesn't actually check the partitin constraint except in very specific cases; what it does is expect that the partition routing code got it right. So the comment you're adding about that is wrong, and it did misled me into changing your code in a way

Re: report expected contrecord size

2020-09-03 Thread Alvaro Herrera
On 2020-Sep-03, Tom Lane wrote: > Alvaro Herrera writes: > > A pretty minor issue: when reporting that WAL appears invalid because > > contrecord length doesn't match, we may as well print to the server log > > the value that we're expect

Re: report expected contrecord size

2020-09-03 Thread Alvaro Herrera
On 2020-Sep-03, Tom Lane wrote: > Alvaro Herrera writes: > > Well, the intention there is to cast the first operand (which is uint32) > > so that it turns into signed 64-bits; the subtraction then occurs in 64 > > bit arithmetic normally. If I let the subtraction

Re: [PATCH] - Provide robust alternatives for replace_string

2020-09-03 Thread Alvaro Herrera
Note that starting with commit 67a472d71c98 you can use pg_get_line and not worry about the hard part of this anymore :-) -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Rare deadlock failure in create_am test

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-03, Tom Lane wrote: > So it's not hard to understand the problem: DROP of an index AM, cascading > to an index, will need to acquire lock on the index and then lock on the > index's table. Any other operation on the table, like say autovacuum, > is going to acquire locks in the other

Re: Division in dynahash.c due to HASH_FFACTOR

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Jakub Wartak wrote: > After removing HASH_FFACTOR PostgreSQL still compiles...  Would > removing it break some external API/extensions ? FWIW, HASH_FFACTOR has *never* been used in Postgres core code. https://postgr.es/m/20160418180711.55ac82c0@fujitsu already reported that this

Re: Change a constraint's index - ALTER TABLE ... ALTER CONSTRAINT ... USING INDEX ...

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Laurenz Albe wrote: > The value I see in this is: > - replacing a primary key index > - replacing the index behind a constraint targeted by a foreign key But why is this better than using REINDEX CONCURRENTLY? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ Postgre

Re: [PATCH]Fix ja.po error

2020-09-04 Thread Alvaro Herrera
On 2020-Aug-19, Lu, Chenyang wrote: > Ping: sorry, did Alvaro and Peter forget this email?( Maybe didn't see this > email~ ), I found that the patch of ja.po has not been applied to the > Translation Repository. Apologies. I have pushed this to all branches of the translation repo now. The bo

Re: Change a constraint's index - ALTER TABLE ... ALTER CONSTRAINT ... USING INDEX ...

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Laurenz Albe wrote: > On Fri, 2020-09-04 at 10:41 -0400, Alvaro Herrera wrote: > > > The value I see in this is: > > > - replacing a primary key index > > > - replacing the index behind a constraint targeted by a foreign key > > > >

Re: report expected contrecord size

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-03, Tom Lane wrote: > Alvaro Herrera writes: > > On 2020-Sep-03, Tom Lane wrote: > >> Uh ... is it really possible for gotlen to be more than total_len? > >> (I've not looked at the surrounding code here, but that seems weird.) > > > Well,

Re: A micro-optimisation for walkdir()

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Thomas Munro wrote: > @@ -10,6 +10,7 @@ struct dirent > { > longd_ino; > unsigned short d_reclen; > + unsigned char d_type; > unsigned short d_namlen; > chard_name[MAX_PATH]; > }; > @@ -20,4 +21,26 @@ DIR *open

Re: Concurrent failure in create_am with buildfarm member conchuela

2020-09-04 Thread Alvaro Herrera
Hello, On 2020-Sep-04, Michael Paquier wrote: > conchuela has just reported the following error: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2020-09-03%2023%3A00%3A36 Fixed in d54f99e4154. Thanks -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreS

Re: A micro-optimisation for walkdir()

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Juan José Santamaría Flecha wrote: > On Fri, Sep 4, 2020 at 9:37 PM Alvaro Herrera > wrote: > > > On 2020-Sep-04, Thomas Munro wrote: > > > > > > +/* File types for 'd_type'. */ > > > +enum > > > + { > > >

Re: Questionable ping logic in LogicalRepApplyLoop

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Tom Lane wrote: > While playing around with clang's scan-build I noticed this warning: > > worker.c:2281:7: warning: Value stored to 'ping_sent' is never read > ping_sent = true; > ^

Re: A micro-optimisation for walkdir()

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Juan José Santamaría Flecha wrote: > If will fail to detect that the patch makes the optimisation available for > WIN32: > > +#if defined(DT_UNKNOWN) && defined(DT_REG) && defined(DT_DIR) && > defined(DT_LNK) Oh, I see. I suggest that it'd be better to change this line instead.

Re: PATCH: Batch/pipelining support for libpq

2020-09-04 Thread Alvaro Herrera
On 2020-Aug-31, Matthieu Garrigues wrote: > It seems like this patch is nearly finished. I fixed all the remaining > issues. I'm also asking a confirmation of the test scenarios you want > to see in the next version of the patch. Hmm, apparently nobody noticed that this patch is not registered in

Re: A micro-optimisation for walkdir()

2020-09-04 Thread Alvaro Herrera
On 2020-Sep-04, Tom Lane wrote: > I think that it's standard to test for such symbols by seeing > if they're defined as macros ... not least because that's the *only* > way to test their existence in C. I guess since what we're doing is emulating standard readdir(), that makes sense. > Personall

Re: Change a constraint's index - ALTER TABLE ... ALTER CONSTRAINT ... USING INDEX ...

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-07, Laurenz Albe wrote: > This patch would provide a more convenient way to do that. > > Again, I am not sure if that justifies the effort. I have to admit I've seen cases where it'd be useful to have included columns in primary keys. TBH I think if we really wanted the feature of p

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-04, Amit Langote wrote: Hello > FWIW, I still prefer "minimally valid ResultRelInfo" to "fake > ResultRelInfo", because those aren't really fake, are they? They are > as valid as any other ResultRelInfo as far I can see. I said > "minimally valid" because a fully-valid partition Res

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-07, Alvaro Herrera wrote: > Well, they are fake in that the ri_RangeTableIndex they carry is bogus, > which means that ExecBuildSlotValueDescription will misbehave if the > partitioned default partition has a different column order than its > parent. That can be evidenced

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
/www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 7289a0706d95aa621b9d6f626a836ac381fd4f61 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Mon, 7 Sep 2020 19:26:37 -0300 Subject: [PATCH] Avoid invalid RRI --- src/backend/executor/execMain.c

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-07, Alvaro Herrera wrote: > Ah, it looks like we can get away with initializing the RRI to 0, and > then explicitly handle that case in ExecPartitionCheckEmitError, as in > the attached (which means reindenting, but I left it alone to make it > easy to read). Well, th

Re: default partition and concurrent attach partition

2020-09-08 Thread Alvaro Herrera
Hello Amit, On 2020-Sep-08, Amit Langote wrote: > On Tue, Sep 8, 2020 at 8:44 AM Alvaro Herrera > wrote: > > On 2020-Sep-07, Alvaro Herrera wrote: > > > > > Ah, it looks like we can get away with initializing the RRI to 0, and > > >

Re: Logical Replication - detail message with names of missing columns

2020-09-08 Thread Alvaro Herrera
On 2020-Sep-08, Bharath Rupireddy wrote: > + /* Find the remote attributes that are missing in the local relation. */ > + for (i = 0; i < remoterel->natts; i++) > + { > + if (!bms_is_member(i, localattnums)) > + { > + if (missingatts->len ==

Re: Change a constraint's index - ALTER TABLE ... ALTER CONSTRAINT ... USING INDEX ...

2020-09-08 Thread Alvaro Herrera
On 2020-Sep-08, Laurenz Albe wrote: > We should at least have > > ALTER TABLE ... ADD PRIMARY KEY (id) INCLUDE (val); > > or something before we consider this patch. Agreed. Now the trick in this new command is to let the user change the included columns afterwards, which remains useful (sin

Re: default partition and concurrent attach partition

2020-09-08 Thread Alvaro Herrera
loop, which is sensible since the condition is true on loop entrance. Minor comment rewording also. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From ef139f89531ba15f480cbb64c2bddeee03dc20ab Mon Sep 1

Re: VACUUM (INTERRUPTIBLE)?

2020-09-08 Thread Alvaro Herrera
On 2020-Sep-08, Andres Freund wrote: > That made me wonder if it'd be worthwhile to add an option that'd make > user invoked VACUUM be interruptible by conflicting lock requests, just > like autovacuum is. Yeah, I recall a request for this in the past, too. > So how about adding an INTERRUPTIBLE

Re: default partition and concurrent attach partition

2020-09-08 Thread Alvaro Herrera
On 2020-Sep-08, Alvaro Herrera wrote: > Andres added to CC because of TTS interface: apparently calling > slot_getallattrs() on a virtual slot raises error that "getsomeattrs is > not required to be called on a virtual tuple table slot". I'm thinking > that this exp

<    1   2   3   4   5   6   7   8   9   10   >