psql tab completion for \h with IMPORT FOREIGN SCHEMA

2021-03-17 Thread Michael Paquier
Hi all, Well, as $subject tells, I just found confusing that \h does not complete the so-said command, the only one using IMPORT as keyword, so I'd like to do the following: --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -1493,7 +1493,7 @@ psql_completion(const char *text,

Re: fdatasync performance problem with large number of DB files

2021-03-17 Thread Fujii Masao
On 2021/03/17 12:45, Thomas Munro wrote: On Tue, Mar 16, 2021 at 9:29 PM Fujii Masao wrote: On 2021/03/16 8:15, Thomas Munro wrote: I don't want to add a hypothetical sync_after_crash=none, because it seems like generally a bad idea. We already have a running-with-scissors mode you could u

Re: a verbose option for autovacuum

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 08:50:26AM -0300, Euler Taveira wrote: > Since commit 5f8727f5a6, this patch doesn't apply anymore. Fortunately, it is > just a small hunk. I reviewed this patch and it looks good to me. There is > just > a small issue (double space after 'if') that I fixed in the attached

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Peter Geoghegan
On Wed, Mar 17, 2021 at 11:23 PM Masahiko Sawada wrote: > Attached the updated patch that can be applied on top of your v3 patches. Some feedback on this: * I think that we can afford to be very aggressive here, because we're checking dynamically. And we're concerned about extremes only. So an a

Re: [PATCH] Identify LWLocks in tracepoints

2021-03-17 Thread Craig Ringer
On Thu, 11 Mar 2021 at 15:57, Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 10.03.21 06:38, Craig Ringer wrote: > > On Wed, 3 Mar 2021 at 20:50, David Steele > > wrote: > > > > On 1/22/21 6:02 AM, Peter Eisentraut wrote: > > > > This patch s

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Masahiko Sawada
On Thu, Mar 18, 2021 at 12:23 PM Peter Geoghegan wrote: > > On Wed, Mar 17, 2021 at 7:16 PM Masahiko Sawada wrote: > > Since I was thinking that always skipping index vacuuming on > > anti-wraparound autovacuum is legitimate, skipping index vacuuming > > only when we're really close to the point

Re: Detecting File Damage & Inconsistencies

2021-03-17 Thread Craig Ringer
On Mon, 15 Mar 2021 at 21:01, David Steele wrote: > On 11/18/20 5:23 AM, Simon Riggs wrote: > > On Wed, 18 Nov 2020 at 06:42, Craig Ringer > > wrote: > >> > >> On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs > wrote: > >>> > >>> > >>> What I'm proposing is an option to add 16 bytes onto each COMMIT

Re: fdatasync performance problem with large number of DB files

2021-03-17 Thread Fujii Masao
On 2021/03/17 12:02, Thomas Munro wrote: On Tue, Mar 16, 2021 at 9:10 PM Fujii Masao wrote: Thanks for the patch! +When set to fsync, which is the default, +PostgreSQL will recursively open and fsync +all files in the data directory before crash recovery begins. Isn

Re: Get memory contexts of an arbitrary backend process

2021-03-17 Thread Fujii Masao
On 2021/03/17 22:24, torikoshia wrote: I remade the patch and introduced a function pg_print_backend_memory_contexts(PID) which prints the memory contexts of the specified PID to elog. Thanks for the patch!   =# SELECT pg_print_backend_memory_contexts(450855);   ** log output **   202

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-17 Thread Amit Kapila
On Wed, Mar 17, 2021 at 11:27 AM Amit Kapila wrote: > > 5. I have modified the comments atop worker.c to explain the design > and some of the problems clearly. See attached. If you are fine with > this, please include it in the next version of the patch. > I have further expanded these comments t

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-17 Thread Fujii Masao
On 2021/03/18 11:59, kuroda.hay...@fujitsu.com wrote: Dear Ikeda-san, I confirmed new patch and no problem was found. Thanks. (I'm not a native English speaker, so I cannot check your comments correctly, sorry) One user-visible side-effect by this change is; with the patch, the stats is cl

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 23:16 odesílatel Hannu Krosing napsal: > why are you using yet another special syntax for this ? > > would it not be better to do something like this: > CREATE FUNCTION a_reall_long_and_winding_function_name(i int, out o int) > LANGUAGE plpgsql AS $plpgsql$ > DECLARE >args fu

Re: pg_amcheck contrib application

2021-03-17 Thread Tom Lane
Mark Dilger writes: >> On Mar 16, 2021, at 12:52 PM, Robert Haas wrote: >> Since we now know that shutting autovacuum off makes the problem go >> away, I don't see a reason to commit 0001. We should fix pg_amcheck >> instead, if, as presently seems to be the case, that's where the >> problem is.

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 23:32 odesílatel Michael Paquier napsal: > On Wed, Mar 17, 2021 at 05:04:48PM +0100, Pavel Stehule wrote: > > This tree has a different direction than is usual, and then replacing the > > root node is not simple. > > Yeah, it is not like we should redesign this whole part just fo

Re: pg_amcheck contrib application

2021-03-17 Thread Mark Dilger
> On Mar 16, 2021, at 12:52 PM, Robert Haas wrote: > > On Mon, Mar 15, 2021 at 10:10 PM Mark Dilger > wrote: >> It is unfortunate that the failing test only runs pg_amcheck after creating >> numerous corruptions, as we can't know if pg_amcheck would have complained >> about pg_statistic befo

Re: PITR promote bug: Checkpointer writes to older timeline

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 05:09:50PM +0900, Michael Paquier wrote: > Currently with HEAD and back branches, nothing would be broken as > logical contexts cannot exist in recovery. Still it would be easy > to miss the new behavior for anybody attempting to work more on this > feature in the future if

RE: libpq debug log

2021-03-17 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > In pqTraceOutputString(), you can use the return value from fprintf to > move the cursor -- no need to count chars. Yes, precisely, 2 bytes for the double quotes needs to be subtracted as follows: len = fprintf(...); *cursor += (len - 2); > I still think

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread houzj.f...@fujitsu.com
> > If a table parameter value is set and the > > equivalent toast. parameter is not, the TOAST table > > will use the table's parameter value. > > -Specifying these parameters for partitioned tables is not supported, > > -but you may specify them for individual leaf partitio

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread Amit Kapila
On Thu, Mar 18, 2021 at 8:30 AM Justin Pryzby wrote: > > diff --git a/doc/src/sgml/ref/create_table.sgml > b/doc/src/sgml/ref/create_table.sgml > index ff1b642722..d5d356f2de 100644 > --- a/doc/src/sgml/ref/create_table.sgml > +++ b/doc/src/sgml/ref/create_table.sgml > @@ -1338,8 +1338,10 @@ WITH

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Peter Geoghegan
On Wed, Mar 17, 2021 at 7:16 PM Masahiko Sawada wrote: > Since I was thinking that always skipping index vacuuming on > anti-wraparound autovacuum is legitimate, skipping index vacuuming > only when we're really close to the point of going into read-only mode > seems a bit conservative, but maybe

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > The most glaring case is the RelationInitTableAccessMethod() call in > RelationBuildLocalRelation(). Seems like the best fix is to just move > the MemoryContextSwitchTo() to just before the > RelationInitTableAccessMethod(). Although I wonder if we shouldn't go > further,

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
On 2021-03-17 22:33:59 -0400, Tom Lane wrote: > >> I've not yet tried to grok the comment that purports to justify it, > >> but I fail to see why it'd ever be useful to drop values inherited > >> from the postmaster. > > > I can't really make sense of it either. I think it may be trying to > > res

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 00:01:55 -0400, Tom Lane wrote: > As for the particular point about ParallelBlockTableScanWorkerData, > I agree with your question to David about why that's in TableScanDesc > not HeapScanDesc, but I can't get excited about it not being freed in > heap_endscan. That's mainly beca

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 07:30:04PM -0700, Andres Freund wrote: > I suspect it might be easier to reproduce the issue with smaller WAL > segments, a short checkpoint_timeout, and multiple jobs generating WAL > and then sleeping for random amounts of time. Not sure if that's the > sole ingredient, bu

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread Justin Pryzby
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index ff1b642722..d5d356f2de 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1338,8 +1338,10 @@ WITH ( MODULUS numeric_literal, REM If a table parameter value is s

RE: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-17 Thread kuroda.hay...@fujitsu.com
Dear Ikeda-san, I confirmed new patch and no problem was found. Thanks. (I'm not a native English speaker, so I cannot check your comments correctly, sorry) Best Regards, Hayato Kuroda FUJITSU LIMITED

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Peter Geoghegan
On Mon, Mar 15, 2021 at 4:11 PM Andres Freund wrote: > I kinda wonder whether this case should just be handled by just gotoing > back to the start of the blkno loop, and redoing the pruning. The only > thing that makes that a bit more complicatd is that we've already > incremented vacrelstats->{sc

RE: libpq debug log

2021-03-17 Thread tsunakawa.ta...@fujitsu.com
I'll look at the comments from Alvaro-san and Horiguchi-san. Here are my review comments: (23) + /* Trace message only when there is first 1 byte */ + if (conn->Pfdebug && conn->outCount < conn->outMsgStart) + { + if (conn->outCount < conn->outMsgStart) +

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread Justin Pryzby
On Fri, Mar 12, 2021 at 04:05:09PM +0900, Amit Langote wrote: > On Fri, Mar 12, 2021 at 6:10 AM Justin Pryzby wrote: > > Note also this CF entry > > https://commitfest.postgresql.org/32/2987/ > > | Allow setting parallel_workers on partitioned tables I'm rebasing that other patch on top of master

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > On 2021-03-17 21:30:48 -0400, Tom Lane wrote: >> I believe I've just tracked down the cause of that. Those errors >> are (as far as I've seen) only happening in parallel workers, and >> the reason is this gem in RestoreGUCState: ... > Ouch. At least it's a short lived iss

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Andres Freund
Hi, On 2021-03-18 09:55:46 +0900, Michael Paquier wrote: > Let's see how it goes from this point, but, FWIW, I have not been able > to reproduce again my similar problem with the archive command :/ -- I suspect it might be easier to reproduce the issue with smaller WAL segments, a short checkpoin

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Masahiko Sawada
On Wed, Mar 17, 2021 at 7:21 AM Peter Geoghegan wrote: > > On Tue, Mar 16, 2021 at 6:40 AM Masahiko Sawada wrote: > > > Note that I've merged multiple existing functions in vacuumlazy.c into > > > one: the patch merges lazy_vacuum_all_indexes() and lazy_vacuum_heap() > > > into a single function

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 21:30:48 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-17 18:07:36 -0400, Tom Lane wrote: > >> Huh, interesting. I wonder why that makes the ident problem go away? > > > I spent a bit of time looking at valgrind, and it looks to be explicit > > behaviour: > > >

Re: EXPLAIN/EXPLAIN ANALYZE REFRESH MATERIALIZED VIEW

2021-03-17 Thread Japin Li
On Tue, 16 Mar 2021 at 20:13, Bharath Rupireddy wrote: > On Tue, Mar 16, 2021 at 1:15 AM Tom Lane wrote: >> >> [ Sorry for not looking at this thread sooner ] >> >> Bharath Rupireddy writes: >> > Currently, $subject is not allowed. We do plan the mat view query >> > before every refresh. I p

Re: replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 19:36:46 -0400, Tom Lane wrote: > > But it seems like we just shouldn't allocate it dynamically at all? > > max_replication_slots doesn't change during postmaster lifetime, so it > > seems like it should just be allocated once? > > Meh. I don't see a need to wire in such an ass

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Andres Freund
Hi, On 2021-03-16 16:20:37 +0900, Michael Paquier wrote: > Fujii-san has mentioned that on twitter, but one area that has changed > during the v13 cycle is aaa3aed, where the code recycling segments has > been switched from a pgrename() (with a retry loop) to a > CreateHardLinkA()+pgunlink() (with

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Justin Pryzby
On Wed, Mar 17, 2021 at 02:50:34PM -0700, Andres Freund wrote: > On 2021-03-17 16:01:58 -0400, Robert Haas wrote: > > > - why is HIDE_TOAST_COMPRESSION useful? Doesn't quite seem to be > > > comparable to HIDE_TABLEAM? > > > > Andres, what do you mean by this exactly? It's exactly the same issue

Re: replication slot stats memory bug

2021-03-17 Thread Amit Kapila
On Thu, Mar 18, 2021 at 4:55 AM Andres Freund wrote: > > Hi, > > On 2021-03-17 16:04:47 -0700, Andres Freund wrote: > > I'll push the minimal fix of forcing the allocation to happen in > > pgStatLocalContext and setting it to NULL in pgstat_clear_snapshot(). > > Done: > https://git.postgresql.org

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > On 2021-03-17 18:07:36 -0400, Tom Lane wrote: >> Huh, interesting. I wonder why that makes the ident problem go away? > I spent a bit of time looking at valgrind, and it looks to be explicit > behaviour: > >// Our goal is to construct a set of chunks that includes eve

Re: Calendar support in localization

2021-03-17 Thread Vik Fearing
On 3/17/21 3:48 PM, Tom Lane wrote: > Also, the SQL spec says in so many words > that the SQL-defined datetime types follow the Gregorian calendar. We already don't follow the SQL spec for timestamps (and I, for one, think we are better for it) so I don't think we should worry about that. -- Vik

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Michael Paquier
On Tue, Mar 16, 2021 at 11:40:12AM +0100, Magnus Hagander wrote: > If we can provide a new .EXE built with exactly the same flags as the > EDB downloads that they can just drop into a directory, I think it's a > lot easier to get that done. Yeah, multiple people have been complaining about that bu

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 18:07:36 -0400, Tom Lane wrote: > Andres Freund writes: > >> I found a way around that late last night. Need to mark the context > >> itself as an allocation. But I made a mess on the way to that and need > >> to clean the patch up before sending it (and need to drop my > >> gir

Re: replication slot stats memory bug

2021-03-17 Thread Tom Lane
Andres Freund writes: > I saw a leak in pgstat_read_statsfiles(), more precisely: > /* Allocate the space for replication slot statistics */ > replSlotStats = palloc0(max_replication_slots * > sizeof(PgStat_ReplSlotStats)); Yeah, I just found that myself. I think your fix is good.

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Julien Rouhaud
On Wed, Mar 17, 2021 at 06:32:16PM -0400, Bruce Momjian wrote: > On Wed, Mar 17, 2021 at 04:04:44PM -0400, Robert Haas wrote: > > On Wed, Mar 17, 2021 at 12:48 PM Julien Rouhaud wrote: > > > I originally suggested to make it clearer by having an enum GUC rather > > > than a > > > boolean, say com

Re: replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 16:04:47 -0700, Andres Freund wrote: > I'll push the minimal fix of forcing the allocation to happen in > pgStatLocalContext and setting it to NULL in pgstat_clear_snapshot(). Done: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5f79580ad69f6e696365bdc63bc265f45

replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, in the course of https://postgr.es/m/3471359.1615937770%40sss.pgh.pa.us I saw a leak in pgstat_read_statsfiles(), more precisely: /* Allocate the space for replication slot statistics */ replSlotStats = palloc0(max_replication_slots * sizeof(PgStat_ReplSlotStats)); the issue

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Tomas Vondra
Hi, On 3/17/21 10:43 PM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: >> Right, I was just going to point out the FPIs are not necessary - what >> matters is the presence of long streaks of WAL records touching the same >> set of blocks. But people w

Re: Calendar support in localization

2021-03-17 Thread Thomas Munro
On Thu, Mar 18, 2021 at 3:48 AM Tom Lane wrote: > Robert Haas writes: > > It's not very obvious how to scale this kind of approach to a wide > > variety of calendar types, and as Thomas says, it would much cooler to > > be able to handle all of the ones that ICU knows how to support rather > > th

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 05:04:48PM +0100, Pavel Stehule wrote: > This tree has a different direction than is usual, and then replacing the > root node is not simple. Yeah, it is not like we should redesign this whole part just for the feature discussed here, and that may impact performance as the

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 04:04:44PM -0400, Robert Haas wrote: > On Wed, Mar 17, 2021 at 12:48 PM Julien Rouhaud wrote: > > I originally suggested to make it clearer by having an enum GUC rather than > > a > > boolean, say compute_queryid = [ none | core | external ], and if set to > > external the

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Hannu Krosing
why are you using yet another special syntax for this ? would it not be better to do something like this: CREATE FUNCTION a_reall_long_and_winding_function_name(i int, out o int) LANGUAGE plpgsql AS $plpgsql$ DECLARE args function_name_alias BEGIN args.o = 2 * args.i END; $plpgsql$; or at le

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: >> I found a way around that late last night. Need to mark the context >> itself as an allocation. But I made a mess on the way to that and need >> to clean the patch up before sending it (and need to drop my >> girlfriend off first). > Unfortunately I didn't immediately fin

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 16:01:58 -0400, Robert Haas wrote: > > - why is HIDE_TOAST_COMPRESSION useful? Doesn't quite seem to be > > comparable to HIDE_TABLEAM? > > Andres, what do you mean by this exactly? It's exactly the same issue: > without this, if you change the default compression method, ever

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > Right, I was just going to point out the FPIs are not necessary - what > matters is the presence of long streaks of WAL records touching the same > set of blocks. But people with workloads where this is common likely > don't need t

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
On 3/17/21 9:58 PM, Dean Rasheed wrote: > On Wed, 17 Mar 2021 at 20:48, Dean Rasheed wrote: >> >> For reference, here is the test case I was using (which isn't really very >> good for >> catching dependence between columns): >> > > And here's a test case with much more dependence between the

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Wed, 17 Mar 2021 at 20:48, Dean Rasheed wrote: > > For reference, here is the test case I was using (which isn't really very > good for > catching dependence between columns): > And here's a test case with much more dependence between the columns: DROP TABLE IF EXISTS foo; CREATE TABLE foo (

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread John Naylor
On Wed, Mar 17, 2021 at 3:16 PM Tomas Vondra wrote: > Ummm, no attachment ;-) Oops, here it is. -- John Naylor EDB: http://www.enterprisedb.com jcn-costing-test.sql Description: Binary data

Re: non-HOT update not looking at FSM for large tuple update

2021-03-17 Thread John Naylor
On Fri, Mar 12, 2021 at 8:45 AM Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > > If this case isn't added, the lower else branch will fail to find > fitting pages for len > maxPaddedFsmRequest tuples; potentially > extending the relation when there is actually still enough space > a

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2021-03-17 Thread Simon Riggs
On Fri, 12 Mar 2021 at 22:16, Tomas Vondra wrote: > > On 1/28/21 2:33 PM, Simon Riggs wrote: > > On Thu, 28 Jan 2021 at 12:53, Masahiko Sawada wrote: > > > >> This entry has been "Waiting on Author" status and the patch has not > >> been updated since Nov 30. Are you still planning to work on thi

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Wed, 17 Mar 2021 at 19:07, Tomas Vondra wrote: > > On 3/17/21 7:54 PM, Dean Rasheed wrote: > > > > it might have been better to estimate the first case as > > > > ndistinct((a+b)) * ndistinct(c) * ndistinct(d) > > > > and the second case as > > > > ndistinct((a+b)) * ndistinct((c+d))

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 12:48 PM Julien Rouhaud wrote: > I originally suggested to make it clearer by having an enum GUC rather than a > boolean, say compute_queryid = [ none | core | external ], and if set to > external then a hook would be explicitely called. Right now, "none" and > "external"

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Robert Haas
).On Mon, Mar 15, 2021 at 6:58 PM Andres Freund wrote: > - Adding all these indirect function calls via toast_compression[] just > for all of two builtin methods isn't fun either. Yeah, it feels like this has too many layers of indirection now. Like, toast_decompress_datum() first gets TOAST_CO

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 2:17 PM Andres Freund wrote: > OTOH heap_form_flattened_tuple() has the advantage that we can optimize > it further (e.g. to do the conversion to flattened values in fill_val()) > without changing the outside API. Well, in my view, that does change the outside API, because

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread Tomas Vondra
On 3/17/21 7:59 PM, John Naylor wrote: > On Thu, Mar 11, 2021 at 12:26 PM Tomas Vondra > mailto:tomas.von...@enterprisedb.com>> > wrote: >> >> Hi, >> >> Here is an updated version of the patch series. >> >> It fixes the assert failure (just remove the multiplication from it) and >> adds a simple

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
On 3/17/21 7:54 PM, Dean Rasheed wrote: > On Wed, 17 Mar 2021 at 17:26, Tomas Vondra > wrote: >> >> My concern is that the current behavior (where we prefer expression >> stats over multi-column stats to some extent) works fine as long as the >> parts are independent, but once there's dependenc

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread John Naylor
On Thu, Mar 11, 2021 at 12:26 PM Tomas Vondra wrote: > > Hi, > > Here is an updated version of the patch series. > > It fixes the assert failure (just remove the multiplication from it) and > adds a simple regression test that exercises this. > > Based on the discussion so far, I've decided to kee

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Wed, 17 Mar 2021 at 17:26, Tomas Vondra wrote: > > My concern is that the current behavior (where we prefer expression > stats over multi-column stats to some extent) works fine as long as the > parts are independent, but once there's dependency it's probably more > likely to produce underestim

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-17 Thread Justin Pryzby
The v8 patch has the "broken constraint" problem. Also, it "fails to avoid" adding duplicate constraints: Check constraints: "c" CHECK (i IS NOT NULL AND i > 1 AND i < 2) "cc" CHECK (i IS NOT NULL AND i >= 1 AND i < 2) "p1_check" CHECK (true) "p1_i_check" CHECK (i IS NOT NULL AND

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 13:31:14 -0400, Robert Haas wrote: > On Wed, Mar 17, 2021 at 7:41 AM Dilip Kumar wrote: > > 0002: > > - Wrapper over heap_form_tuple and used in ExecEvalRow() and > > ExecEvalFieldStoreForm() > > Instead of having heap_form_flattened_tuple(), how about > heap_flatten_values(tup

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, (really need to fix my mobile phone mail program to keep the CC list...) On 2021-03-17 08:15:43 -0700, Andres Freund wrote: > On Wed, Mar 17, 2021, at 07:16, Tom Lane wrote: > > Andres Freund writes: > > > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > > Meanwhile, I'm still trying to

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-17 Thread Alvaro Herrera
On 2021-Mar-15, Alvaro Herrera wrote: > Here's a fixup patch to do it that way. I tried running the commands > you showed and one of them immediately dies with the new error message; > I can't cause the bogus constraint to show up anymore. Actually, that was a silly fix that didn't actually work

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Tomas Vondra
On 3/17/21 6:02 PM, Dmitry Dolgov wrote: >> On Wed, Mar 17, 2021 at 03:28:00AM +0100, Tomas Vondra wrote: >> Hi, >> >> I took a look at the new patch series, focusing mostly on the uniquekeys >> part. It'd be a bit tedious to explain all the review comments here, so >> attached is a patch series

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 7:41 AM Dilip Kumar wrote: > 0002: > - Wrapper over heap_form_tuple and used in ExecEvalRow() and > ExecEvalFieldStoreForm() Instead of having heap_form_flattened_tuple(), how about heap_flatten_values(tupleDesc, values, isnull) that is documented to modify the values arra

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
Hi, On 3/17/21 4:55 PM, Dean Rasheed wrote: > On Sun, 7 Mar 2021 at 21:10, Tomas Vondra > wrote: >> >> 2) ndistinct >> >> There's one thing that's bugging me, in how we handle "partial" matches. >> For each expression we track both the original expression and the Vars >> we extract from it. If w

Re: pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Tom Lane
Vik Fearing writes: > On 3/17/21 6:00 PM, Lætitia Avrot wrote: >> However, when I finally got the time to look at it in detail, I found out >> there was no way to solve the dependencies in the functions and procedures, >> so that the exported file, when re-played could lead to invalid objects. >>

Re: pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Fabrízio de Royes Mello
On Wed, Mar 17, 2021 at 2:00 PM Lætitia Avrot wrote: > > Hey hackers, > > I had this idea, that I raised and cherished like my baby to add a switch in `pg_dump` to allow exporting stored functions (and procedures) only. > > However, when I finally got the time to look at it in detail, I found out

Re: [patch] [doc] Minor variable related cleanup and rewording of plpgsql docs

2021-03-17 Thread Tom Lane
I wrote: > I took a stab at doing that, just to see what it might look like. > I thought it comes out pretty well, really -- see what you think. Hearing nothing further, I pushed that after another round of copy-editing. There's still plenty of time to revise it if anybody has further comments.

Re: pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Vik Fearing
On 3/17/21 6:00 PM, Lætitia Avrot wrote: > Hey hackers, > > I had this idea, that I raised and cherished like my baby to add a switch > in `pg_dump` to allow exporting stored functions (and procedures) only. > > However, when I finally got the time to look at it in detail, I found out > there was

pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Lætitia Avrot
Hey hackers, I had this idea, that I raised and cherished like my baby to add a switch in `pg_dump` to allow exporting stored functions (and procedures) only. However, when I finally got the time to look at it in detail, I found out there was no way to solve the dependencies in the functions and

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Dmitry Dolgov
> On Wed, Mar 17, 2021 at 03:28:00AM +0100, Tomas Vondra wrote: > Hi, > > I took a look at the new patch series, focusing mostly on the uniquekeys > part. It'd be a bit tedious to explain all the review comments here, so > attached is a patch series with a "review" patch for some of the parts. Gre

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Julien Rouhaud
On Wed, Mar 17, 2021 at 12:24:44PM -0400, Bruce Momjian wrote: > On Wed, Mar 17, 2021 at 05:16:50PM +0100, Pavel Stehule wrote: > > > > > > st 17. 3. 2021 v 17:03 odesílatel Tom Lane napsal: > > > > Bruce Momjian writes: > > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: >

Re: libpq debug log

2021-03-17 Thread Alvaro Herrera
Hello In pqTraceOutputString(), you can use the return value from fprintf to move the cursor -- no need to count chars. I still think that the message-type specific functions should print the message type, rather than having the string arrays. -- Álvaro Herrera Valdivia, Chile "La gente v

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 05:16:50PM +0100, Pavel Stehule wrote: > > > st 17. 3. 2021 v 17:03 odesílatel Tom Lane napsal: > > Bruce Momjian writes: > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > >> I still say that it's a serious mistake to sanctify a query ID >

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 17:03 odesílatel Tom Lane napsal: > Bruce Momjian writes: > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > >> I still say that it's a serious mistake to sanctify a query ID > calculation > >> method that was designed only for pg_stat_statement's needs as the one >

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 12:01:38PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > >> I still say that it's a serious mistake to sanctify a query ID calculation > >> method that was designed only for pg_stat_statement's needs as the on

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Tomas Vondra
On 2/15/21 12:18 AM, Stephen Frost wrote: > Greetings, > > ... > I think there's potential for some significant optimization going forward, but I think it's basically optimization over what we're doing today. As this is already a nontrivial patch, I'd argue for doing so separat

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 9:20 odesílatel Michael Paquier napsal: > On Wed, Mar 17, 2021 at 02:06:57PM +0800, Julien Rouhaud wrote: > > I also think that there should be a single usable top label, otherwise > it will > > lead to confusing code and it can be a source of bug. > > Okay, that's fine by me. C

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Tom Lane
Bruce Momjian writes: > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: >> I still say that it's a serious mistake to sanctify a query ID calculation >> method that was designed only for pg_stat_statement's needs as the one >> true way to do it. But that's what exposing it in a core vie

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Sun, 7 Mar 2021 at 21:10, Tomas Vondra wrote: > > 2) ndistinct > > There's one thing that's bugging me, in how we handle "partial" matches. > For each expression we track both the original expression and the Vars > we extract from it. If we can't find a statistics matching the whole > expressio

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > We are reaching the two-year mark on this feature, that everyone seems > > to agree is needed. Is any committer going to work on this to get it > > into PG 14? Should I take it? > > I still say that it's a ser

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Tom Lane
Bruce Momjian writes: > We are reaching the two-year mark on this feature, that everyone seems > to agree is needed. Is any committer going to work on this to get it > into PG 14? Should I take it? I still say that it's a serious mistake to sanctify a query ID calculation method that was design

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Sun, Mar 14, 2021 at 04:06:45PM +0800, Julien Rouhaud wrote: > Recent conflict, thanks to cfbot. v18 attached. We are reaching the two-year mark on this feature, that everyone seems to agree is needed. Is any committer going to work on this to get it into PG 14? Should I take it? I just rea

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On Wed, Mar 17, 2021, at 07:16, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > Meanwhile, I'm still trying to understand why valgrind is whining > about the rd_indexcxt identifier strings. AFAICS it shouldn't. I found a way around that late

RE: libpq debug log

2021-03-17 Thread iwata....@fujitsu.com
Hi Tsunakawa san, Alvaro san, Thank you very much for your review. It helped me a lot to make the patch better. I update patch to v26. This patch has been updated according to Tsunakawa san and Alvaro san review comments. The output is following; ``` 2021-03-17 14:43:16.411238 > Terminate

Re: Calendar support in localization

2021-03-17 Thread Tom Lane
Robert Haas writes: > It's not very obvious how to scale this kind of approach to a wide > variety of calendar types, and as Thomas says, it would much cooler to > be able to handle all of the ones that ICU knows how to support rather > than just one. But, the problem I see with using timestamptz

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: >> What I meant was that I didn't understand how there's not a leak >> danger when compilation fails halfway through, given that the context >> in question is below TopMemoryContext and that I didn't see a relevant >> TRY b

Re: Calendar support in localization

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 9:54 AM Surafel Temesgen wrote: > As you mention above whatever the calendar type is we ended up storing an > integer that represent the date so rather than re-implementing every > function and operation for every calendar we can use existing Gerigorian > implementation

Re: Calendar support in localization

2021-03-17 Thread Surafel Temesgen
On Tue, Mar 16, 2021 at 12:20 PM Thomas Munro wrote: > On Wed, Mar 17, 2021 at 6:31 AM Surafel Temesgen > wrote: > > Ethiopice calendar have 13 months so it can not be stored as date and > timestamp type and you approach seems more complicated and i suggest to > have this feature on the purpose

Re: pgsql: Add libpq pipeline mode support to pgbench

2021-03-17 Thread Daniel Verite
Fabien COELHO wrote: > For consistency with the existing \if … \endif, ISTM that it could have > been named \batch … \endbatch or \pipeline … \endpipeline? "start" mirrors "end". To me, the analogy with \if-\endif is not obvious. Grammatically \if is meant to introduce the expression aft

Re: Get memory contexts of an arbitrary backend process

2021-03-17 Thread torikoshia
On 2021-03-05 14:22, Fujii Masao wrote: On 2021/03/04 18:32, torikoshia wrote: On 2021-01-14 19:11, torikoshia wrote: Since pg_get_target_backend_memory_contexts() waits to dump memory and it could lead dead lock as below.   - session1   BEGIN; TRUNCATE t;   - session2   BEGIN; TRUNCATE t; -

  1   2   >