Re: Fix handling of unlogged tables in FOR ALL TABLES publications

2019-03-13 Thread Amit Langote
On 2019/03/14 15:03, Kyotaro HORIGUCHI wrote: > At Thu, 14 Mar 2019 11:30:12 +0900, Amit Langote > wrote in > <59e5a734-9e06-1035-385b-626717581...@lab.ntt.co.jp> >> On 2019/03/13 21:03, Peter Eisentraut wrote: >>> If a FOR ALL TABLES publication exists, unlogged tables are ignored >>> for publi

Improve the generic plan mechanism

2019-03-13 Thread Kato, Sho
Hi, hackers I want to improve the generic plan mechanism and speed up the UPDATE/DELETE planning of a table partitioned into thousands. However, I am not sure if this is realistic, I would like advice. The current generic plan creates access plans for all child tables without using the paramete

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread Amit Kapila
On Thu, Mar 14, 2019 at 7:08 AM John Naylor wrote: > > > [segfault problems] > > This now seems spurious. I ran make distclean, git pull, reapplied the > patch (leaving out the gettimeofday() calls), and now my upgrade perf > test works with default compiler settings. Not sure what happened, but >

RE: Timeout parameters

2019-03-13 Thread Fabien COELHO
HI think that your patch is responsible for the added option at least. I agree that the underlying issue that other parameters should probably also be reused, which would be a bug fix, does not belong to this thread. This doesn't seem to be a bug. \connect just establishes a new connection,

Re: Fix handling of unlogged tables in FOR ALL TABLES publications

2019-03-13 Thread Kyotaro HORIGUCHI
At Thu, 14 Mar 2019 11:30:12 +0900, Amit Langote wrote in <59e5a734-9e06-1035-385b-626717581...@lab.ntt.co.jp> > On 2019/03/13 21:03, Peter Eisentraut wrote: > > If a FOR ALL TABLES publication exists, unlogged tables are ignored > > for publishing changes. But CheckCmdReplicaIdentity() would s

Re: Progress reporting for pg_verify_checksums

2019-03-13 Thread Michael Paquier
On Thu, Mar 14, 2019 at 11:54:17AM +0900, Kyotaro HORIGUCHI wrote: > Why this patch changes the behavior for temprary directories? It > seems like a bug fix of pg_checksums. Oops, that's a thinko from 5c995139, so fixed. Note this has no actual consequence though as PG_TEMP_FILE_PREFIX and PG_TEM

Re: Timeout parameters

2019-03-13 Thread Kyotaro HORIGUCHI
At Thu, 14 Mar 2019 03:33:20 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FBC4191@G01JPEXMBYT05> > From: Robert Haas [mailto:robertmh...@gmail.com] > > But that's not what it will do. As long as the server continues to > > dribble out protocol messages from time to ti

Re: Sparse bit set data structure

2019-03-13 Thread Andrey Borodin
Hi! > 14 марта 2019 г., в 0:18, Heikki Linnakangas написал(а): > <0001-Add-SparseBitset-to-hold-a-large-set-of-64-bit-ints-.patch><0002-Andrey-Borodin-s-test_blockset-tool-adapted-for-Spar.patch> That is very interesting idea. Basically, B-tree and radix tree is a tradeoff between space and tim

Re: current_logfiles not following group access and instead follows log_file_mode permissions

2019-03-13 Thread Michael Paquier
On Tue, Mar 12, 2019 at 04:08:53PM -0400, Robert Haas wrote: > Anybody who has permission to read the log files but not the data > directory will presumably hit the directory-level permissions on > $PGDATA before the issue of the permissions on current_logfiles() per > se become relevant, except in

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Tom Lane
Dean Rasheed writes: > I'm amazed that jacana's asinh() returned -0 for an input of +0. Even more amusingly, it returns NaN for acosh('infinity'), cf https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2019-03-14%2003%3A00%3A34 Presumably that means they calculated "infinity - infi

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Michael Paquier
On Wed, Mar 13, 2019 at 08:56:33PM +0900, Michael Paquier wrote: > On Wed, Mar 13, 2019 at 12:09:24PM +0100, Michael Banck wrote: > > The attached patch should do the above, on top of Michael's last > > patchset. > > What you are doing here looks like a good defense in itself. More thoughts on th

Proposal to suppress errors thrown by to_reg*()

2019-03-13 Thread Takuma Hoshiai
Hi, hackers, According to the document, "to_reg* functions return null rather than throwing an error if the name is not found", but this is not the case if the arguments to those functions are schema qualified and the caller does not have access permission of the schema even if the table (or other

Re: BUG #15668: Server crash in transformPartitionRangeBounds

2019-03-13 Thread Michael Paquier
On Wed, Mar 13, 2019 at 03:17:47PM +0900, Amit Langote wrote: > but on HEAD, you get: > > create table foo (a int default (avg(foo.a))); > ERROR: aggregate functions are not allowed in DEFAULT expressions I actually think that what you propose here makes more sense than what HEAD does because th

Re: Tid scan improvements

2019-03-13 Thread David Rowley
On Mon, 4 Feb 2019 at 18:37, Edmund Horner wrote: > 2. v6-0002-Support-backward-scans-over-restricted-ranges-in-hea.patch > 3. v6-0003-Support-range-quals-in-Tid-Scan.patch > 4. v6-0004-TID-selectivity-reduce-the-density-of-the-last-page-.patch These ones need a rebase. -- David Rowley

Re: Special role for subscriptions

2019-03-13 Thread Kyotaro HORIGUCHI
At Wed, 13 Mar 2019 23:03:26 -0400, Stephen Frost wrote in <20190314030326.gq6...@tamriel.snowman.net> > Greetings, > > * Robert Haas (robertmh...@gmail.com) wrote: > > On Mon, Mar 11, 2019 at 10:39 PM Michael Paquier > > wrote: > > > On Mon, Mar 11, 2019 at 06:32:10PM -0700, Jeff Davis wrote:

Re: Tid scan improvements

2019-03-13 Thread David Rowley
On Mon, 4 Feb 2019 at 18:37, Edmund Horner wrote: > 1. v6-0001-Add-selectivity-estimate-for-CTID-system-variables.patch I think 0001 is good to go. It's a clear improvement over what we do today. (t1 = 1 million row table with a single int column.) Patched: # explain (analyze, timing off) selec

RE: Timeout parameters

2019-03-13 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > But that's not what it will do. As long as the server continues to > dribble out protocol messages from time to time, the timeout will > never fire no matter how much time passes. I saw a system once where > every 8kB read took many seconds to co

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Kyotaro HORIGUCHI
At Wed, 13 Mar 2019 23:18:27 -0400, Tom Lane wrote in <2503.1552533...@sss.pgh.pa.us> tgl> Andrew Dunstan writes: tgl> > Or we could possibly call the function and then turn a result of -0 into 0? tgl> tgl> But -0 is the correct output if the input is -0. So that approach tgl> requires distin

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Tom Lane
Andrew Dunstan writes: > Or we could possibly call the function and then turn a result of -0 into 0? But -0 is the correct output if the input is -0. So that approach requires distinguishing -0 from 0, which is annoyingly difficult. regards, tom lane

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Andrew Dunstan
On 3/13/19 5:56 PM, Tom Lane wrote: > Michael Paquier writes: >> On Tue, Mar 12, 2019 at 11:16:42PM -0400, Tom Lane wrote: >>> I'm inclined to leave it as-is for a day or so and see if any >>> other failures turn up, before deciding what to do about it. >> Fine by me. > Well, so far jacana is th

Re: Special role for subscriptions

2019-03-13 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Mar 11, 2019 at 10:39 PM Michael Paquier wrote: > > On Mon, Mar 11, 2019 at 06:32:10PM -0700, Jeff Davis wrote: > > > * Is the original idea of a special role still viable? > > > > In my opinion, that part may be valuable. The la

Re: Progress reporting for pg_verify_checksums

2019-03-13 Thread Kyotaro HORIGUCHI
Hello. At Wed, 13 Mar 2019 16:25:15 +0900, Michael Paquier wrote in <20190313072515.gb2...@paquier.xyz> > On Wed, Mar 13, 2019 at 07:22:28AM +0100, Fabien COELHO wrote: > > Does not apply because of the renaming committed by Michaël. > > > > Could you rebase? > > This stuff touches pg_checksum

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Tom Lane
Robert Haas writes: > On Wed, Mar 13, 2019 at 10:39 PM Tom Lane wrote: >> No, but that's not the hazard. I have a very fresh-in-mind example: >> at one point while tweaking Laetitia's patch, I'd accidentally changed >> datanh so that it called tanh not atanh. The previous set of tests did >> no

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 10:39 PM Tom Lane wrote: > Robert Haas writes: > > On Wed, Mar 13, 2019 at 8:49 PM Tom Lane wrote: > >> Meh. As I said before, we're not in the business of improving on what > >> libm does --- if someone has a beef with the results, they need to take > >> it to their pla

Re: Timeout parameters

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 10:20 PM Tsunakawa, Takayuki wrote: > I think the purpose of socket_timeout is to avoid infinite or unduely long > wait and return response to users, where other existing timeout parameters > wouldn't help. For example, OS's process scheduling or paging/swapping > probl

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Tom Lane
Robert Haas writes: > On Wed, Mar 13, 2019 at 8:49 PM Tom Lane wrote: >> Meh. As I said before, we're not in the business of improving on what >> libm does --- if someone has a beef with the results, they need to take >> it to their platform's libm maintainer, not us. The point of testing >> th

Re: Fix handling of unlogged tables in FOR ALL TABLES publications

2019-03-13 Thread Amit Langote
On 2019/03/13 21:03, Peter Eisentraut wrote: > If a FOR ALL TABLES publication exists, unlogged tables are ignored > for publishing changes. But CheckCmdReplicaIdentity() would still > check in that case that such a table has a replica identity set before > accepting updates. That is useless, so

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 8:49 PM Tom Lane wrote: > Meh. As I said before, we're not in the business of improving on what > libm does --- if someone has a beef with the results, they need to take > it to their platform's libm maintainer, not us. The point of testing > this at all is just to ensure

RE: Timeout parameters

2019-03-13 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > The first thing I notice about the socket_timeout patch is that the > documentation is definitely wrong: Agreed. I suppose the description should be clearer about: * the purpose and what situation this timeout will help: not for canceling a lon

Re: ToDo: show size of partitioned table

2019-03-13 Thread Amit Langote
On 2019/03/14 2:11, Pavel Stehule wrote: > st 13. 3. 2019 v 8:02 odesílatel Amit Langote > napsal: >> Not a native speaker either, but I like Justin's changes. Although I >> noticed that he missed changing one sentence to look like other similar >> sentences. >> >> What Justin did: >> >> -

Re: Using the return value of strlcpy() and strlcat()

2019-03-13 Thread Tom Lane
Ashwin Agrawal writes: > On Wed, Mar 13, 2019 at 9:51 AM Tom Lane wrote: >> I don't think that's a safe transformation: what strlcpy returns is >> strlen(src), which might be different from what it was actually >> able to fit into the destination. >> Sure, they're equivalent if no truncation occu

RE: Timeout parameters

2019-03-13 Thread Tsunakawa, Takayuki
From: Fabien COELHO [mailto:coe...@cri.ensmp.fr] > >> If the user reconnects, eg "\c db", the setting is lost. The > >> re-connection handling should probably take care of this parameter, and > maybe others. > > I think your opinion is reasonable, but it seems not in this thread. > > HI think that

Re: Using the return value of strlcpy() and strlcat()

2019-03-13 Thread Ashwin Agrawal
On Wed, Mar 13, 2019 at 9:51 AM Tom Lane wrote: > ilm...@ilmari.org (Dagfinn Ilmari =?utf-8?Q?Manns=C3=A5ker?=) writes: > > [ let's convert > > + strlcpy(buf + buflen, name, NAMEDATALEN); > > + buflen += strlen(buf + buflen); > > to > > + buflen += strlcpy(buf

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Amit Langote
On 2019/03/14 5:18, Tom Lane wrote: > Robert Haas writes: >> On Wed, Mar 13, 2019 at 3:14 PM Tom Lane wrote: >>> Meanwhile, who's going to take point on cleaning up rd_partcheck? >>> I don't really understand this code well enough to know whether that >>> can share one of the existing partitionin

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread John Naylor
> [segfault problems] This now seems spurious. I ran make distclean, git pull, reapplied the patch (leaving out the gettimeofday() calls), and now my upgrade perf test works with default compiler settings. Not sure what happened, but hopefully we can move forward. -- John Naylorh

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Tom Lane
Dean Rasheed writes: > It's not unreasonable to expect these functions to be accurate to within > the last 1 or 2 digits, so testing with extra_float_digits or whatever > seems reasonable, but I think a wider variety of test inputs is required. Meh. As I said before, we're not in the business of

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Dean Rasheed
On Wed, 13 Mar 2019, 21:56 Tom Lane, wrote: > > Of these, probably the least bad is #3, even though it might require > a few rounds of experimentation to find the best extra_float_digits > setting to use. I'll go try it without any roundoff, just to see > what the raw results look like ... > Y

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Michael Paquier
On Wed, Mar 13, 2019 at 12:24:21PM +0100, Magnus Hagander wrote: > Enabling or disabling the checksums offline on the master quite clearly > requires a rebuild of the standby, there is no other way (this is one of > the reasons for the online enabling in that patch, so I still hope we can > get tha

Re: pg_rewind : feature to rewind promoted standby is broken!

2019-03-13 Thread Michael Paquier
On Thu, Mar 14, 2019 at 12:15:57AM +0530, Mithun Cy wrote: > I have not looked into the patch but quick test show it has fixed the above > issue. Thanks for confirming, Mythun. I'll think about the logic of this patch for a couple of days in the background, then I'll try to commit it likely at th

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2019-03-13 Thread Peter Geoghegan
On Tue, Mar 12, 2019 at 11:40 AM Robert Haas wrote: > I think it's pretty clear that we have to view that as acceptable. I > mean, we could reduce contention even further by finding a way to make > indexes 40% larger, but I think it's clear that nobody wants that. I found this analysis of bloat

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Michael Paquier
On Wed, Mar 13, 2019 at 02:43:39PM +0300, Sergei Kornilov wrote: > Seems good. And I think we need backpath this check to pg11. similar > to cross-version compatibility checks +fprintf(stderr, _("%s: data directory block size %d is different to compiled-in block size %d.\n"), +pr

Re: Inadequate executor locking of indexes

2019-03-13 Thread David Rowley
Thanks for having a look at this. On Wed, 13 Mar 2019 at 22:45, Amit Langote wrote: > I have one question about the relation between idxlockmode and > rellockmode? From skimming the patch, it appears that they're almost > always set to the same value. If so, why not use rellockmode for index >

Re: REINDEX CONCURRENTLY 2.0

2019-03-13 Thread Peter Eisentraut
On 2019-03-13 15:13, Sergei Kornilov wrote: > Patch is marked as target version 12, but is inactive few weeks long. I think > many users want this feature and patch is in good shape. We have open > questions on this thread? > > Latest patch still can be aplied cleanly; it builds and pass tests.

Re: Using condition variables to wait for checkpoints

2019-03-13 Thread Thomas Munro
On Thu, Mar 14, 2019 at 1:15 AM Robert Haas wrote: > On Tue, Mar 12, 2019 at 7:12 PM Andres Freund wrote: > > Having useful infrastructure is sure cool. > > Yay! +1 I renamed the CVs because the names I had used before broke the convention that variables named ckpt_* are protected by ckpt_lck,

Re: pgsql: Add support for hyperbolic functions, as well as log10().

2019-03-13 Thread Tom Lane
Michael Paquier writes: > On Tue, Mar 12, 2019 at 11:16:42PM -0400, Tom Lane wrote: >> I'm inclined to leave it as-is for a day or so and see if any >> other failures turn up, before deciding what to do about it. > Fine by me. Well, so far jacana is the only critter that's shown any problem. I

Re: [HACKERS] Block level parallel vacuum

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:56 AM Masahiko Sawada wrote: > I don't have a strong opinion but the using a Node would be more > suitable in the future when we add more options to vacuum. And it > seems to me that it's unlikely to change a Node to a plain struct. So > there is an idea of doing it now a

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Andres Freund
Hi, On 2019-03-13 17:10:55 -0400, Tom Lane wrote: > There's already a mechanism in there to suppress child contexts after > 100 or so, which would almost inevitably kick in on the relcache if we > did this. So I don't believe we'd have a problem with the context dumps > getting too long --- more

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
Andres Freund writes: > On 2019-03-13 16:50:53 -0400, Robert Haas wrote: >> On Wed, Mar 13, 2019 at 4:38 PM Robert Haas wrote: >>> On Wed, Mar 13, 2019 at 4:18 PM Tom Lane wrote: Off topic for the moment, since this clearly wouldn't be back-patch material, but I'm starting to wonder if

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Andres Freund
Hi, On 2019-03-13 16:50:53 -0400, Robert Haas wrote: > On Wed, Mar 13, 2019 at 4:38 PM Robert Haas wrote: > > On Wed, Mar 13, 2019 at 4:18 PM Tom Lane wrote: > > > Off topic for the moment, since this clearly wouldn't be back-patch > > > material, but I'm starting to wonder if we should just hav

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 4:38 PM Robert Haas wrote: > On Wed, Mar 13, 2019 at 4:18 PM Tom Lane wrote: > > Off topic for the moment, since this clearly wouldn't be back-patch > > material, but I'm starting to wonder if we should just have a context > > for each relcache entry and get rid of most or

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 4:18 PM Tom Lane wrote: > Off topic for the moment, since this clearly wouldn't be back-patch > material, but I'm starting to wonder if we should just have a context > for each relcache entry and get rid of most or all of the retail > cleanup logic in RelationDestroyRelatio

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Justin Pryzby
Hi, On Wed, Mar 13, 2019 at 01:25:11PM -0400, Robert Haas wrote: > On Wed, Mar 13, 2019 at 1:19 PM Tom Lane wrote: > > Oh, and yes, I think QueuePartitionConstraintValidation's usage > > is an unacceptable abuse of INFO level. I'm surprised we haven't > > gotten complaints about it yet. > > Per

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
Robert Haas writes: > On Wed, Mar 13, 2019 at 3:14 PM Tom Lane wrote: >> Meanwhile, who's going to take point on cleaning up rd_partcheck? >> I don't really understand this code well enough to know whether that >> can share one of the existing partitioning-related sub-contexts. > To your questio

Re: Sparse bit set data structure

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 3:18 PM Heikki Linnakangas wrote: > I started to consider rewriting the data structure into something more > like B-tree. Then I remembered that I wrote a data structure pretty much > like that last year already! We discussed that on the "Vacuum: allow > usage of more than

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 3:14 PM Tom Lane wrote: > Yeah, but usually there's some comment pointing it out. I also wonder > if there aren't corner-case bugs; it seems a bit bogus for example that > rd_pdcxt is created without any thought as to whether it might be set > already. It's not clear whet

Sparse bit set data structure

2019-03-13 Thread Heikki Linnakangas
Hi, I was reviewing Andrey Borodin's patch for GiST VACUUM [1], which includes a new "block set" data structure, to track internal and empty pages while vacuuming a GiST. The blockset data structure was a pretty simple radix tree, that can hold a set of BlockNumbers. The memory usage of the

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
Robert Haas writes: > On Wed, Mar 13, 2019 at 2:26 PM Tom Lane wrote: >> I think that RelationBuildPartitionDesc could use some additional cleanup >> or at least better commenting. In particular, it's neither documented nor >> obvious to the naked eye why rel->rd_partdesc mustn't get set till th

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 2:26 PM Tom Lane wrote: > OK, in that case it's definitely all the temporary data that gets created > that is the problem. I've not examined your patch in great detail but > it looks plausible for fixing that. Cool. > I think that RelationBuildPartitionDesc could use som

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 2:21 PM Alvaro Herrera wrote: > On 2019-Mar-13, Robert Haas wrote: > > On Wed, Mar 13, 2019 at 1:38 PM Alvaro Herrera > > wrote: > > > A bit, yes, but not overly so, and it's less fragile that not having > > > such a protection. Anything that allocates in CacheMemoryCont

Re: pg_rewind : feature to rewind promoted standby is broken!

2019-03-13 Thread Mithun Cy
On Wed, Mar 13, 2019 at 1:38 PM Michael Paquier wrote: > On Tue, Mar 12, 2019 at 06:23:01PM +0900, Michael Paquier wrote: > > And you are pointing out to the correct commit. The issue is that > > process_target_file() has added a call to check_file_excluded(), and > > this skips all the folders

Re: Compressed TOAST Slicing

2019-03-13 Thread Paul Ramsey
> On Mar 13, 2019, at 9:32 AM, Andrey Borodin wrote: > > > >> 13 марта 2019 г., в 21:05, Paul Ramsey >> написал(а): >> >> Here is a new (final?) patch ... >> >> > > This check > > @@ -744,6 +748,8 @@ pglz_decompress(const char *source, int32 slen, char > *dest, >

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
Alvaro Herrera writes: > You do? Unless I misunderstood, your options are: > 1. (the patch you attached) create a temporary memory context that is > used for everything, then at the end copy the good stuff to CacheMemCxt > (or a sub-context thereof). This still needs to copy. > 2. create a tem

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
Robert Haas writes: >> Dunno if that's related to hyrax's issue, though. > It's related in the sense that it's a leak, and any leak will tend to > run the system out of memory more easily, but what I observed was a > large leak into MessageContext, and that would be a leak into > CacheMemoryConte

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Alvaro Herrera
On 2019-Mar-13, Robert Haas wrote: > On Wed, Mar 13, 2019 at 1:38 PM Alvaro Herrera > wrote: > > A bit, yes, but not overly so, and it's less fragile that not having > > such a protection. Anything that allocates in CacheMemoryContext needs > > to be very careful anyway. > > True, but I think

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:30 PM Tom Lane wrote: > Ah, here we are: it was rel->rd_partcheck. I'm not sure exactly how > complicated that structure can be, but I'm pretty sure that this is > a laughably inadequate method of cleaning it up: > > if (relation->rd_partcheck) >

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:38 PM Alvaro Herrera wrote: > A bit, yes, but not overly so, and it's less fragile that not having > such a protection. Anything that allocates in CacheMemoryContext needs > to be very careful anyway. True, but I think it's more fragile than either of the options I prop

Re: Timeout parameters

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:25 PM Fabien COELHO wrote: > Hmmm... ISTM that we are not talking about the same patch... You are correct! I was talking about the patches that allow user control of TCP_USER_TIMEOUT, which is apparently totally different from the socket_timeout patch that you're talkin

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Alvaro Herrera
On 2019-Mar-13, Robert Haas wrote: > On Wed, Mar 13, 2019 at 12:42 PM Alvaro Herrera > wrote: > > I remember going over this code's memory allocation strategy a bit to > > avoid the copy while not incurring potential leaks CacheMemoryContext; > > as I recall, my idea was to use two contexts, one

RE: Timeout parameters

2019-03-13 Thread Fabien COELHO
Hello Fabien-san. The 2nd patch is 700 KB, I think that there is a unvoluntary file copy. If the user reconnects, eg "\c db", the setting is lost. The re-connection handling should probably take care of this parameter, and maybe others. I think your opinion is reasonable, but it seems not

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
I wrote: > I recall having noticed someplace where I thought the relcache partition > support was simply failing to make provisions for cleaning up a cached > structure at relcache entry drop, but I didn't have time to pursue it > right then. Let me see if I can reconstruct what I was worried abou

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:15 PM Tom Lane wrote: > I'm a bit confused as to why there's an issue here at all. The usual > plan for computed-on-demand relcache sub-structures is that we compute > a working copy that we're going to return to the caller using the > caller's context (which is presumab

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:19 PM Tom Lane wrote: > Oh, and yes, I think QueuePartitionConstraintValidation's usage > is an unacceptable abuse of INFO level. I'm surprised we haven't > gotten complaints about it yet. Perhaps that's because users aren't as direly opposed to informational messages f

Re: Timeout parameters

2019-03-13 Thread Fabien COELHO
Hello Robert, Also, I do not see the downside of sending a cancel query before severing the connection. If it is not processed, too bad, but if it is then it is for the better. If the network connection is dead, which is the situation the patch intends to detect, Hmmm... ISTM that we are n

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 1:09 PM Tom Lane wrote: > Sergei Kornilov writes: > >> Ugh, I guess so. Or how about changing the message itself to use > >> INFO, like we already do in QueuePartitionConstraintValidation? > > > Fine for me. But year ago this was implemented in my patch and Tom voted > >

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Tom Lane
I wrote: > Sergei Kornilov writes: >>> Ugh, I guess so. Or how about changing the message itself to use >>> INFO, like we already do in QueuePartitionConstraintValidation? >> Fine for me. But year ago this was implemented in my patch and Tom voted >> against using INFO level for such purpose: >

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Tom Lane
Robert Haas writes: > On Wed, Mar 13, 2019 at 12:42 PM Alvaro Herrera > wrote: >> I remember going over this code's memory allocation strategy a bit to >> avoid the copy while not incurring potential leaks CacheMemoryContext; >> as I recall, my idea was to use two contexts, one of which is tempor

Re: Timeout parameters

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 2:05 AM Fabien COELHO wrote: > Also, I do not see the downside of sending a cancel query before severing > the connection. If it is not processed, too bad, but if it is then it is > for the better. If the network connection is dead, which is the situation the patch intends

Re: ToDo: show size of partitioned table

2019-03-13 Thread Pavel Stehule
st 13. 3. 2019 v 8:02 odesílatel Amit Langote napsal: > On 2019/02/22 1:41, Pavel Stehule wrote: > > čt 21. 2. 2019 v 0:56 odesílatel Justin Pryzby > > napsal: > > > >> On Sat, Feb 16, 2019 at 10:52:35PM +0100, Pavel Stehule wrote: > >>> I like your changes. I merged all - updated patch is attac

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Tom Lane
Sergei Kornilov writes: >> Ugh, I guess so. Or how about changing the message itself to use >> INFO, like we already do in QueuePartitionConstraintValidation? > Fine for me. But year ago this was implemented in my patch and Tom voted > against using INFO level for such purpose: > https://www.po

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Magnus Hagander
On Wed, Mar 13, 2019 at 4:51 PM Michael Banck wrote: > Hi, > > Am Mittwoch, den 13.03.2019, 12:43 +0100 schrieb Magnus Hagander: > > I think this is dangerous enough that it needs to be enforced and not > > documented. > > Changing the cluster ID might have some other side-effects, I think > ther

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Magnus Hagander
On Wed, Mar 13, 2019 at 4:46 PM Michael Banck wrote: > Hi, > > Am Mittwoch, den 13.03.2019, 12:24 +0100 schrieb Magnus Hagander: > > On Wed, Mar 13, 2019 at 11:54 AM Sergei Kornilov wrote: > > > One new question from me: how about replication? > > > Case: primary+replica, we shut down primary an

Re: Using the return value of strlcpy() and strlcat()

2019-03-13 Thread Tom Lane
ilm...@ilmari.org (Dagfinn Ilmari =?utf-8?Q?Manns=C3=A5ker?=) writes: > [ let's convert > + strlcpy(buf + buflen, name, NAMEDATALEN); > + buflen += strlen(buf + buflen); > to > + buflen += strlcpy(buf + buflen, name, NAMEDATALEN); > ] I don't think that's a safe

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 12:42 PM Alvaro Herrera wrote: > I remember going over this code's memory allocation strategy a bit to > avoid the copy while not incurring potential leaks CacheMemoryContext; > as I recall, my idea was to use two contexts, one of which is temporary > and used for any poten

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Sergei Kornilov
Hi > Ugh, I guess so. Or how about changing the message itself to use > INFO, like we already do in QueuePartitionConstraintValidation? Fine for me. But year ago this was implemented in my patch and Tom voted against using INFO level for such purpose: https://www.postgresql.org/message-id/1142.

Re: hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Alvaro Herrera
On 2019-Mar-13, Robert Haas wrote: > RelationBuildPartitionDesc() creates basically all of the data > structures it needs and then copies them into rel->rd_pdcxt, which has > always seemed a bit inefficient to me. Another way to redesign this > would be to have the function create a temporary con

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 12:20 AM Kyotaro HORIGUCHI wrote: > +Multivariate MCV (most-common values) lists are a straightforward extension > of > > "lists are *a*" is wrong? No, that's correct. Not sure exactly what your concern is, but it's probably related to the fact that the first parent of

Re: GIN indexes on an = ANY(array) clause

2019-03-13 Thread Tom Lane
Corey Huinker writes: > A client had an issue with a where that had a where clause something like > this: > WHERE 123456 = ANY(integer_array_column) > I was surprised that this didn't use the pre-existing GIN index on > integer_array_column, whereas recoding as > WHERE ARRAY[123456] <@ integer_arr

Re: Compressed TOAST Slicing

2019-03-13 Thread Andrey Borodin
> 13 марта 2019 г., в 21:05, Paul Ramsey написал(а): > > Here is a new (final?) patch ... > > This check @@ -744,6 +748,8 @@ pglz_decompress(const char *source, int32 slen, char *dest, { *dp = dp[-off];

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Robert Haas
On Wed, Mar 13, 2019 at 11:17 AM Sergei Kornilov wrote: > > The buildfarm thinks additional nitpicking is needed. > > hm. Patch was committed with debug1 level tests and many animals uses > log_statement = 'all'. Therefore they have additional line in result: LOG: > statement: alter table pg_cl

hyrax vs. RelationBuildPartitionDesc

2019-03-13 Thread Robert Haas
Hi, Amit Kapila pointed out to be that there are some buidfarm failures on hyrax which seem to have started happening around the time I committed 898e5e3290a72d288923260143930fb32036c00c. It failed like this once: 2019-03-07 19:57:40.231 EST [28073:11] DETAIL: Failed process was running: /* sam

Re: Compressed TOAST Slicing

2019-03-13 Thread Paul Ramsey
On Mar 13, 2019, at 8:25 AM, Paul Ramsey wrote:On Mar 13, 2019, at 3:09 AM, Tomas Vondra wrote:On 3/13/19 3:19 AM, Michael Paquier wrote:On Tue, Mar 12, 2019 at 07:01:17PM -0700, Andres Freund wrote:I don't think this is even close to popul

Re: Checksum errors in pg_stat_database

2019-03-13 Thread Julien Rouhaud
On Wed, Mar 13, 2019 at 4:53 PM Julien Rouhaud wrote: > > On Sun, Mar 10, 2019 at 1:13 PM Julien Rouhaud wrote: > > > > On Sat, Mar 9, 2019 at 7:58 PM Julien Rouhaud wrote: > > > > > > On Sat, Mar 9, 2019 at 7:50 PM Magnus Hagander > > > wrote: > > > > > > > > On Sat, Mar 9, 2019 at 10:41 AM J

Using the return value of strlcpy() and strlcat()

2019-03-13 Thread Dagfinn Ilmari Mannsåker
Hi hackers, Over in the "Include all columns in default names for foreign key constraints" thread[1], I noticed the patch added the following: + strlcpy(buf + buflen, name, NAMEDATALEN); + buflen += strlen(buf + buflen); Seeing as strlcpy() returns the copied length,

Re: Checksum errors in pg_stat_database

2019-03-13 Thread Julien Rouhaud
On Sun, Mar 10, 2019 at 1:13 PM Julien Rouhaud wrote: > > On Sat, Mar 9, 2019 at 7:58 PM Julien Rouhaud wrote: > > > > On Sat, Mar 9, 2019 at 7:50 PM Magnus Hagander wrote: > > > > > > On Sat, Mar 9, 2019 at 10:41 AM Julien Rouhaud wrote: > > >> > > >> Sorry, I have again new comments after a l

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Michael Banck
Hi, Am Mittwoch, den 13.03.2019, 12:43 +0100 schrieb Magnus Hagander: > I think this is dangerous enough that it needs to be enforced and not > documented. Changing the cluster ID might have some other side-effects, I think there are several cloud-native 3rd party solutions that use the cluster I

Re: Offline enabling/disabling of data checksums

2019-03-13 Thread Michael Banck
Hi, Am Mittwoch, den 13.03.2019, 12:24 +0100 schrieb Magnus Hagander: > On Wed, Mar 13, 2019 at 11:54 AM Sergei Kornilov wrote: > > One new question from me: how about replication? > > Case: primary+replica, we shut down primary and enable checksum, and > > "started streaming WAL from primary" wi

Re: WIP: Avoid creation of the free space map for small tables

2019-03-13 Thread Amit Kapila
On Wed, Mar 13, 2019 at 7:42 PM John Naylor wrote: > > On Wed, Mar 13, 2019 at 8:18 PM Amit Kapila wrote: > > > First, I had a problem: On MacOS with their "gcc" wrapper around > > > clang, I got a segfault 11 when compiled with no debugging symbols. > > > > > > > Did you get this problem with th

Re: Compressed TOAST Slicing

2019-03-13 Thread Paul Ramsey
> On Mar 13, 2019, at 3:09 AM, Tomas Vondra > wrote: > > On 3/13/19 3:19 AM, Michael Paquier wrote: >> On Tue, Mar 12, 2019 at 07:01:17PM -0700, Andres Freund wrote: >>> I don't think this is even close to popular enough to incur the >>> maybe of a separate function / more complicated interfa

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Sergei Kornilov
Hi > The buildfarm thinks additional nitpicking is needed. hm. Patch was committed with debug1 level tests and many animals uses log_statement = 'all'. Therefore they have additional line in result: LOG: statement: alter table pg_class alter column relname drop not null; and similar for other

Re: Minimal logical decoding on standbys

2019-03-13 Thread tushar
Hi , I am getting a server crash on standby while executing pg_logical_slot_get_changes function   , please refer this scenario Master cluster( ./initdb -D master) set wal_level='hot_standby in master/postgresql.conf file start the server , connect to  psql terminal and create a physical repl

  1   2   >