Re: MCV lists for highly skewed distributions

2018-03-17 Thread Dean Rasheed
> On 03/11/2018 10:23 AM, Dean Rasheed wrote: >> I'm moving this back to a status of "Needs review" partly because the >> code has changed significantly, but also because I want to do more >> testing, particularly with larger datasets. >> John, Tomas, Thanks for looking at this, and sorry for my

Re: [HACKERS] Restrict concurrent update/delete with UPDATE of partition key

2018-03-17 Thread Amit Kapila
On Mon, Mar 12, 2018 at 6:33 PM, amul sul wrote: > On Mon, Mar 12, 2018 at 11:45 AM, amul sul wrote: >> On Sat, Mar 10, 2018 at 5:25 PM, Amit Kapila wrote: >>> complete CTID. >> >> Sure, will do that. > > I did the aforementioned changes in the attached patch, thanks. > --- a/src/include/storag

[GSOC 18] Perf Farm——Description of patch

2018-03-17 Thread Hongyuan Ma
Hi, This email is a description of 0001-add-apps-directory.patch. In this patch, I created the apps directory and created two applications, test_item and user_operation. The test_item application model classes are: TestBranch, TestItem The model classes in the user_operation application are: Alias

Re: SSL passphrase prompt external command

2018-03-17 Thread Peter Eisentraut
On 3/16/18 12:38, Daniel Gustafsson wrote: >> Maybe this is a bit too cute. We could instead add another setting >> "ssl_passphrase_command_support_reload”. > I think thats a good idea, this feels like an easy thing to be confused about > and get wrong (which I might have done with the above). >

strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
Hi, I happened to be updating our machine running our buildfarm animals, and I noticed something quite strange - the machine was unexpectedly running out of disk space, which is rather suspicious as it's running just the regression tests :-/ After a bit of investigation, I found this: # pwd /mnt

Problems with Error Messages wrt Domains, Checks

2018-03-17 Thread john frazer
Today I realized a number of points where PostgreSQL (v10.3) is rather lackluster and sparse in its error messages. The first point is illustrated by this code: drop schema if exists X cascade;create schema X; create domain X.an_illegal_regex as text check ( value ~ '(' ); create table X.table_wi

Re: disable SSL compression?

2018-03-17 Thread Peter Eisentraut
On 3/11/18 12:07, Magnus Hagander wrote: > I think it's worth mentioning in the docs around "it's now considered > insecure" that it's still an option to use if compression is the main > thing one is looking for, rather than security. As in, it doesn't make > it any less secure than no ssl at all.

ECPG oracle mode test program patch

2018-03-17 Thread Shinoda, Noriyoshi
Hi, Hackers The attached small patch is a modification to the test program of the function recently added to ECPG. [Add Oracle like handling of char arrays.] https://git.postgresql.org/pg/commitdiff/3b7ab4380440d7b14ee390fabf39f6d87d7491e2 When the char_array program is executed, the follow

Re: Re: Re: [GSOC 18] Performance Farm Project——Initialization Project

2018-03-17 Thread Magnus Hagander
On Wed, Mar 14, 2018 at 10:33 PM, Dave Page wrote: > Hi > > On Tue, Mar 13, 2018 at 11:31 PM, Hongyuan Ma > wrote: > >> Hi Dave, >> I am willing to use React to re-create the front-end application. Since >> I plan to use separate front-end and back-end development methods, this >> means that the

Re: disable SSL compression?

2018-03-17 Thread Peter Eisentraut
On 3/11/18 13:28, Tom Lane wrote: >> My proposal is the attached patch that sets the default in libpq to off >> and adjusts the documentation a bit so it doesn't sound like we have >> missed the news altogether. > > Seems reasonable as far as it goes, but do we need to make corresponding > server-

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
Tomas Vondra writes: > I happened to be updating our machine running our buildfarm animals, and > I noticed something quite strange - the machine was unexpectedly running > out of disk space, which is rather suspicious as it's running just the > regression tests :-/ > After a bit of investigation

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
On 03/17/2018 03:32 PM, Tom Lane wrote: > Tomas Vondra writes: >> I happened to be updating our machine running our buildfarm animals, and >> I noticed something quite strange - the machine was unexpectedly running >> out of disk space, which is rather suspicious as it's running just the >> regr

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
I wrote: > Ouch. That test is in fact new as of 31 Dec, and what this seems to > prove is that plpgsql's handling of loop-variable overflow doesn't > work on fulmar. Some of the other icc-using critters haven't reported in since December, either :-( Looking at the code, we do this like so:

Re: Problems with Error Messages wrt Domains, Checks

2018-03-17 Thread David G. Johnston
On Sat, Mar 17, 2018 at 6:14 AM, john frazer wrote: > Today I realized a number of points where PostgreSQL (v10.3) is rather > lackluster and sparse in its error messages. > > ​You may find the following thread and its predecessors enlightening. https://www.postgresql.org/message-id/CAD3a31WR8X1

Re: Google Summer of Code: Potential Applicant

2018-03-17 Thread Christos Maris
I just submitted my draft proposal. Could you please take a look at it and give me any feedback on how to improve it? Thanks a lot in advance! On Wed, Mar 14, 2018 at 10:12 AM, Aleksander Alekseev < a.aleks...@postgrespro.ru> wrote: > Hello Stephen, > > > > Protobuf is fine too, but unfortunate

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Dean Rasheed
On 13 March 2018 at 08:39, John Naylor wrote: >> Also, this is common enough that in fact that distribution >> can be reasonably approximated by a normal distribution. > > For the archives, because it's typically seen 10 times in the sample, > per the rule of thumb mention upthread? > Actually, I

Re: ECPG oracle mode test program patch

2018-03-17 Thread Michael Meskes
> The attached patch corrects the cursor name. Fixed, thanks for the patch. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Meskes at (Debian|Postgresql) dot Org Jabber: michael at xmpp dot meskes dot org VfL Borussia! Força Barça! SF 49ers! Use Deb

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Dean Rasheed
On 17 March 2018 at 17:16, Dean Rasheed wrote: > Using the calculator above, you can see that the distribution is > fairly normal-like, but with a noticeable positive skew. The 2-stddev > interval is 0.6 to 9.4, and according to the calculator the > probability of the value being less than or equa

Re: [PATCH] pg_hba.conf : new auth option : clientcert=verify-full

2018-03-17 Thread Magnus Hagander
On Fri, Mar 9, 2018 at 2:11 PM, Julian Markwort < julian.markw...@uni-muenster.de> wrote: > Hello Magnus, > > > I think this makes a lot of sense, and can definitely be a useful > > option. > > I was hesistant to write a long and elaborate patch as I wasn't certain > if there was any interest for

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Andres Freund
On March 17, 2018 7:56:40 AM PDT, Tom Lane wrote: >I wrote: >> Ouch. That test is in fact new as of 31 Dec, and what this seems to >> prove is that plpgsql's handling of loop-variable overflow doesn't >> work on fulmar. > >Some of the other icc-using critters haven't reported in since >December

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
On 03/17/2018 06:27 PM, Andres Freund wrote: > > > On March 17, 2018 7:56:40 AM PDT, Tom Lane wrote: >> I wrote: >>> Ouch. That test is in fact new as of 31 Dec, and what this seems to >>> prove is that plpgsql's handling of loop-variable overflow doesn't >>> work on fulmar. >> >> Some of the

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Andres Freund
Hi, On 2018-03-17 18:55:11 +0100, Tomas Vondra wrote: > Not sure, but the backbranches seem to be working fine, and the commit > that triggers the issue is from December 31. Well, that added the test. Are you saying that if you execute similar code on an older branch it doesn't fail? > Anyway,

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
Tomas Vondra writes: > Not sure, but the backbranches seem to be working fine, and the commit > that triggers the issue is from December 31. Maybe the issue was there > but we were lucky not to trip on it before. Yeah, we were simply not testing that overflow-detection code before. Undoubtedly it

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Dean Rasheed
On 16 March 2018 at 15:26, Tomas Vondra wrote: > Actually, one question - when deciding whether to keep the item in the > MCV list, analyze_mcv_list only compares it's frequency with an average > of the rest. But as we're removing items from the MCV list, the average > frequency of the non-MCV ite

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
Andres Freund writes: > On the current branch just using the new overflow safe functions in > int.h should work. But unless we are OK leaving this broken in the back > branches, or want to backport the functionality, that's probably not > sufficient. Yeah ... I don't like either of the last two t

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Andres Freund
Hi, On 2018-03-17 14:20:26 -0400, Tom Lane wrote: > It might be worth studying the icc manual to see if it has an > equivalent of -fwrapv. Yes. A *quick* look through https://software.intel.com/en-us/node/522795 unfortunately didn't show anything. > Although we can and probably should fix this

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Tomas Vondra
On 03/17/2018 07:28 PM, Dean Rasheed wrote: > On 16 March 2018 at 15:26, Tomas Vondra wrote: >> Actually, one question - when deciding whether to keep the item in the >> MCV list, analyze_mcv_list only compares it's frequency with an average >> of the rest. But as we're removing items from the M

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
On 03/17/2018 07:20 PM, Tom Lane wrote: > Tomas Vondra writes: >> Not sure, but the backbranches seem to be working fine, and the commit >> that triggers the issue is from December 31. Maybe the issue was there >> but we were lucky not to trip on it before. > > Yeah, we were simply not testing

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Andres Freund
On March 17, 2018 11:32:36 AM PDT, Tom Lane wrote: >Andres Freund writes: >> On the current branch just using the new overflow safe functions in >> int.h should work. But unless we are OK leaving this broken in the >back >> branches, or want to backport the functionality, that's probably not >>

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
Andres Freund writes: > I don't think performance is a prime driver here, or shouldn't be at least. > Obviousness / grepability seem much more important. I'd vote for using my > version in master, and yours in the back branches. I can do that, of you > want. I dunno, I think the code as I ha

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Dean Rasheed
On 17 March 2018 at 18:40, Tomas Vondra wrote: > Currently, analyze_mcv_list only checks if the frequency of the current > item is significantly higher than the non-MCV selectivity. My question > is if it shouldn't also consider if removing the item from MCV would not > increase the non-MCV select

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Andres Freund
On March 17, 2018 12:25:57 PM PDT, Tom Lane wrote: >Andres Freund writes: >> I don't think performance is a prime driver here, or shouldn't be at >least. Obviousness / grepability seem much more important. I'd vote >for using my version in master, and yours in the back branches. I can >do tha

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
Andres Freund writes: > On March 17, 2018 12:25:57 PM PDT, Tom Lane wrote: >> If we're not putting a test into the back branches, then we darn well >> better be using the same code there as in HEAD, else we won't know that >> it actually solves the problem. > I was thinking of committing your ve

Fwd: Problems with Error Messages wrt Domains, Checks

2018-03-17 Thread john frazer
-- Forwarded message -- From: john frazer Date: Sat, Mar 17, 2018 at 6:28 PM Subject: Re: Problems with Error Messages wrt Domains, Checks To: "David G. Johnston" Thanks for your consideration, I'll try to be brief. > As such, it could be anywhere in my many, many kLOCs

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Tomas Vondra
On 03/17/2018 08:32 PM, Dean Rasheed wrote: > On 17 March 2018 at 18:40, Tomas Vondra wrote: >> Currently, analyze_mcv_list only checks if the frequency of the >> current item is significantly higher than the non-MCV selectivity. >> My question is if it shouldn't also consider if removing the item

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
On 03/17/2018 08:41 PM, Tom Lane wrote: > Andres Freund writes: >> On March 17, 2018 12:25:57 PM PDT, Tom Lane wrote: >>> If we're not putting a test into the back branches, then we darn >>> well better be using the same code there as in HEAD, else we >>> won't know that it actually solves the pr

Re: [PATCH] Verify Checksums during Basebackups

2018-03-17 Thread Michael Banck
Hi, On Fri, Mar 09, 2018 at 10:35:33PM +0100, Michael Banck wrote: > Possibly open questions: > > 1. I have not so far changed the replication protocol to make verifying > checksums optional. I can go about that next if the consensus is that we > need such an option (and cannot just check it ever

Re: pgbench randomness initialization

2018-03-17 Thread Chapman Flack
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, failed Spec compliant: not tested Documentation:tested, failed This is a simple patch, includes documentation, includes and pass

Re: Precision loss casting float to numeric

2018-03-17 Thread Chapman Flack
On 03/09/18 12:05, Emre Hasegeli wrote: > In this case, I cannot see any other option than adding those as > separate cast functions. Should we mark this entry as "returned with > feedback"? > > We can also consider turning the current float to numeric casts to > explicit as they are causing data

Re: [bug fix] Cascaded standby cannot start after a clean shutdown

2018-03-17 Thread Michael Paquier
On Fri, Mar 16, 2018 at 06:02:25AM +, Tsunakawa, Takayuki wrote: > Ouch, you're right. If memory allocation fails, the startup process > would emit a LOG message and continue to fetch new WAL records. Then, > I'm completely happy with your patch. Thanks for double-checking, Tsunakawa-san. --

Re: SSL passphrase prompt external command

2018-03-17 Thread Michael Paquier
On Fri, Mar 16, 2018 at 12:07:59PM -0400, Peter Eisentraut wrote: > On 3/15/18 12:13, Daniel Gustafsson wrote: >> * In src/tools/msvc/Mkvcbuild.pm >> >> # if building without OpenSSL >> if (!$solution->{options}->{openssl}) >> { >> + $postgres->RemoveFile('src/backend/libpq/be-se

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2018-03-17 Thread Chapman Flack
On 03/16/18 17:14, Daniel Gustafsson wrote: > The attached patch adds the test, and a neccessary extension to > check_pg_config > to allow for extracting values from pg_config.h as opposed to just returning > the number of regex matches. (needed for XLOG_BLCKSZ.) Thanks for the review. I notice t

[GSoC 2018] Proposal Draft

2018-03-17 Thread Kefan Yang
Hi everyone, I am Kefan Yang, a third-year Computing Science student from Simon Fraser University, Canada. I am very interested in the *sorting algorithm benchmarking and implementation* issue you mentioned on the idealist of Google Summer of Code 2018. I am currently working on my proposal, but

Re: [GSoC 2018] Proposal Draft

2018-03-17 Thread Peter Geoghegan
On Sat, Mar 17, 2018 at 5:34 PM, Kefan Yang wrote: > I am Kefan Yang, a third-year Computing Science student from Simon Fraser > University, Canada. I am very interested in the sorting algorithm > benchmarking and implementation issue you mentioned on the idealist of > Google Summer of Code 2018.

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tom Lane
Tomas Vondra writes: > I guess it might want to notify people running affected animals, because > otherwise they may stay stuck for a long time. Yeah, I sent something out to buildfarm-members already. regards, tom lane

Re: Precision loss casting float to numeric

2018-03-17 Thread Tom Lane
Chapman Flack writes: > I wonder whether even changing a formerly-implicit cast to explicit > would be too much of a behavior change for existing code that expects > the current behavior? We did exactly that during the 8.3 cycle, and got one heck of a lot of pushback about it, despite the existen

Re: [GSoC 2018] Proposal Draft

2018-03-17 Thread Kefan Yang
Thanks for your quick feedback! """ Industrial implementation of selected sorting algorithm: The industrial version is basically an optimization based on the benchmark implementation. I plan to use optimizations like checking if input array is already sorted or applying insertion sort directly for

Re: Precision loss casting float to numeric

2018-03-17 Thread Tom Lane
Emre Hasegeli writes: > We can also consider turning the current float to numeric casts to > explicit as they are causing data loss. I am not sure how much it > would impact backwards-compatibility. The counter argument is the > numeric to float casts being IMPLICIT. They are causing data loss

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2018-03-17 Thread Tom Lane
Chapman Flack writes: > Thanks for the review. I notice that cfbot has now flagged the patch as > failing, and when I look into it, it appears that cfbot is building with > your test patch, and without the xlog.c patch, and so the test naturally > fails. Does the cfbot require both patches to be a

Re: [GSoC 2018] Proposal Draft

2018-03-17 Thread Peter Geoghegan
On Sat, Mar 17, 2018 at 7:00 PM, Kefan Yang wrote: > What I am trying to say here is that similar optimizations can be applied to > novel algorithms or other implementations of quicksort. A novel algorithm is something to avoid here, because novel techniques tend to only work out for specific dat

Re: ON CONFLICT DO UPDATE for partitioned tables

2018-03-17 Thread Alvaro Herrera
Alvaro Herrera wrote: > I think what I should be doing is the same as the returning stuff: keep > a tupdesc around, and use a single slot, whose descriptor is changed > just before the projection. Yes, this works, though it's ugly. Not any uglier than what's already there, though, so I think it'

Recently-introduced segfault in initdb?

2018-03-17 Thread Isaac Morland
I am trying to develop a small proof-of-concept patch for a proposal I have, but recently I found that initdb started segfaulting after I did a git pull. I used git bisect and it identified the following commit as the first one with the problem: 1733460f0205fc6d6bbe4c14911049a918c6e073 is the firs

Re: Recently-introduced segfault in initdb?

2018-03-17 Thread Isaac Morland
OK, I must have done something wrong with the bisect the first time. Now I'm getting the following as the problem commit: fd1a421fe66173fb9b85d3fe150afde8e812cbe4 is the first bad commit commit fd1a421fe66173fb9b85d3fe150afde8e812cbe4 Author: Peter Eisentraut Date: Fri Mar 2 08:57:38 2018 -0

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-17 Thread Pavan Deolasee
On Mon, Mar 12, 2018 at 5:43 PM, Pavan Deolasee wrote: > > > On Sun, Mar 11, 2018 at 11:18 AM, Peter Geoghegan wrote: > >> >> >> As you know, there is an ON CONFLICT DO UPDATE + partitioning patch in >> the works from Alvaro. In your explanation about that approach that >> you cited, you wondere