Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread David Rowley
On 6 April 2017 at 10:17, Simon Riggs wrote: > On 5 April 2017 at 10:47, David Rowley wrote: > >> I've attached an updated patch to address Tomas' concerns and yours too. > > Commited, with some doc changes and additions based upon my explorations. Great. Thanks for committing! -- David Rowl

Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread Simon Riggs
On 5 April 2017 at 10:47, David Rowley wrote: > I've attached an updated patch to address Tomas' concerns and yours too. Commited, with some doc changes and additions based upon my explorations. For the record, I measured the time to calc extended statistics as +800ms on 2 million row sample.

Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread David Rowley
On 6 April 2017 at 07:19, Tels wrote: > I know I'm a bit late, but isn't the syntax backwards? > > "CREATE STATISTICS s1 WITH (dependencies) ON (col_a, col_b) FROM table;" > > These do it the other way round: > > CREATE INDEX idx ON table (col_a); > > AND: > >CREATE TABLE t ( > id INT RE

Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread Tels
Moin, On Wed, April 5, 2017 2:52 pm, Simon Riggs wrote: > On 5 April 2017 at 10:47, David Rowley > wrote: > >>> I have some other comments. > > Me too. > > > CREATE STATISTICS should take ShareUpdateExclusiveLock like ANALYZE. > > This change is in line with other changes in this and earlier > re

Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread Simon Riggs
On 5 April 2017 at 10:47, David Rowley wrote: >> I have some other comments. Me too. CREATE STATISTICS should take ShareUpdateExclusiveLock like ANALYZE. This change is in line with other changes in this and earlier releases. Comments and docs included. Patch ready to be applied directly bar

Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread David Rowley
On 5 April 2017 at 14:53, Kyotaro HORIGUCHI wrote: > At Tue, 4 Apr 2017 20:19:39 +0200, Tomas Vondra > wrote in > <56f40b20-c464-fad2-ff39-06b668fac...@2ndquadrant.com> >> Two minor comments: >> >> 1) DEPENDENCY_MIN_GROUP_SIZE >> >> I'm not sure we still need the min_group_size, when evaluating

Re: [HACKERS] multivariate statistics (v25)

2017-04-05 Thread Tomas Vondra
On 04/05/2017 08:41 AM, Sven R. Kunze wrote: Thanks Tomas and David for hacking on this patch. On 04.04.2017 20:19, Tomas Vondra wrote: I'm not sure we still need the min_group_size, when evaluating dependencies. It was meant to deal with 'noisy' data, but I think it after switching to the '

Re: [HACKERS] multivariate statistics (v25)

2017-04-04 Thread Sven R. Kunze
Thanks Tomas and David for hacking on this patch. On 04.04.2017 20:19, Tomas Vondra wrote: I'm not sure we still need the min_group_size, when evaluating dependencies. It was meant to deal with 'noisy' data, but I think it after switching to the 'degree' it might actually be a bad idea. Consi

Re: [HACKERS] multivariate statistics (v25)

2017-04-04 Thread Kyotaro HORIGUCHI
At Tue, 4 Apr 2017 20:19:39 +0200, Tomas Vondra wrote in <56f40b20-c464-fad2-ff39-06b668fac...@2ndquadrant.com> > On 04/04/2017 09:55 AM, David Rowley wrote: > > On 1 April 2017 at 04:25, David Rowley > > wrote: > >> I've attached an updated patch. > > > > I've made another pass at this and ende

Re: [HACKERS] multivariate statistics (v25)

2017-04-04 Thread Tomas Vondra
On 04/04/2017 09:55 AM, David Rowley wrote: On 1 April 2017 at 04:25, David Rowley wrote: I've attached an updated patch. I've made another pass at this and ended up removing the tryextstats variable. We now only try to use extended statistics when clauselist_selectivity() is given a valid Re

Re: [HACKERS] multivariate statistics (v25)

2017-04-04 Thread David Rowley
On 1 April 2017 at 04:25, David Rowley wrote: > I've attached an updated patch. I've made another pass at this and ended up removing the tryextstats variable. We now only try to use extended statistics when clauselist_selectivity() is given a valid RelOptInfo with rtekind == RTE_RELATION, and of

Re: [HACKERS] multivariate statistics (v25)

2017-03-31 Thread David Rowley
On 31 March 2017 at 21:18, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > > When adding these two parameters I had 2nd thoughts that the > "tryextstats" > > was required at all. We could just have this controlled by if the rel is > a > > base rel of kind RTE_RELATION. I ended up hav

Re: [HACKERS] multivariate statistics (v25)

2017-03-31 Thread David Rowley
On 31 March 2017 at 21:18, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Hello, > > At Fri, 31 Mar 2017 03:03:06 +1300, David Rowley < > david.row...@2ndquadrant.com> wrote in T5JLce5ynCi1vvezXxX=w...@mail.gmail.com> > > FWIW, I tries this. This cleanly applied on it but make ends

Re: [HACKERS] multivariate statistics (v25)

2017-03-31 Thread Kyotaro HORIGUCHI
Hello, At Fri, 31 Mar 2017 03:03:06 +1300, David Rowley wrote in > On 25 March 2017 at 07:35, Alvaro Herrera wrote: > > > As I said in another thread, I pushed parts 0002,0003,0004. Tomas said > > he would try to rebase patches 0001,0005,0006 on top of what was > > committed. My intention i

Re: [HACKERS] multivariate statistics (v25)

2017-03-30 Thread David Rowley
On 25 March 2017 at 07:35, Alvaro Herrera wrote: > As I said in another thread, I pushed parts 0002,0003,0004. Tomas said > he would try to rebase patches 0001,0005,0006 on top of what was > committed. My intention is to give that one a look as soon as it is > available. So we will have n-dist

Re: [HACKERS] multivariate statistics (v25)

2017-03-24 Thread Alvaro Herrera
Alvaro Herrera wrote: > Here's a rebased series on top of today's a3eac988c267. I call this > v28. > > I put David's pg_dump and COMMENT patches as second in line, just after > the initial infrastructure patch. I suppose those three have to be > committed together, while the others (which add su

Re: [HACKERS] multivariate statistics (v25)

2017-03-16 Thread David Rowley
On 17 March 2017 at 11:20, Alvaro Herrera wrote: > (I think I lost some regression test files. I couldn't make up my mind > about putting each statistic type's tests in a separate file, or all > together in stats_ext.sql.) +1 for stats_ext.sql. I wanted to add some tests for pg_statisticsextdef(

Re: [HACKERS] multivariate statistics (v25)

2017-03-16 Thread David Rowley
On 16 March 2017 at 09:45, Alvaro Herrera wrote: > Here's another version of 0002 after cleaning up almost everything from > David's review. I also added tests for ALTER STATISTICS in > sql/alter_generic.sql which made me realize there were three crasher bug > in here; fixed all those. It also

Re: [HACKERS] multivariate statistics (v25)

2017-03-15 Thread Alvaro Herrera
David Rowley wrote: > + k = -1; > + while ((k = bms_next_member(attnums, k)) >= 0) > + { > + bool attr_found = false; > + for (i = 0; i < info->stakeys->dim1; i++) > + { > + if (info->stakeys->values[i] == k) > + { > + attr_found = true; > + break; > + } > + } > + > + /* found attribute not covere

Re: [HACKERS] multivariate statistics (v25)

2017-03-14 Thread David Rowley
On 15 March 2017 at 12:18, David Fetter wrote: > > Is the plan to convert completely from "multivariate" to "extended?" > I ask because I found a "multivariate" in there. > I get the idea that Tomas would like to keep the multivariate when it's actually referencing multivariate stats. The idea o

Re: [HACKERS] multivariate statistics (v25)

2017-03-14 Thread David Fetter
On Tue, Mar 14, 2017 at 07:10:49PM -0300, Alvaro Herrera wrote: > Alvaro Herrera wrote: > > I tried patch 0002 today and again there are conflicts, so I rebased and > > fixed the merge problems. > > ... and attached the patch. Is the plan to convert completely from "multivariate" to "extended?" I

Re: [HACKERS] multivariate statistics (v25)

2017-03-14 Thread Alvaro Herrera
Alvaro Herrera wrote: > I tried patch 0002 today and again there are conflicts, so I rebased and > fixed the merge problems. ... and attached the patch. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git

Re: [HACKERS] multivariate statistics (v25)

2017-03-14 Thread Alvaro Herrera
I tried patch 0002 today and again there are conflicts, so I rebased and fixed the merge problems. I also changed a number of minor things, all AFAICS cosmetic in nature: * moved src/backend/statistics/common.h to src/include/statistics/common.h, as previously commented. I also took out post

Re: [HACKERS] multivariate statistics (v25)

2017-03-14 Thread David Rowley
On 13 March 2017 at 23:00, David Rowley wrote: > > 0003: > > No more time today. Will try and get to those soon. > 0003: I've now read this patch. My main aim here was to learn what it does and how it works. I need to spend much longer understanding how your calculating the functional dependenci

Re: [HACKERS] multivariate statistics (v25)

2017-03-13 Thread David Rowley
On 3 March 2017 at 03:53, Tomas Vondra wrote: > This time with the attachments It's been a long while since I looked at this patch, but I'm now taking another look. I've made a list of stuff I've found from making my first pass on 0001 and 0002. Some of the stuff may seem a little pedanti

Re: [HACKERS] multivariate statistics (v25)

2017-03-02 Thread Tomas Vondra
On 03/02/2017 07:42 AM, Kyotaro HORIGUCHI wrote: Hello, At Thu, 2 Mar 2017 04:05:34 +0100, Tomas Vondra wrote in OK, attached is v24 of the patch series, addressing most of the reported issues and comments (at least I believe so). The main changes are: Unfortunately, 0002 conflicts with t