Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Merlin Moncure
On Wed, Jun 10, 2015 at 4:37 PM, Patrick Krecker wrote: > On Wed, Jun 10, 2015 at 2:08 PM, Merlin Moncure wrote: >> On Wed, Jun 10, 2015 at 3:55 PM, Patrick Krecker >> wrote: >>> OK. Well, fortunately for us, we have a lot of possible solutions this >>> problem, and it sounds like actually gett

Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Patrick Krecker
On Wed, Jun 10, 2015 at 2:08 PM, Merlin Moncure wrote: > On Wed, Jun 10, 2015 at 3:55 PM, Patrick Krecker wrote: >> OK. Well, fortunately for us, we have a lot of possible solutions this >> problem, and it sounds like actually getting statistics for attributes >> ? 'reference' is not realistic. I

Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Merlin Moncure
On Wed, Jun 10, 2015 at 3:55 PM, Patrick Krecker wrote: > OK. Well, fortunately for us, we have a lot of possible solutions this > problem, and it sounds like actually getting statistics for attributes > ? 'reference' is not realistic. I just wanted to make sure it wasn't > some configuration erro

Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Patrick Krecker
OK. Well, fortunately for us, we have a lot of possible solutions this problem, and it sounds like actually getting statistics for attributes ? 'reference' is not realistic. I just wanted to make sure it wasn't some configuration error on our part. Can anyone explain where exactly the estimate for

Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Merlin Moncure
On Wed, Jun 10, 2015 at 2:40 PM, Josh Berkus wrote: > On 06/10/2015 11:32 AM, Merlin Moncure wrote: >> This is a fundamental issue with using 'database in a box' datatypes >> like hstore and jsonb. They are opaque to the statistics gathering >> system and so are unable to give reasonable estimate

Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Josh Berkus
On 06/10/2015 11:32 AM, Merlin Moncure wrote: > This is a fundamental issue with using 'database in a box' datatypes > like hstore and jsonb. They are opaque to the statistics gathering > system and so are unable to give reasonable estimates beyond broad > assumptions. Speaking generally, the wor

Re: [PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Merlin Moncure
On Wed, Jun 10, 2015 at 12:32 PM, Patrick Krecker wrote: > Hi everyone -- > > I had an issue the other day where a relatively simple query went from > taking about 1 minute to execute to taking 19 hours. It seems that the > planner chooses to use a materialize sometimes [1] and not other times > [

[PERFORM] Row estimates off by two orders of magnitude with hstore

2015-06-10 Thread Patrick Krecker
Hi everyone -- I had an issue the other day where a relatively simple query went from taking about 1 minute to execute to taking 19 hours. It seems that the planner chooses to use a materialize sometimes [1] and not other times [2]. I think the issue is that the row count estimate for the result o

Re: [PERFORM] Slow query - lots of temporary files.

2015-06-10 Thread Tomas Vondra
On 06/10/15 15:42, Johann Spies wrote: On 10 June 2015 at 15:02, Claudio Freire mailto:klaussfre...@gmail.com>> wrote: The joins are different on both versions, and the most likely culprit is the join against D. It's probably wrong, and the first query is building a cartesian prod

Re: [PERFORM] Slow query - lots of temporary files.

2015-06-10 Thread Johann Spies
On 10 June 2015 at 15:02, Claudio Freire wrote: > > The joins are different on both versions, and the most likely culprit > is the join against D. It's probably wrong, and the first query is > building a cartesian product. > > Without more information about the schema it's difficult to be sure th

Re: [PERFORM] Slow query - lots of temporary files.

2015-06-10 Thread Claudio Freire
On Wed, Jun 10, 2015 at 9:39 AM, Johann Spies wrote: > COPY > (SELECT A.ut, > B.go AS funding_org, > B.gn AS grant_no, > C.gt AS thanks, > D.au >FROM isi.funding_text C, > isi.rauthor D, > isi.africa_uts A >LEFT JOIN isi.funding_org

[PERFORM] Slow query - lots of temporary files.

2015-06-10 Thread Johann Spies
I have stopped this query after about 16 hours. At the same time I ran a 'explain analyze' on the same query to find out why it took so long. These two processes generated temporary files of 173GB in /var/lib/postgresql/9.4/main/base/pgsql_tmp. COPY (SELECT A.ut, B.go AS funding_org,