Re: [GENERAL] free RAM not being used for page cache

2014-09-04 Thread Kevin Goess
This is a super-interesting topic, thanks for all the info. On Thu, Sep 4, 2014 at 7:44 AM, Shaun Thomas wrote: > > Check /proc/meminfo for a better breakdown of how the memory is being > used. This should work: > > grep -A1 Active /proc/meminfo > > I suspect your inactive file cache is larger th

Re: [GENERAL] free RAM not being used for page cache

2014-09-03 Thread Kevin Goess
On Tue, Aug 5, 2014 at 8:27 AM, Shaun Thomas wrote: > On 07/30/2014 12:51 PM, Kevin Goess wrote: > > A couple months ago we upgraded the RAM on our database servers from >> 48GB to 64GB. Immediately afterwards the new RAM was being used for >> page cache, which is what we

Re: [GENERAL] free RAM not being used for page cache

2014-07-30 Thread Kevin Goess
Good suggestion, but nope, that ain't it: $ cat /proc/sys/vm/zone_reclaim_mode 0 On Wed, Jul 30, 2014 at 11:49 AM, Merlin Moncure wrote: > On Wed, Jul 30, 2014 at 12:51 PM, Kevin Goess wrote: > > A couple months ago we upgraded the RAM on our database servers from >

[GENERAL] free RAM not being used for page cache

2014-07-30 Thread Kevin Goess
A couple months ago we upgraded the RAM on our database servers from 48GB to 64GB. Immediately afterwards the new RAM was being used for page cache, which is what we want, but that seems to have dropped off over time, and there's currently actually like 12GB of totally unused RAM. http://s76.phot

Re: [GENERAL] Merge a sharded master into a single read-only slave

2014-06-02 Thread Kevin Goess
> So my conclusion is that for now, the best way to scale read-only queries for a sharded master is to > implement map-reduce at the application level. That's the conclusion I would expect. It's the price you pay for sharding, it's part of the deal. But it's also the benefit you get from sharding

Re: [GENERAL] are analyze statistics synced with replication?

2014-05-15 Thread Kevin Goess
On Thu, May 15, 2014 at 6:39 AM, Dorian Hoxha wrote: > If you don't do read queries on the slave than it will not have hot > data/pages/rows/tables/indexes in ram like the primary ? > Yeah, that was the first thing we noticed, the cacti graph shows it took two hours for the page cache to fill up

[GENERAL] are analyze statistics synced with replication?

2014-05-14 Thread Kevin Goess
We have a master/slave setup with replication. Today we failed over to the slave and saw disk I/O go through the roof. Are the pg_statistic statistics synced along with streaming replication? Are you expected to have to do a vacuum analyze after failing over? That's what we're trying now to see

[GENERAL] upgrading from debian 6 to 7--do in place or wipe-and-install?

2014-03-28 Thread Kevin Goess
We're looking at upgrading our database hosts running postgres 9.2 from debian 6/squeeze to debian 7/wheezy. It seems to me that using apt to upgrade in-place would be less work than wiping the boxes and reinstalling debian 7 from scratch, but the latter way would be cleaner. Does anybody have an

Re: [GENERAL] pgpgout/s without swapping--what does it mean?

2014-03-18 Thread Kevin Goess
014, Kevin Goess wrote: > >> We had a big increase in load, iowait, and disk i/o on a dedicated >> database host the other night. >> >> Looking at the sar logs, the problem shows itself in a big increase in >> pgpgout/s, which I believe is postgres paging out parts

[GENERAL] pgpgout/s without swapping--what does it mean?

2014-03-17 Thread Kevin Goess
We had a big increase in load, iowait, and disk i/o on a dedicated database host the other night. Looking at the sar logs, the problem shows itself in a big increase in pgpgout/s, which I believe is postgres paging out parts of itself to disk? 02:15:01 AM pgpgin/s pgpgout/s fault/s majflt/s

Re: [GENERAL] simple query with radically different plan after 9.0 -> 9.2 upgrade

2013-11-17 Thread Kevin Goess
On Tue, Nov 12, 2013 at 2:47 PM, Tom Lane wrote: > > That's right, we store 90 days and roll up data older than that into a > > different table. > > Ah-hah. The default statistics target is 100, so indeed ANALYZE is going > to be able to fit every date entry in the table into the > most-common-v

Re: [GENERAL] simple query with radically different plan after 9.0 -> 9.2 upgrade

2013-11-12 Thread Kevin Goess
Thanks for the reply! Your analysis matches everything I see here, so what you say is probably the case. As to why it changed for us with the 9.0 => 9.2 upgrade, I also don't know--the change was pretty dramatic though. Since we've compensated for it, and since you say the current behavior is act

[GENERAL] simple query with radically different plan after 9.0 -> 9.2 upgrade

2013-09-20 Thread Kevin Goess
Given this table articles_1=> \d hits_user_daily_count; Table "public.hits_user_daily_count" Column | Type | Modifiers ---+-+--- userid| integer | not null date | date| not null num

Re: [GENERAL] Why are stored procedures looked on so negatively?

2013-07-23 Thread Kevin Goess
On Tue, Jul 23, 2013 at 5:40 PM, Adrian Klaver wrote: > On 07/23/2013 05:29 PM, Some Developer wrote: > >> I'm in the middle of building a database and was going to make extensive >> use of stored procedures and trigger functions because it makes more >> sense for the actions to happen at the data

[GENERAL] process deadlocking on its own transactionid?

2013-07-23 Thread Kevin Goess
We're seeing a problem with some of our processes hanging on locks. The select below makes it look like it's *waiting* for a ShareLock on transactionid, but it *has* an ExclusiveLock on the same value in virtualxid. That makes it look like the process has deadlocked on its own transactionid. Or

Re: [GENERAL] WHERE IN (subselect) versus WHERE IN (1,2,3,)

2012-03-20 Thread Kevin Goess
Thanks for looking into it, Tom. We're using 9.0.4, so that might indeed be the problem. What additional data (if any) would you like to see? If you want to look into it further, I can give you schema, though I hesitate to spam the whole list. I could also mock up some tables and see what's the

Re: [GENERAL] WHERE IN (subselect) versus WHERE IN (1,2,3,)

2012-03-19 Thread Kevin Goess
On Mon, Mar 19, 2012 at 9:24 AM, Albe Laurenz wrote: > That means that your statistics are not accurate. > > As a first measure, you should ANALYZE the tables involved and see if > the problem persists. If yes, post the new plans. > Aha, thanks, that explains why my test table with one row was s

[GENERAL] WHERE IN (subselect) versus WHERE IN (1,2,3,)

2012-03-15 Thread Kevin Goess
My apologies, I'm sure this question has been asked before but I couldn't find anything on the list that meant anything to me. We have a table "contexts" with 1.6 million rows, and a table "articles" with 1.4 million rows, where an "article" is a particular kind of "context". We want to select fr