Hmm...heap usage slowly growing over time...the picture doesnt look like the problems we were running into, and I dont think they should be related to mmap-ing your data (since that is not counted in heap mem usage).
Kyusik Chung On May 29, 2010, at 10:16 AM, James Golick wrote: > Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks > to be about the same: > > http://skitch.com/jamesgolick/df46f/munin-fetlife.com-cassandra0.fetlife.com-cassandra-memory > > There's only one cache turned on, and it's a row cache, but the sizes of the > rows are identical and it's been full since an hour after I rebooted the > nodes, so it's not that. > > On Thu, May 27, 2010 at 1:00 PM, Kyusik Chung <kyu...@discovereads.com> wrote: > > I tried setting the IO mode to standard, but it seemed to be a little slower > > and couldn't get the machine to come back online with adequate read > > performance, so I set it back. I'll have to write a solid cache warming > > script if I'm going to try that again. > > What cache are you talking about? Did you turn on row caching? > > When we turned on row caching, repeat hits to the same rows was fast, of > course, but we didnt (given our data access patterns) see significant > differences compared to mmap-ing the data. And once we hit the limit of our > row cache, out-of-cache hits were pretty costly (dont have hard numbers, but > I recall it being worse than having mmap page in/out). > > Is your client making random reads of more rows than will fit in RAM on your > box? We found that in that scenario, after cassandra has used up all of the > free memory on the box, using mmap was slightly worse than using standard > data access. > > We happened to be lucky that our real world data access is limited to a small > subset of rows in any given time period, so mmap works great for us. I guess > the best thing to do is to try to figure out how to make a cassandra node > only need to service requests for data that can fit into memory in a given > time period. More nodes, a lower replication factor, more memory, I guess... > > Im definitely waiting to hear how things change with 0.6.2. > > Kyusik Chung >
