> Does this mean we should not enable row caches until we are absolutely sure > about what's hot (I think there is a reason why row caches are disabled by > default) ? Yes and Yes. Row cache takes memory and CPU, unless you know you are getting a benefit from it leave it off. The key cache and os disk cache will help. If you find latency is an issue then start poking around.
Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 5/12/2012, at 4:23 AM, Yiming Sun <yiming....@gmail.com> wrote: > Hi Aaron, > > Thank you,and your explanation makes sense. At the time, I thought having > 1GB of row cache on each node was plenty enough, because there was an > aggregated 6GB cache, but you are right, with each row in 10's of MBs, some > of the nodes can go into a constant load and evict cycle and would have > negative effects on the performance. I will try as you suggested to 1.) > reduce the requested entry set, and 2.) increase the row cache size and see > if they get better hits, and also do 3) by reversing the requested entry list > in alternate runs. > > Our data space has close to 3 million rows, but we haven't gotten enough > usage statistics to know what rows are hot. Does this mean we should not > enable row caches until we are absolutely sure about what's hot (I think > there is a reason why row caches are disabled by default) ? It also seems > from my test that OS page cache works much better, but it could be that OS > page cache can utilize all the available memory so it is essentially larger > -- I guess I will find out by doing 2.) above. > > best, > > -- Y. > > > > On Tue, Dec 4, 2012 at 4:47 AM, aaron morton <aa...@thelastpickle.com> wrote: > > Row Cache : size 1072651974 (bytes), capacity 1073741824 (bytes), 0 > > hits, 2576 requests, NaN recent hit rate, 0 save period in seconds > > So the cache is pretty much full, there is only 1 MB free. > > There were 2,576 read requests that tried to get a row from the cache. Zero > of those had a hit. If you have 6 nodes and RF 2, each node has one third of > the data in the cluster (from the effective ownership info). So depending on > the read workload the number of read requests on each node may be different. > > What I think is happening is reads are populating the row cache, then > subsequent reads are evicting items from the row cache before you get back to > reading the original rows. So if you read rows 1 to 5, they are put in the > cache, when you read rows 6 to 10 they are put in and evict rows 1 to 5. Then > you read rows 1 to 5 again they are not in the cache. > > Try testing with a lower number of hot rows, and/or a bigger row cache. > > But to be honest, with rows in the 10's of MB you will probably only get good > cache performance with a small set of hot rows. > > Hope that helps. > > > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 1/12/2012, at 5:11 AM, Yiming Sun <yiming....@gmail.com> wrote: > > > Does anyone have any comments/suggestions for me regarding this? Thanks > > > > > > I am trying to understand some strange behavior of cassandra row cache. We > > have a 6-node Cassandra cluster in a single data center on 2 racks, and the > > neighboring nodes on the ring are from alternative racks. Each node has > > 1GB row cache, with key cache disabled. The cluster uses > > PropertyFileSnitch, and the ColumnFamily I fetch from uses > > NetworkTopologyStrategy, with replication factor of 2. My client code uses > > Hector to fetch a fixed set of rows from cassandra > > > > What I don't quite understand is even after I ran the client code several > > times, there are always some nodes with 0 row cache hits, despite that the > > row cache from all nodes are filled and all nodes receive requests. > > > > Which nodes have 0 hits seem to be strongly related to the following: > > > > - the set of row keys to fetch > > - the order of the set of row keys to fetch > > - the list of hosts passed to Hector's CassandraHostConfigurator > > - the order of the list of hosts passed to Hector > > > > Can someone shed some lights on how exactly the row cache works and > > hopefully also explain the behavior I have been seeing? I thought if the > > fixed set of the rows keys are the only thing I am fetching (each row > > should be on the order of 10's of MBs, no more than 100MB), and each node > > gets requests, and its row cache is filled, there's gotta be some hits. > > Apparent this is not the case. Thanks. > > > > cluster information: > > > > Address DC Rack Status State Load > > Effective-Ownership Token > > > > 141784319550391026443072753096570088105 > > x.x.x.1 DC1 r1 Up Normal 587.46 GB 33.33% > > 0 > > x.x.x.2 DC1 r2 Up Normal 591.21 GB 33.33% > > 28356863910078205288614550619314017621 > > x.x.x.3 DC1 r1 Up Normal 594.97 GB 33.33% > > 56713727820156410577229101238628035242 > > x.x.x.4 DC1 r2 Up Normal 587.15 GB 33.33% > > 85070591730234615865843651857942052863 > > x.x.x.5 DC1 r1 Up Normal 590.26 GB 33.33% > > 113427455640312821154458202477256070484 > > x.x.x.6 DC1 r2 Up Normal 583.21 GB 33.33% > > 141784319550391026443072753096570088105 > > > > > > [user@node]$ ./checkinfo.sh > > *************** x.x.x.4 > > Token : 85070591730234615865843651857942052863 > > Gossip active : true > > Thrift active : true > > Load : 587.15 GB > > Generation No : 1354074048 > > Uptime (seconds) : 36957 > > Heap Memory (MB) : 2027.29 / 3948.00 > > Data Center : DC1 > > Rack : r2 > > Exceptions : 0 > > > > Key Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, > > NaN recent hit rate, 14400 save period in seconds > > Row Cache : size 1072651974 (bytes), capacity 1073741824 (bytes), 0 > > hits, 2576 requests, NaN recent hit rate, 0 save period in seconds > > > > *************** x.x.x.6 > > Token : 141784319550391026443072753096570088105 > > Gossip active : true > > Thrift active : true > > Load : 583.21 GB > > Generation No : 1354074461 > > Uptime (seconds) : 36535 > > Heap Memory (MB) : 828.71 / 3948.00 > > Data Center : DC1 > > Rack : r2 > > Exceptions : 0 > > > > Key Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, > > NaN recent hit rate, 14400 save period in seconds > > Row Cache : size 1072602906 (bytes), capacity 1073741824 (bytes), 0 > > hits, 3194 requests, NaN recent hit rate, 0 save period in seconds > > > > > >
