To add to this, if the 24G is the JVM limit, Cassandra will actually be using even more than this for bloom filters etc. that are managed off-heap. So the amount of page cache left is almost certainly inadequate.
On 14 March 2014 13:01, Andras Szerdahelyi < andras.szerdahe...@ignitionone.com> wrote: > Is row cache enabled on this CF? Try disabling it. Seems like you might > have a very wide row there. > > Can you grep for GCInspector in your Cassandra log? 24G might be a bit > too much for the Cassandra JVM, bogging down GC, and not leaving much to > page cache ( 32G -24G - Tomcat ). I don't quite understand your reasoning > here: > > (I know that there is a lot of heap but I also have write heavy tasks > and I want them to get into mem fast) . > > So, I would try with default cassandra-env.sh JVM params too > > From: Batranut Bogdan <batra...@yahoo.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>, > Batranut Bogdan <batra...@yahoo.com> > Date: Friday 14 March 2014 13:50 > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Cassandra slow on some reads > > Hello all, > > Here is the environment: > > I have a 6 node Cassandra cluster. On each node I have: > - 32 G RAM > - 24 G RAM for cassa > - ~150 - 200 MB/s disk speed > - tomcat 6 with axis2 webservice that uses the datastax java driver to make > asynch reads / writes > - replication factor for the keyspace is 3 > > (I know that there is a lot of heap but I also have write heavy tasks and > I want them to get into mem fast) . > > All nodes in the same data center > The clients that read / write are in the same datacenter so network is > Gigabit. > > The table structure is like this: PK(key String , timestam int, column1 > string, col2 string) , list1 , list 2, list 3 . > There are about 300 milions individual keys. > There are about 100 timestamps for each key now, so the rows will get > wider as time passes. > > I am using datastax java driver to query the cluster. > > I have ~450 queries that are like this: SELECT * FROM table where key = > 'some string' and ts = some value; some value is close to present time. > > The problem: > > About 10 - 20 % of these queries take more than 5 seconds to execute, in > fact, the majority of those take around 10 seconds. > When investigating I saw that if I have a slow response and I redo the > query it will finish in 8 - 10 MILIseconds like the rest of the queries > that I have. > I could not see using JConsole any spikes in CPU / memory when executing > the queries. The rise in resource consumtion is very small on all nodes on > the cluster. I expect such delays to be generated by a BIG increase in > resource consumption. > > Any comments will be appreciated. > > Thank you. > > > > >
