Ruslan,
Why did you suggest changing the disk_access_mode ?
Gurpreet,
I would leave the disk_access_mode with the default until you have a
reason to change it.
> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
is swap disabled ?
> Gradually,
> > the system cpu becomes high almost 70%, and the client starts getting
> > continuous timeouts
70% of one core or 70% of all cores ?
Check the server logs, is there GC activity ?
check nodetool cfstats to see the read latency for the cf.
Take a look at vmstat to see if you are swapping, and look at iostats to see if
io is the problem
http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/06/2012, at 9:00 PM, Gurpreet Singh wrote:
> Thanks Ruslan.
> I will try the mmap_index_only.
> Is there any guideline as to when to leave it to auto and when to use
> mmap_index_only?
>
> /G
>
> On Fri, Jun 8, 2012 at 1:21 AM, ruslan usifov <ruslan.usi...@gmail.com> wrote:
> disk_access_mode: mmap??
>
> set to disk_access_mode: mmap_index_only in cassandra yaml
>
> 2012/6/8 Gurpreet Singh <gurpreet.si...@gmail.com>:
> > Hi,
> > I am testing cassandra 1.1 on a 1 node cluster.
> > 8 core, 16 gb ram, 6 data disks raid0, no swap configured
> >
> > cassandra 1.1.1
> > heap size: 8 gigs
> > key cache size in mb: 800 (used only 200mb till now)
> > memtable_total_space_in_mb : 2048
> >
> > I am running a read workload.. about 30 reads/second. no writes at all.
> > The system runs fine for roughly 12 hours.
> >
> > jconsole shows that my heap size has hardly touched 4 gigs.
> > top shows -
> > SHR increasing slowly from 100 mb to 6.6 gigs in these 12 hrs
> > RES increases slowly from 6 gigs all the way to 15 gigs
> > buffers are at a healthy 25 mb at some point and that goes down to 2 mb in
> > these 12 hrs
> > VIRT stays at 85 gigs
> >
> > I understand that SHR goes up because of mmap, RES goes up because it is
> > showing SHR value as well.
> >
> > After around 10-12 hrs, the cpu utilization of the system starts increasing,
> > and i notice that kswapd0 process starts becoming more active. Gradually,
> > the system cpu becomes high almost 70%, and the client starts getting
> > continuous timeouts. The fact that the buffers went down from 20 mb to 2 mb
> > suggests that kswapd0 is probably swapping out the pagecache.
> >
> > Is there a way out of this to avoid the kswapd0 starting to do things even
> > when there is no swap configured?
> > This is very easily reproducible for me, and would like a way out of this
> > situation. Do i need to adjust vm memory management stuff like pagecache,
> > vfs_cache_pressure.. things like that?
> >
> > just some extra information, jna is installed, mlockall is successful. there
> > is no compaction running.
> > would appreciate any help on this.
> > Thanks
> > Gurpreet
> >
> >
>
