Re: Riak 1.3.1 high swap usage

2015-02-19 Thread Daniel Iwan
I absolutely agree. That is why we've change the setting vm.swappiness to 1 so it swaps only when absolutely necessary. I think we underestimated how much swap may be needed, but I also don't understand why so much hungry on memory. Is there a particular activity, like 2i queries, AAE or levelDB c

Re: Riak 1.3.1 high swap usage

2015-02-19 Thread Sean Cribbs
Daniel, Our tuning guides specifically recommend disabling swap. If your machine is so memory-starved that it needs to use swap during normal Riak operation, having it on is only going to make things worse. http://docs.basho.com/riak/latest/ops/tuning/linux/#Storage-and-File-System-Tuning On Thu

Re: Riak 1.3.1 high swap usage

2015-02-19 Thread Daniel Iwan
We are using levelDB as backend without any tuning. Also we are aware that performance may suffer due to potentially storing some of the copies (n=3) twice on the server. We are not so much concerned about latencies caused by that. What is worrying though is almost unbounded growth of swap used, wh

Re: Riak 1.3.1 high swap usage

2015-02-19 Thread John Daily
Daniel, you may be aware of this, but a 3-node Riak cluster is not recommended and may be playing a minor role in your resource problems. Every request will hit every server (except for some requests that are being made twice against a single server, making I/O that much worse), and depending on yo

Re: Riak 1.3.1 high swap usage

2015-02-19 Thread Christopher Meiklejohn
> On Feb 19, 2015, at 9:05 AM, Daniel Iwan wrote: > > Hi > On 3 node cluster Ubuntu 12.04, nodes 8GB RAM all nodes show 6GB taken > beam.smp, 2GB by our process. > beam started swapping and currently is using 23GB of swap space. > vm.swappiness is set to 1 > We are using ring 128. /var/lib/ria

Riak 1.3.1 high swap usage

2015-02-19 Thread Daniel Iwan
Hi On 3 node cluster Ubuntu 12.04, nodes 8GB RAM all nodes show 6GB taken beam.smp, 2GB by our process. beam started swapping and currently is using 23GB of swap space. vm.swappiness is set to 1 We are using ring 128. /var/lib/riak is 37GB in size 11GB of which is used by anti-entropy Is there a