RC Hangout 006 Now Live - Riak and Datomic

2014-01-10 Thread Mark Phillips
Sorry for the late notice here, but today's Hangout is now live. https://www.youtube.com/watch?v=VD9UCfQohQE Hector Castro and Stuart Halloway are talking about running Riak and Datomic. Enjoy. Mark ___ riak-users mailing list riak-users@lists.basho.

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Sean McKibben
Excellent and informative explanation, thank you very much. We’re very happy that our adjustments have returned the cluster to its normal operating parameters. Also glad that Riak 2 will be handling this stuff programmatically, as prior to your spreadsheet and explanation it was pure voodoo for

Re: riak cs api integration

2014-01-10 Thread Kota Uenishi
Hi, Riak CS outputs Apache httpd style log to /var/log/riak-cs/access.log . Tailing it with logstash or whatever at every Riak CS server would make it much easier. The good point of tailing the log file is that you don't have to set up other servers. On Thu, Jan 9, 2014 at 11:23 PM, red sky wrot

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Martin May
Matthew, Thanks for the help and suggestions, we really appreciate it. We’re planning on giving Riak 2.0 a shot as soon as it’s released, and are looking forward to the new features. Best, Martin On Jan 10, 2014, at 7:51 AM, Matthew Von-Maszewski wrote: > Martin, > > Assuming your business

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Matthew Von-Maszewski
Martin, Assuming your business continues to grow, this problem will come back under 1.4 … but not for a while. We can push the cache_size as far down as 8Mbytes to make room for a little more file cache space if needed. The manual tunings I gave you and the subsequent block_size tuning I men

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Sean McKibben
We need all the results right away anyway, so we don't paginate, so once we get to 1.4.6+, being able to skip sorting ought to return some speed to us (and maybe we will leave +S at 6:6). With our small ring size and SSDs we see 3M keys returning in about 120 sec. While that case isn't rare, there

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Martin May
Hi Matthew, We applied this change to node 4, started it up, and it seems much happier (no crazy CPU). We’re going to keep an eye on it for a little while, and then apply this setting to all the other nodes as well. Is there anything we can do to prevent this scenario in the future, or should

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Matthew Von-Maszewski
Sean, Also you mentioned concern about +S 6:6. 2i queries in 1.4 added "sorting". Another heavy 2i user noticed that the sorting need more CPU for Erlang. They were happier after removing the +S. And finally, those 2i queries that return "millions of results" … how long do those queries tak

Re: Single node causing cluster to be extremely slow (leveldb)

2014-01-10 Thread Matthew Von-Maszewski
Sean, I did some math based upon the app.config and LOG files. I am guessing that you are starting to thrash your file cache. This theory should be easy to prove / disprove. On that one node, change the cache_size and max_open_files to: cache_size 68435456 max_open_files 425 If I am correct

RE: DecodeErrors in protobuf

2014-01-10 Thread Erik Søe Sørensen
It's usually one of these two: 1) the data has been corrupted since encoding. 2) what you're trying to decode is not protobuf data but something else. If the code logged the binary data it's trying to decode, that'd be a help. Given that the data is from network and not from disk, possible causes