How to make Riak work faster (writing)
Hi, I'm new here and with Riak. If I do something wrong, please, let me know. I've made a Riak cluster with two identical machines: Intel core i3 2.3GHz 4GB RAM 1TB HD. They are connected by a gigabit ethernet network. Everything is working fine. I'm using a Bitcask backend. I've made a PHP script to do a performance test and find out how fast Riak can be with these settings. What my script is doing is: to store 1 objects with a 10K long data (string of 10240 x 'A'), and calculating how many objects it stores per second. Right now, using Bitcask backend, it can store roughly 68 objects per second. It seems to be a small number to me, but I don't know too much about Riak. I've tested the same script on a Couchbase cluster, with the same settings, and it could store roughly 1000 objects per second. Obviously, on Couchbase test, the data is not sent to non-volatile media immediately. Data is kept in memory to acknowledge the reception as fast as possible, and is sent to non-volatile media in background. I want Riak to behave the same way to increase the "writing speed", but I don't know how to do it or if it is possible. May be I'm trying to do something completely out of the purpose of Riak. I've tested Riak with Memory backend too, but it achieved only 72 objects per second. I expected it to work faster with Memory backend, cause there is no disk activity involved on it, but the final result is not that high. Again, I don't know if I'm trying to do something inappropriate. I think I'm missing something. Is there any way to do it? If I could not make myself clear, please, let me know. Thank you. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to make Riak work faster (writing)
I'm hitting just one of them. At the beginning I tried to use the default settings (n_val = 3), then I started to create the bucket with n_val = 2. I've tested a lot of combinations to w, but I could not see any substantial improvement. If you have any suggestion, please, let me know. I can do any test. On Fri, Nov 2, 2012 at 9:04 AM, Sebastian Cohnen wrote: > What level of concurrency are you using in your test setup? Are you > hitting both servers with your test? What is your n_val and w? > > > On 02.11.2012, at 03:42, Uruka Dark wrote: > > > Hi, > > > > I'm new here and with Riak. If I do something wrong, please, let me know. > > > > I've made a Riak cluster with two identical machines: Intel core i3 > 2.3GHz 4GB RAM 1TB HD. They are connected by a gigabit ethernet network. > Everything is working fine. I'm using a Bitcask backend. > > > > I've made a PHP script to do a performance test and find out how fast > Riak can be with these settings. What my script is doing is: to store 1 > objects with a 10K long data (string of 10240 x 'A'), and calculating how > many objects it stores per second. > > > > Right now, using Bitcask backend, it can store roughly 68 objects per > second. It seems to be a small number to me, but I don't know too much > about Riak. I've tested the same script on a Couchbase cluster, with the > same settings, and it could store roughly 1000 objects per second. > Obviously, on Couchbase test, the data is not sent to non-volatile media > immediately. Data is kept in memory to acknowledge the reception as fast as > possible, and is sent to non-volatile media in background. I want Riak to > behave the same way to increase the "writing speed", but I don't know how > to do it or if it is possible. May be I'm trying to do something completely > out of the purpose of Riak. > > > > I've tested Riak with Memory backend too, but it achieved only 72 > objects per second. I expected it to work faster with Memory backend, cause > there is no disk activity involved on it, but the final result is not that > high. > > > > Again, I don't know if I'm trying to do something inappropriate. I think > I'm missing something. > > Is there any way to do it? > > > > If I could not make myself clear, please, let me know. > > Thank you. > > > > > > > > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to make Riak work faster (writing)
I understand. To eliminate any problems related to Bitcask, I changed to Memory backend and now I can store roughly 80 objs/sec. This speed can be achieved hitting just one of them. I tried to hit both of them at same time, and the speed drops to roughly 68 objs/sec (each). Do you have any suggestion about it? Thank you. On Fri, Nov 2, 2012 at 9:20 AM, Sebastian Cohnen wrote: > You should hit both servers and not just with a concurrency level of 1. > > There are many more factors to consider, but these are highly dependent on > your actual problem (not just a simple benchmark). Just to name a few: > bitcask settings ( > http://docs.basho.com/riak/latest/tutorials/choosing-a-backend/Bitcask/#Tuning-Bitcask), > w-quorum, HTTP vs ProtoBuf, ring_creation_size, ... > > On 02.11.2012, at 13:15, Uruka Dark wrote: > > I'm hitting just one of them. > At the beginning I tried to use the default settings (n_val = 3), then I > started to create the bucket with n_val = 2. I've tested a lot of > combinations to w, but I could not see any substantial improvement. > If you have any suggestion, please, let me know. I can do any test. > > On Fri, Nov 2, 2012 at 9:04 AM, Sebastian Cohnen < > sebastian.coh...@gmail.com> wrote: > >> What level of concurrency are you using in your test setup? Are you >> hitting both servers with your test? What is your n_val and w? >> >> >> On 02.11.2012, at 03:42, Uruka Dark wrote: >> >> > Hi, >> > >> > I'm new here and with Riak. If I do something wrong, please, let me >> know. >> > >> > I've made a Riak cluster with two identical machines: Intel core i3 >> 2.3GHz 4GB RAM 1TB HD. They are connected by a gigabit ethernet network. >> Everything is working fine. I'm using a Bitcask backend. >> > >> > I've made a PHP script to do a performance test and find out how fast >> Riak can be with these settings. What my script is doing is: to store 1 >> objects with a 10K long data (string of 10240 x 'A'), and calculating how >> many objects it stores per second. >> > >> > Right now, using Bitcask backend, it can store roughly 68 objects per >> second. It seems to be a small number to me, but I don't know too much >> about Riak. I've tested the same script on a Couchbase cluster, with the >> same settings, and it could store roughly 1000 objects per second. >> Obviously, on Couchbase test, the data is not sent to non-volatile media >> immediately. Data is kept in memory to acknowledge the reception as fast as >> possible, and is sent to non-volatile media in background. I want Riak to >> behave the same way to increase the "writing speed", but I don't know how >> to do it or if it is possible. May be I'm trying to do something completely >> out of the purpose of Riak. >> > >> > I've tested Riak with Memory backend too, but it achieved only 72 >> objects per second. I expected it to work faster with Memory backend, cause >> there is no disk activity involved on it, but the final result is not that >> high. >> > >> > Again, I don't know if I'm trying to do something inappropriate. I >> think I'm missing something. >> > Is there any way to do it? >> > >> > If I could not make myself clear, please, let me know. >> > Thank you. >> > >> > >> > >> > >> > >> > ___ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to make Riak work faster (writing)
As I told, maybe I'm trying to use Riak in an improperly way, but this is a kind of requirement for me and I expected to be able to do it with Riak. I'm looking for a "NoSQL" solution and after try out a few of them, I think that I have two great solutions, Couchbase and Riak. To me, Riak seems to be much more mature, and this is my primary option, but, as I told, I can achieve some numbers with Couchbase that I can't with Riak. Given the maturity of Riak, I'm pretty sure that this is a configuration problem, but I need some help to find out how to solve it. If any one could help me with it, I'll be thankful. I don't want to give up on Riak so fast. Thank you. On Fri, Nov 2, 2012 at 9:46 AM, Uruka Dark wrote: > I understand. > To eliminate any problems related to Bitcask, I changed to Memory backend > and now I can store roughly 80 objs/sec. This speed can be achieved hitting > just one of them. > I tried to hit both of them at same time, and the speed drops to roughly > 68 objs/sec (each). > Do you have any suggestion about it? > > Thank you. > > On Fri, Nov 2, 2012 at 9:20 AM, Sebastian Cohnen < > sebastian.coh...@gmail.com> wrote: > >> You should hit both servers and not just with a concurrency level of 1. >> >> There are many more factors to consider, but these are highly dependent >> on your actual problem (not just a simple benchmark). Just to name a few: >> bitcask settings ( >> http://docs.basho.com/riak/latest/tutorials/choosing-a-backend/Bitcask/#Tuning-Bitcask), >> w-quorum, HTTP vs ProtoBuf, ring_creation_size, ... >> >> On 02.11.2012, at 13:15, Uruka Dark wrote: >> >> I'm hitting just one of them. >> At the beginning I tried to use the default settings (n_val = 3), then I >> started to create the bucket with n_val = 2. I've tested a lot of >> combinations to w, but I could not see any substantial improvement. >> If you have any suggestion, please, let me know. I can do any test. >> >> On Fri, Nov 2, 2012 at 9:04 AM, Sebastian Cohnen < >> sebastian.coh...@gmail.com> wrote: >> >>> What level of concurrency are you using in your test setup? Are you >>> hitting both servers with your test? What is your n_val and w? >>> >>> >>> On 02.11.2012, at 03:42, Uruka Dark wrote: >>> >>> > Hi, >>> > >>> > I'm new here and with Riak. If I do something wrong, please, let me >>> know. >>> > >>> > I've made a Riak cluster with two identical machines: Intel core i3 >>> 2.3GHz 4GB RAM 1TB HD. They are connected by a gigabit ethernet network. >>> Everything is working fine. I'm using a Bitcask backend. >>> > >>> > I've made a PHP script to do a performance test and find out how fast >>> Riak can be with these settings. What my script is doing is: to store 1 >>> objects with a 10K long data (string of 10240 x 'A'), and calculating how >>> many objects it stores per second. >>> > >>> > Right now, using Bitcask backend, it can store roughly 68 objects per >>> second. It seems to be a small number to me, but I don't know too much >>> about Riak. I've tested the same script on a Couchbase cluster, with the >>> same settings, and it could store roughly 1000 objects per second. >>> Obviously, on Couchbase test, the data is not sent to non-volatile media >>> immediately. Data is kept in memory to acknowledge the reception as fast as >>> possible, and is sent to non-volatile media in background. I want Riak to >>> behave the same way to increase the "writing speed", but I don't know how >>> to do it or if it is possible. May be I'm trying to do something completely >>> out of the purpose of Riak. >>> > >>> > I've tested Riak with Memory backend too, but it achieved only 72 >>> objects per second. I expected it to work faster with Memory backend, cause >>> there is no disk activity involved on it, but the final result is not that >>> high. >>> > >>> > Again, I don't know if I'm trying to do something inappropriate. I >>> think I'm missing something. >>> > Is there any way to do it? >>> > >>> > If I could not make myself clear, please, let me know. >>> > Thank you. >>> > >>> > >>> > >>> > >>> > >>> > ___ >>> > riak-users mailing list >>> > riak-users@lists.basho.com >>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to make Riak work faster (writing)
Hi Dimitri,
I don't know why, but I could not receive your reply. I saw it following
the url of the mailing list.
Any way, thank you for your reply.
This is my PHP script:
*isAlive())*
* {*
* echo "$str - ALIVE\n";*
* }*
* else*
* {*
* echo "$str - DEAD\n";*
* die();*
* }*
*
*
* $bucket = $client->bucket('zbucket');*
* $bucket->setNVal(2);*
*
*
* $cnt = 0;*
* $last = time();*
* $data = '';*
* for ($i=0; $i<10240; $i++)*
* {*
* $data .= 'A';*
* }*
*
*
* // main loop*
* for ($i=0; true; $i++)*
* {*
* $obj = $bucket->newObject(NULL, $data . $i);*
* $obj->store();*
* $cnt++;*
* if (time() - $last == 1)*
* {*
* echo "$cnt in $i\n";*
* $cnt = 0;*
* $last = time();*
* }*
* }*
*?>*
---
Uruka,
What does your load testing script look like? If you post the code
somewhere, I'll take a look to see if any obvious stumbling block comes to
mind. I agree, that's way lower than you should be getting with Riak on
that hardware.
Dmitri
On Fri, Nov 2, 2012 at 2:30 PM, Uruka Dark wrote:
> As I told, maybe I'm trying to use Riak in an improperly way, but this is
> a kind of requirement for me and I expected to be able to do it with Riak.
> I'm looking for a "NoSQL" solution and after try out a few of them, I think
> that I have two great solutions, Couchbase and Riak. To me, Riak seems to
> be much more mature, and this is my primary option, but, as I told, I can
> achieve some numbers with Couchbase that I can't with Riak. Given the
> maturity of Riak, I'm pretty sure that this is a configuration problem, but
> I need some help to find out how to solve it. If any one could help me with
> it, I'll be thankful. I don't want to give up on Riak so fast.
> Thank you.
>
> On Fri, Nov 2, 2012 at 9:46 AM, Uruka Dark wrote:
>
>> I understand.
>> To eliminate any problems related to Bitcask, I changed to Memory backend
>> and now I can store roughly 80 objs/sec. This speed can be achieved hitting
>> just one of them.
>> I tried to hit both of them at same time, and the speed drops to roughly
>> 68 objs/sec (each).
>> Do you have any suggestion about it?
>>
>> Thank you.
>>
>> On Fri, Nov 2, 2012 at 9:20 AM, Sebastian Cohnen <
>> sebastian.coh...@gmail.com> wrote:
>>
>>> You should hit both servers and not just with a concurrency level of 1.
>>>
>>> There are many more factors to consider, but these are highly dependent
>>> on your actual problem (not just a simple benchmark). Just to name a few:
>>> bitcask settings (
>>> http://docs.basho.com/riak/latest/tutorials/choosing-a-backend/Bitcask/#Tuning-Bitcask),
>>> w-quorum, HTTP vs ProtoBuf, ring_creation_size, ...
>>>
>>> On 02.11.2012, at 13:15, Uruka Dark wrote:
>>>
>>> I'm hitting just one of them.
>>> At the beginning I tried to use the default settings (n_val = 3), then I
>>> started to create the bucket with n_val = 2. I've tested a lot of
>>> combinations to w, but I could not see any substantial improvement.
>>> If you have any suggestion, please, let me know. I can do any test.
>>>
>>> On Fri, Nov 2, 2012 at 9:04 AM, Sebastian Cohnen <
>>> sebastian.coh...@gmail.com> wrote:
>>>
>>>> What level of concurrency are you using in your test setup? Are you
>>>> hitting both servers with your test? What is your n_val and w?
>>>>
>>>>
>>>> On 02.11.2012, at 03:42, Uruka Dark wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > I'm new here and with Riak. If I do something wrong, please, let me
>>>> know.
>>>> >
>>>> > I've made a Riak cluster with two identical machines: Intel core i3
>>>> 2.3GHz 4GB RAM 1TB HD. They are connected by a gigabit ethernet network.
>>>> Everything is working fine. I'm using a Bitcask backend.
>>>> >
>>>> > I've made a PHP script to do a performance test and find out how fast
>>>> Riak can be with these settings. What my script is doing is: to store 1
>>>> objects with a 10K long data (string of 10240 x 'A'), and calculating how
>>>> many objects it stores per second.
>>>> >
>>>> > Right now, using Bitcask backend, it can store roughly 68 objects per
>>>> second. It seems to be a small number to me, but I don't know too much
>>>> about Riak. I've tested the same script on a Couchbase cluster, with the
>>>> same settings, and it could store roughly 1
Re: How to make Riak work faster (writing)
First of all, thank you for your replies, Brian and Christian.
I rewrote the script in python:
###
import riak
import time
import sys
hostname = '10.1.1.221'
portnumber = 8087
#portnumber = 8098
if len(sys.argv) > 1:
hostname = sys.argv[1]
#client = riak.RiakClient(host=hostname,port=portnumber)
client =
riak.RiakClient(host=hostname,port=portnumber,transport_class=riak.RiakPbcTransport)
print hostname + ':' + str(portnumber)
# Choose the bucket to store data in.
bucket = client.bucket('nBucket')
data = 'X' * 10240
cnt = 0
last = time.time()
while 1 :
obj = bucket.new(None, data)
obj.store()
cnt += 1
if time.time() - last >= 1:
print cnt
last = time.time()
cnt = 0
###
This time I got better numbers just changing from PHP to Python.
Using HTTP in python, I got roughly 96 objs/sec (against 72 objs/sec in
php).
Using Protocol Buffers in python, I got roughly 175 objs/sec (not available
in php).
I also tried to run more than one process in python, with PB:
1 proc = 175 objs/sec
2 proc = 157 objs/sec (each)
3 proc = 124 objs/sec (each)
As we can see, using a new language and Protocol Buffers I can get much
better results. Unfortunately, I think those numbers are still too low
(considering that I'm using the Memory backend). Do you think we can get
better results?
And again, thank you for your support.
On Fri, Nov 2, 2012 at 5:30 PM, Christian Dahlqvist wrote:
> On 02/11/2012 19:10, Uruka Dark wrote:
>
> Hi Dimitri,
> I don't know why, but I could not receive your reply. I saw it following
> the url of the mailing list.
>
> Any way, thank you for your reply.
>
> This is my PHP script:
>
> * * require_once "riak_lib/riak.php";*
> *
> *
> * if (isset($argv[1]))*
> * {*
> * define('HOST', $argv[1]);*
> * }*
> * else*
> * {*
> * define('HOST', '10.1.1.221');*
> * }*
> * define('PORT', 8098);*
> *
> *
> * $client = new RiakClient(HOST, PORT);*
> * $str = HOST . ":" . PORT;*
> * if ($client->isAlive())*
> * {*
> * echo "$str - ALIVE\n";*
> * }*
> * else*
> * {*
> * echo "$str - DEAD\n";*
> * die();*
> * }*
> *
> *
> * $bucket = $client->bucket('zbucket');*
> * $bucket->setNVal(2);*
> *
> *
> * $cnt = 0;*
> * $last = time();*
> * $data = '';*
> * for ($i=0; $i<10240; $i++)*
> * {*
> * $data .= 'A';*
> * }*
> *
> *
> * // main loop*
> * for ($i=0; true; $i++)*
> * {*
> * $obj = $bucket->newObject(NULL, $data . $i);*
> * $obj->store();*
> * $cnt++;*
> * if (time() - $last == 1)*
> * {*
> * echo "$cnt in $i\n";*
> * $cnt = 0;*
> * $last = time();*
> * }*
> * }*
> *?>*
>
>
> ---
>
> Uruka,
>
> What does your load testing script look like? If you post the code
> somewhere, I'll take a look to see if any obvious stumbling block comes to
> mind. I agree, that's way lower than you should be getting with Riak on
> that hardware.
>
> Dmitri
>
>
> On Fri, Nov 2, 2012 at 2:30 PM, Uruka Dark wrote:
>
>> As I told, maybe I'm trying to use Riak in an improperly way, but this
>> is a kind of requirement for me and I expected to be able to do it with
>> Riak. I'm looking for a "NoSQL" solution and after try out a few of them, I
>> think that I have two great solutions, Couchbase and Riak. To me, Riak
>> seems to be much more mature, and this is my primary option, but, as I
>> told, I can achieve some numbers with Couchbase that I can't with Riak.
>> Given the maturity of Riak, I'm pretty sure that this is a configuration
>> problem, but I need some help to find out how to solve it. If any one could
>> help me with it, I'll be thankful. I don't want to give up on Riak so fast.
>> Thank you.
>>
>> On Fri, Nov 2, 2012 at 9:46 AM, Uruka Dark wrote:
>>
>>> I understand.
>>> To eliminate any problems related to Bitcask, I changed to Memory
>>> backend and now I can store roughly 80 objs/sec. This speed can be achieved
>>> hitting just one of them.
>>> I tried to hit both of them at same time, and the speed drops to roughly
>>> 68 objs/sec (each).
>>> Do you have any suggestion about it?
>>>
>>> Thank you.
>>>
>>> On Fri, Nov 2, 2012 at 9:20 AM, Sebastian Cohnen <
>>> sebastian.coh...@gmail.com> wrote:
>>>
>>>> You should hit both servers and not just with a concurrency level of
&
Re: How to make Riak work faster (writing)
Jared,
Thank you for you time and reply.
I got impressed by your numbers and I started to double check my settings.
I found a big problem here, my third machine (the one out of the cluster,
making the load), was not talking to Riak in gigabit speed, it was 100 Mbs.
I changed the network cable and it's working fine now.
I ran my python script again and I already could see better results: 252
ops/sec (before the fix it was 175 ops/sec).
I also ran your benchmark .config, and these are my numbers:
https://dl.dropbox.com/u/308392/summary.png
As you can see, even so, I'm still far from your results.. not even close,
and now I'm using Bitcask.
Anyway, my current position is much better than at the beggining. I'll
double-check all over again, cause now I have a confirmation that there is
something wrong.
If you have any suggestion, please, let me know.
Once again, thank you.
On Sat, Nov 3, 2012 at 3:08 AM, Jared Morrow wrote:
> I forgot to mention that 2000 ops/sec was on bitcask, not memory. I
> didn't bother with the memory backend.
>
> -Jared
>
>
> On Sat, Nov 3, 2012 at 12:05 AM, Jared Morrow wrote:
>
>> Uruka,
>>
>> So looking at your results something is really wrong with your setup. I
>> was surprised by your numbers, so I made two VM's each with only 1gb of RAM
>> on two different boxes also on a 1gb switch.
>>
>> I ran a put of 100,000 keys at 10kb in size.
>>
>> I didn't do any tuning at all on the VM's and these were quick Ubuntu
>> 10.04 VM's with 2 virtual CPU's and 1 gig of ram. I also didn't change any
>> settings in Riak, except for the IP address and listening ports.
>>
>> Here is the summary of the results showing around 2000 ops/sec
>> https://dl.dropbox.com/u/183971/summary.png
>>
>> So my main thought is that you weren't actually using N=1 for your puts
>> and you were using the default N value of 3, meaning you were writing each
>> key/value 3 times, and with 2 nodes this is doing a lot of writes to the
>> same disk multiple times.
>>
>> To be sure you have N=1, you can use 'riak attach' on each node and enter
>> the following command:
>>
>> riak_core_bucket:set_bucket(<<"pop1">>,[{n_val,1}]).
>>
>>
>> If you bucket name is "pop1" as in my case. That name is completely
>> arbitrary.
>>
>> Sorry I'm late to this thread, I had to find some time to setup the test.
>>
>> For reference I used https://github.com/basho/basho_bench for the
>> benchmark. With the following .config file
>> https://gist.github.com/e630b63f4a025a0fb634
>>
>> Hope this helps,
>> Jared
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to make Riak work faster (writing)
r comes first, and
keep
%% the last 5 rotations. See the lager README for a description
of
%% the time rotation format:
%% https://github.com/basho/lager/blob/master/README.org
%%
%% If you wish to disable rotation, you can either set the size
to 0
%% and the rotation time to "", or instead specify a 2-tuple
that only
%% consists of {Logfile, Level}.
{handlers, [
{lager_console_backend, info},
{lager_file_backend, [
{"/var/log/riak/error.log", error, 10485760, "$D0", 5},
{"/var/log/riak/console.log", info, 10485760, "$D0", 5}
]}
]},
%% Whether to write a crash log, and where.
%% Commented/omitted/undefined means no crash logger.
{crash_log, "/var/log/riak/crash.log"},
%% Maximum size in bytes of events in the crash log - defaults
to 65536
{crash_log_msg_size, 65536},
%% Maximum size of the crash log in bytes, before its rotated,
set
%% to 0 to disable rotation - default is 0
{crash_log_size, 10485760},
%% What time to rotate the crash log - default is no time
%% rotation. See the lager README for a description of this
format:
%% https://github.com/basho/lager/blob/master/README.org
{crash_log_date, "$D0"},
%% Number of rotated crash logs to keep, 0 means keep only the
%% current one - default is 0
{crash_log_count, 5},
%% Whether to redirect error_logger messages into lager -
defaults to true
{error_logger_redirect, true}
]},
%% riak_sysmon config
{riak_sysmon, [
%% To disable forwarding events of a particular type, use a
%% limit of 0.
{process_limit, 30},
{port_limit, 2},
%% Finding reasonable limits for a given workload is a matter
%% of experimentation.
{gc_ms_limit, 100},
{heap_word_limit, 40111000},
%% Configure the following items to 'false' to disable logging
%% of that event type.
{busy_port, true},
{busy_dist_port, true}
]},
%% SASL config
{sasl, [
{sasl_error_logger, false}
]},
%% riak_control config
{riak_control, [
%% Set to false to disable the admin panel.
{enabled, true},
%% Authentication style used for access to the admin
%% panel. Valid styles are 'userlist' .
{auth, none},
%% If auth is set to 'userlist' then this is the
%% list of usernames and passwords for access to the
%% admin panel.
{userlist, [{"user", "pass"}
]},
%% The admin panel is broken up into multiple
%% components, each of which is enabled or disabled
%% by one of these settings.
{admin, true}
]}
].
--
I have two machines with those settings: 10.1.1.221 and 10.1.1.222. They
are working together.
Do you see any problem on that?
Again, if you think I can't go any further with those default settings
(without tuning FS, etc), please, let me know.
Thank you.
On Sat, Nov 3, 2012 at 4:43 PM, Jared Morrow wrote:
> Uruka,
>
> Now that you got some somewhat reasonable numbers, it is probably time to
> discuss what you are trying to get out of Riak. We typically recommend 4
> or 5 nodes minimum for a Riak install because that is the point where the
> distribution becomes a performance benefit rather than a hindrance. I know
> you were just load testing, but I'd recommend considering a test with 4 or
> 5 nodes, with default N values. During the test, remove a node (power it
> off, or 'riak stop' it). Or like someone else mentioned start with a 3 or
> 4 node cluster and add a node to see how the performance goes up and no
> further operations work is needed to rebalance the data around the cluster.
> This is really where Riak shines over some alternative databases, the ease
> of scaling and dealing with failures. SIngle node performance although fun
> to try and tune to get the most out of it, isn't as interesting on a long
> timeline when trying to scale the system. Obviously single node
> performance is still important, dont' get me wrong. Riak isn't always the
> best choice, but when it comes with staying available and performance while
> systems are failing no other system has a better real-world story than Riak.
>
> If you still want to get your single node performance up, we have several
> pages on our docs
Re: How to make Riak work faster (writing)
Just an update.
I ran the benchmark again, but now, using Memory backend:
https://dl.dropbox.com/u/308392/memory_summary.png
This was the result using Bitcask backend:
https://dl.dropbox.com/u/308392/bitcask_summary.png
The difference is not that big in my environment. I was expecting much
better results, but I don't know if it was supposed to happen.
Anyway, your results are still much better, even when I'm using memory only
backend (50% of yours).
Maybe it can help to understand what is happening.
On Sat, Nov 3, 2012 at 7:31 PM, Uruka Dark wrote:
> Jared,
>
> Again, thank you very much.
> You helped me a lot.
>
> I perfectly understand your point. I'm just starting to know Riak and I
> want to go much deeper. But, before I keep going, I want make sure that I'm
> starting with the right foot :)
> I double/triple-checked and I still have no additional clues about what is
> happening.
>
> You've reached much better results than mine using your default settings,
> and, given my numbers, I'm still missing something. I would like at least
> to get closer to your results. If you think that I'll not make any better
> than this with my default settings, please, let me know.
>
> Anyway, this is my app.config:
>
> %% -*- mode: erlang;erlang-indent-level: 4;indent-tabs-mode: nil -*-
> %% ex: ft=erlang ts=4 sw=4 et
> [
> %% Riak Client APIs config
> {riak_api, [
> %% pb_backlog is the maximum length to which the queue of
> pending
> %% connections may grow. If set, it must be an integer >= 0.
> %% By default the value is 5. If you anticipate a huge number
> of
> %% connections being initialised *simultaneously*, set this
> number
> %% higher.
> %% {pb_backlog, 64},
>
> %% pb_ip is the IP address that the Riak Protocol Buffers
> interface
> %% will bind to. If this is undefined, the interface will not
> run.
> {pb_ip, "10.1.1.221" },
>
> %% pb_port is the TCP port that the Riak Protocol Buffers
> interface
> %% will bind to
> {pb_port, 8087 }
> ]},
>
> %% Riak Core config
> {riak_core, [
> %% Default location of ringstate
> {ring_state_dir, "/var/lib/riak/ring"},
>
> %% Default ring creation size. Make sure it is a power of 2,
> %% e.g. 16, 32, 64, 128, 256, 512 etc
> %{ring_creation_size, 64},
>
> %% http is a list of IP addresses and TCP ports that the Riak
> %% HTTP interface will bind.
> {http, [ {"10.1.1.221", 8098 } ]},
>
> %% https is a list of IP addresses and TCP ports that the
> Riak
> %% HTTPS interface will bind.
> {https, [{ "10.1.1.221", 8069 }]},
>
> %% Default cert and key locations for https can be overridden
> %% with the ssl config variable, for example:
> {ssl, [
> {certfile, "/etc/riak/server.crt"},
> {keyfile, "/etc/riak/server.key"}
> ]},
>
> %% riak_handoff_port is the TCP port that Riak uses for
> %% intra-cluster data handoff.
> {handoff_port, 8099 },
>
> %% To encrypt riak_core intra-cluster data handoff traffic,
> %% uncomment the following line and edit its path to an
> %% appropriate certfile and keyfile. (This example uses a
> %% single file with both items concatenated together.)
> %{handoff_ssl_options, [{certfile, "/tmp/erlserver.pem"}]},
>
> %% DTrace support
> %% Do not enable 'dtrace_support' unless your Erlang/OTP
> %% runtime is compiled to support DTrace. DTrace is
> %% available in R15B01 (supported by the Erlang/OTP
> %% official source package) and in R14B04 via a custom
> %% source repository & branch.
> {dtrace_support, false},
>
> %% Platform-specific installation paths (substituted by
> rebar)
> {platform_bin_dir, "/usr/sbin"},
> {platform_data_dir, "/var/lib/riak"},
> {platform_etc_dir, "/etc/riak"},
> {platform_lib_dir, "/usr/lib/riak/lib"},
> {platform_log_dir, "/var/log/riak"}
> ]},
>
> %% Riak KV config
> {riak_kv, [
> %% Storage_backend specifies the Erlang modul
Re: How to make Riak work faster (writing)
Sorry, I forgot to add your results, using bitcask:
https://dl.dropbox.com/u/183971/summary.png
On Mon, Nov 5, 2012 at 1:17 PM, Uruka Dark wrote:
> Just an update.
>
> I ran the benchmark again, but now, using Memory backend:
> https://dl.dropbox.com/u/308392/memory_summary.png
>
> This was the result using Bitcask backend:
> https://dl.dropbox.com/u/308392/bitcask_summary.png
>
> The difference is not that big in my environment. I was expecting much
> better results, but I don't know if it was supposed to happen.
> Anyway, your results are still much better, even when I'm using memory
> only backend (50% of yours).
>
> Maybe it can help to understand what is happening.
>
> On Sat, Nov 3, 2012 at 7:31 PM, Uruka Dark wrote:
>
>> Jared,
>>
>> Again, thank you very much.
>> You helped me a lot.
>>
>> I perfectly understand your point. I'm just starting to know Riak and I
>> want to go much deeper. But, before I keep going, I want make sure that I'm
>> starting with the right foot :)
>> I double/triple-checked and I still have no additional clues about what
>> is happening.
>>
>> You've reached much better results than mine using your default settings,
>> and, given my numbers, I'm still missing something. I would like at least
>> to get closer to your results. If you think that I'll not make any better
>> than this with my default settings, please, let me know.
>>
>> Anyway, this is my app.config:
>>
>> %% -*- mode: erlang;erlang-indent-level: 4;indent-tabs-mode: nil -*-
>> %% ex: ft=erlang ts=4 sw=4 et
>> [
>> %% Riak Client APIs config
>> {riak_api, [
>> %% pb_backlog is the maximum length to which the queue of
>> pending
>> %% connections may grow. If set, it must be an integer >= 0.
>> %% By default the value is 5. If you anticipate a huge number
>> of
>> %% connections being initialised *simultaneously*, set this
>> number
>> %% higher.
>> %% {pb_backlog, 64},
>>
>> %% pb_ip is the IP address that the Riak Protocol Buffers
>> interface
>> %% will bind to. If this is undefined, the interface will
>> not run.
>> {pb_ip, "10.1.1.221" },
>>
>> %% pb_port is the TCP port that the Riak Protocol Buffers
>> interface
>> %% will bind to
>> {pb_port, 8087 }
>> ]},
>>
>> %% Riak Core config
>> {riak_core, [
>> %% Default location of ringstate
>> {ring_state_dir, "/var/lib/riak/ring"},
>>
>> %% Default ring creation size. Make sure it is a power of
>> 2,
>> %% e.g. 16, 32, 64, 128, 256, 512 etc
>> %{ring_creation_size, 64},
>>
>> %% http is a list of IP addresses and TCP ports that the
>> Riak
>> %% HTTP interface will bind.
>> {http, [ {"10.1.1.221", 8098 } ]},
>>
>> %% https is a list of IP addresses and TCP ports that the
>> Riak
>> %% HTTPS interface will bind.
>> {https, [{ "10.1.1.221", 8069 }]},
>>
>> %% Default cert and key locations for https can be
>> overridden
>> %% with the ssl config variable, for example:
>> {ssl, [
>> {certfile, "/etc/riak/server.crt"},
>> {keyfile, "/etc/riak/server.key"}
>> ]},
>>
>> %% riak_handoff_port is the TCP port that Riak uses for
>> %% intra-cluster data handoff.
>> {handoff_port, 8099 },
>>
>> %% To encrypt riak_core intra-cluster data handoff traffic,
>> %% uncomment the following line and edit its path to an
>> %% appropriate certfile and keyfile. (This example uses a
>> %% single file with both items concatenated together.)
>> %{handoff_ssl_options, [{certfile, "/tmp/erlserver.pem"}]},
>>
>> %% DTrace support
>> %% Do not enable 'dtrace_support' unless your Erlang/OTP
>> %% runtime is compiled to support DTrace. DTrace is
>> %% available in R15B01 (supported by the Erlang/OTP
>> %% official source package) and in R14B04 via a custom
>> %% source repositor
Re: How to make Riak work faster (writing)
First of all, thank you for your reply.
Well, if you tell me that I can beat myself up trying to get what another
person gets in a benchmark, then I don't understand what's the whole point
in post your results here. I thought that you were trying to tell me that
in an similar setup, you could do much better, and that I probably had some
problem on my setup (and you were right). Considering how different our
results are, I imagine that there is something wrong - by "wrong" I mean: I
have a setup problem; my hardware is not up to the task and our
environments are not that similar; etc. That's what I'm trying to figure
out and also, I'm trying to know the tool.
I have a few different scenarios to face: heavy write, heavy read, both of
them, etc. Now, I'm considering the heavy write scenario and later I'll
deal with others. If I jump from one scenario to another without at least a
minimum solid conclusion, it will not help me.
Based on what you said, I replaced the machine that was producing the load
by a new one identical to the cluster machines (intel core i3 2.3GHz, 4GB
RAM 1TB HD). Now I have 3 machines with the same setup in a gigabit network.
I started using bitcask backend:
https://dl.dropbox.com/u/308392/sum_bit.png
Then I tried memory backend:
https://dl.dropbox.com/u/308392/sum_mem.png
Now we can see a major impact on the results. The memory backend could do
much better with roughly 4000 ops/sec, but bitcask not so much, about +200
ops/sec than the last result. Changing the third machine really worked.
Just to see what would happen, I decided to try the benchmark locally (of
corse, I knew that it would put a heavy load over the CPU). I ran the test
from one of the cluster machines.
Results:
Bitcask:
https://dl.dropbox.com/u/308392/sum_bit_local.png
Memory:
https://dl.dropbox.com/u/308392/sum_mem_local.png
Well, it seems that my bottleneck is related to my disks.
Once again, thank you.
You helped me alot.
On Mon, Nov 5, 2012 at 4:36 PM, Jared Morrow wrote:
> So if you are getting close to similar numbers with the memory and bitcask
> backends, you know that file IO isn't your bottleneck in the load test. My
> guess is that the machine you are running the load with (where you are
> running basho_bench) can't keep up with the test. I'm running basho_bench
> on a core i7 on the same switch as the riak nodes, so that could be enough
> of the difference. Again though, you can beat yourself up trying to just
> get what another person gets in a benchmark, so it really comes down to
> what you want out of it. What are your app's requirements? What do you
> expect to be the requirements a year from now? What do your access patterns
> look like, more read heavy or more write heavy?
>
> Right now, you are only doing bulk load experiments, what about
> Write/Read/Update that simulate your expected usage pattern? What we have
> done in the past as a good test is to do a bulk load like you are doing
> with say 10 or 100 million keys. That'll make sure Riak is loaded with a
> real world amount of data to start. Then run another pareto distribution
> run (using something like {key_generator, {int_to_bin, {pareto_int,
> 1000}}}.) with a spread of puts/gets/updates like {operations, [{get,
> 4},{put, 1},{update, 1}]}.. I'll tell you right now with a two node
> cluster you probably aren't going to be happy with the results as that is
> not how Riak was designed to work at its best. If you can swing it, add two
> more nodes to the test, set N=2 instead of N=1 so you are getting at least
> some data safety and run the above. If it meets the need of your app,
> awesome. if it doesn't come back and discuss it or try another solution
> with the same requirements.
>
> -Jared
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
