Greate thanks to all of you!
I'm very sorry for a late response!
Is it polite to make a reply to all in one email? (If not, ask for forgiveness)
To Alexander:
There are 2 physical machines in our cluster, and 5 riak VMs on each machine.
Two physical CPUs in each machine (totally 12 cores), and I assigned 2 cores
for each riak VM.
All the 12 riak VMs are based on the same shared storage server (by
FreeNAS+iSCSI), whicn means all riak VMs don't use machines' local disks as
their storage.
And they are in a 1Gb LAN, each physical machine has 6 NICs, 4 of them are for
iSCSI(multipath).
So the network bandwidth is probably not the bottleneck, but disk IO probably
be.
To Sean Cribbs:
Yes, as you said, more "concurrent" means more load for basho benchmark machine.
But during my previous tests, I noticed that cpu utilization of basho benchmark
machine is about 50%.
But I want to try run basho benchmark tool on two machines as you suggested.
I noticed the command "basho_bench" has an option "-J --join", I tried but
faild with error report "escript: exception throw:
{no_host,'root@10.11.20.44'} ".
I have configured SSH public-key&no-password login for those two basho
benchmark machines(they have same environments like OS,erlang and basho
benchmark versions). I searched about this, but didn't find how to run basho
benchmark tool on two separate machines.
Could you tell me how to do or is there any guidance about this?
To Christian Dahlqvist:
Thanks for your tips, and I think we will try "basho_bench_driver_riakc_pb".
Not only because of this "Horizontal Scalability" problem, but also for better
performance.
And also, I will change the key generator for subsequent tests.
To Evan Vigil-McClanahan:
Yes, I am using Riak 2.0 for tests.
And "numa zones", "sbt"? Sorry I know little about this, but I will read more
about this to check if it can help me.
I will run more tests with the suggestions from all of you!
Thank you again!
And I will let you know if get any progress or result.
From: Sean Cribbs
Date: 2014-03-08 06:54
To: Alexander Sicular
CC: yanqun....@mesheven.com; riak-users
Subject: Re: Questions about the Riak Horizontal Scalability
Note as well that the 'concurrent' setting will *not* typically increase
throughput beyond a certain point (instead you will simply have many workers
waiting in the run queue). You are also only driving load from a single machine
with limits on its CPU and network throughput. Start by trying a more
reasonable concurrency level, like two (2) times the number of CPU cores in the
basho_bench machine. Also consider running basho_bench from multiple machines
at once. Consider the line rate of your network as well; in the past I have
easily been able to saturate a gigabit network with basho_bench.
On Fri, Mar 7, 2014 at 4:45 PM, Alexander Sicular <sicul...@gmail.com> wrote:
How many physical machines (disks/cores/nics) are in your cluster?
-Alexander
@siculars
http://siculars.posthaven.com
Sent from my iRotaryPhone
On Mar 6, 2014, at 22:07, "yanqun....@mesheven.com" <yanqun....@mesheven.com>
wrote:
Greetings all...
I am running tests with Riak Cluster.
When this cluster has only one node, the max throughput is about 1500 ops/s(CPU
becomes the bottleneck).
But after I added 4 more nodes into the cluster, no matter how I changed the
configuration file of Basho benchmark tool,
the number of throughput have never been more than ~3500 ops/s.
CPU utilization on each node is about 60~65% (captured by iostat and sar)
And when I added 5 more nodes(10 nodes totally), it is the same result.
And CPU utilization on each node is about 60%.
I can't understand why.
During the tests, I tried:
1)Change ring size from 64 to 128
2)Change basho benchmark configuration file, like "mode","concurrent" and
"key_generator"...
3)Modify some system limits on both riak nodes and basho bench servers, such as
file descriptor , tcp settings.
4)Monitoring system like cpu, memory and IO, and none of them become the
bottleneck.
5)Monitoring the basho bench server, low load.
Any ideas?
Or did I make any mistakes in my configuration file?
Configuration file of Basho benchmark tool:
===================================================================
{mode, max}.
{duration, 15}.
{concurrent, 500}.
{driver, basho_bench_driver_http_raw}.
{http_raw_ips,
["10.11.20.46","10.11.20.47","10.11.20.48","10.11.20.49","10.11.20.50","10.11.20.51","10.11.20.52","10.11.20.53"]}.
{http_raw_port, 8098}.
{http_raw_path, "/riak/test"}.
{key_generator, {partitioned_sequential_int,
1000000000000000,8000000000000000}}. %%we want 16 bytes long key-length
{value_generator, {fixed_bin, 1024}}. %%1KB
{operations, [{insert,4}, {get,6}]}.
===============================================================================
Here are our server specifications(VMs based on Xenserver):
1)Riak nodes: 2cores + 4GB RAM + 20GB disk
2)Basho benchmark, 4 cores + 4GB RAM, 30GB disk.
Thank you very much !
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Sean Cribbs <s...@basho.com>
Software Engineer
Basho Technologies, Inc.
http://basho.com/_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
