Chris Read:
Back in 2013 you reported a performance problem with Riak 1.4.2 running on
a 10GbE network where Riak would never hit speeds faster than 2.5Gbps on
the network.
I'm seeing the same thing with Riak 1.4.2 and RiakCS. I've followed all of
the tuning suggestions, my MTU is set to 9000 on the ethernet interfaces, I
have one 10GbE network just for the backend inter-node data and one 10GbE
"public" network where RiakCS listens for connections and which basho_bench
uses to generate the load. I have 1-4 client systems on the public side
running basho_bench and no matter how much traffic I generate with
basho_bench I never see more than 3Gbits/s on the network. (It doesn't seem
to matter if I run 1 or 4 clients, each with 200 concurrent sessions, the
network data rate is about the same.) I'm running jnettop in two different
windows during the tests to watch the aggregate network traffic on the
private inter-node data network and the "public" basho_bench
traffic-generating network.
I've tested the network with iperf3 and it shows 9.92Gbits/s throughput
with a TCP maximum segment size of 9000.
I've tested the filesystems on each of the 6 Riak nodes using fio, and I
can write to the filesystems at ~12.8Gbits/s, so the filesystem is not the
bottleneck. Each node has 128GB RAM and is running the bitcask backend. The
servers are mostly idle.
I tried Sean's solution of increasing these values to:
{riak_core, [
{handoff_batch_threshold, 4194304},
{handoff_concurrency, 10} ]}
... as described in
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-October/013787.html,
but that had no effect.
With my current hardware I'd expect that the 10GbE network would be the
bottleneck, and I'd expect write speeds to top out at the top end of the
network speed.
There was no follow-up message on the mailing list to indicate how or if
you'd solved the problem. Did you find a solution?
(Please direct replies to the mailing list.)
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
