We're using Riak as a simple key value store and we're having write
performance problems which we think is due to the format of our keys which
we can't easily change because they're tied into different parts of the
business and systems.
We're not trying to do anything complicated with Riak: No solr, secondary
indexes or map reducing - just simple keys to strings of around 10Kb of
JSON.
We've got upwards of 3 billion records to store so we've opted for LevelDb
as the backend.
It's a 3 node cluster running on 3 dedicated Ubuntu VMs each with 16 cpu
cores and 12GB memory backed by SSDs on a 10Gb network.
Using basho bench we know that it's capable of speeds upwards of *5000 rows
per sec* when using randomised keys, but the problem comes when we use our
actual data.
The keys are formatted using the following pattern:
USC~1930~100000000~10000~001USC~1930~100000000~10000~002USC~1930~100000000~10000~003USC~1930~100000000~10000~004
Most of the long key stays the same with numbers at the end going up. (The
"~" are changeable - we can set them to whatever character. They're just
delimiters in the keys)
Using these keys, write performance is a tenth of the speed at *400 rows per
sec*.
We don't need to worry about different versions of the data so we've set the
following in our bucket type:
"allow_mult": false"last_write_wins": true"DW": 0"n_val": 2"w": 1"r":
1"basic_quorum": false
On the riak servers we've set the ulimit to:
riak soft nofile 32768riak hard nofile 65536
and other settings like this:
ring_size = 128protobuf.backlog = 1024anti_entropy = passive
We're using the v2 .net client from basho to do the putting which runs in an
API on 3 machines.
We've checked all the usual bottlenecks: CPU, memory, network IO, disk IO
and throttles on the Riak servers and windows API servers.
As a kicker, if we reverse the keys e.g.
100~00001~000000001~0391~CSU
speed goes up to over *3000 rows*, but that's a dirty kludge.
Can anyone explain why Riak doesn't like sequential alpha-numeric keys and
what we can change to improve performance?
Thanks!
--
View this message in context:
http://riak-users.197444.n3.nabble.com/How-to-increase-Riak-write-performance-for-sequential-alpha-numeric-keys-tp4034219.html
Sent from the Riak Users mailing list archive at Nabble.com.
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
