Hello, Leveldb is a sorted backend by key name (and bucket.) I don't know the details on exactly how it does the sort but there may be considerations there.
Another question is why is your ring size 128 on a 3 node cluster? Ring size determines the number of virtual nodes in the cluster which are then equally allocated to physical nodes in the cluster. In your config, each node will have about 42 vnodes. With leveldb each vnode wants a minimum of 300MB ram (with performance increasing through 2.5GB ram.) There is only one configurable leveldb memory setting which defaults to allocating 70% system ram to leveldb. If you do the math you'll see you're way short. This memory issue might not have anything to do with alpha sorting but it will catch up with you eventually. -Alexander On Thursday, May 5, 2016, alexc155 <ahcl...@gmail.com> wrote: > We're using Riak as a simple key value store and we're having write > performance problems which we think is due to the format of our keys which > we can't easily change because they're tied into different parts of the > business and systems. > > We're not trying to do anything complicated with Riak: No solr, secondary > indexes or map reducing - just simple keys to strings of around 10Kb of > JSON. > > We've got upwards of 3 billion records to store so we've opted for LevelDb > as the backend. > > It's a 3 node cluster running on 3 dedicated Ubuntu VMs each with 16 cpu > cores and 12GB memory backed by SSDs on a 10Gb network. > > Using basho bench we know that it's capable of speeds upwards of *5000 > rows per sec* when using randomised keys, but the problem comes when we > use our actual data. > > The keys are formatted using the following pattern: > > USC~1930~100000000~10000~001 > USC~1930~100000000~10000~002 > USC~1930~100000000~10000~003 > USC~1930~100000000~10000~004 > > Most of the long key stays the same with numbers at the end going up. (The > "~" are changeable - we can set them to whatever character. They're just > delimiters in the keys) > > Using these keys, write performance is a tenth of the speed at *400 rows > per sec*. > > We don't need to worry about different versions of the data so we've set > the following in our bucket type: > > "allow_mult": false > "last_write_wins": true > "DW": 0 > "n_val": 2 > "w": 1 > "r": 1 > "basic_quorum": false > > On the riak servers we've set the ulimit to: > > riak soft nofile 32768 > riak hard nofile 65536 > > and other settings like this: > > ring_size = 128 > protobuf.backlog = 1024 > anti_entropy = passive > > We're using the v2 .net client from basho to do the putting which runs in > an API on 3 machines. > > We've checked all the usual bottlenecks: CPU, memory, network IO, disk IO > and throttles on the Riak servers and windows API servers. > > As a kicker, if we reverse the keys e.g. > > 100~00001~000000001~0391~CSU > > speed goes up to over *3000 rows*, but that's a dirty kludge. > > Can anyone explain why Riak doesn't like sequential alpha-numeric keys and > what we can change to improve performance? > > Thanks! > > ------------------------------ > View this message in context: How to increase Riak write performance for > sequential alpha-numeric keys > <http://riak-users.197444.n3.nabble.com/How-to-increase-Riak-write-performance-for-sequential-alpha-numeric-keys-tp4034219.html> > Sent from the Riak Users mailing list archive > <http://riak-users.197444.n3.nabble.com/> at Nabble.com. > > -- Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
