Hi Alexander,
Thanks a lot for your input. I have a few follow-up questions.
> Stupid math:
>
> 3e7 x 3 (replication) / 9 = 1e7 minimum objects per node ( absolutely more
> due to obj > 1MB size )
>
> 1e7 x ~400 bytes per obj in ram = 4e9 ram per node just for bitcask. Aka 4
> GB.
>
> You already hit your limit.
>
It makes sense that this is hitting the limit when all keys are in ram
(i.e., with bitcask), but I thought leveldb did not keep all keys in ram,
so wouldn't have as high ram requirements. Does leveldb require the same
(or more) ram as multi backend?
Assuming defaults:
>
> Default ring_size = 64 / 9 nodes ~ 7 virtual nodes per physical node.
>
> Default leveldb ram allocation = 70%
>
These are good assumptions. I'm running a mostly default configuration.
Aside: I realized too late that I should have used a larger ring size, but
I don't see an easy way to change that short of writing custom data
migration code. FWIW, I can easily spin up a new cluster with a new ring
size. Is there a supported way to change the ring size on a cluster with
data in it or migrate to a new cluster with larger ring size without
writing a custom migration to pull data out of the old (small ring) cluster
and stuff it in a new (larger ring) cluster?
Also it sounds like increasing the ring size might increase my ram
requirements, so maybe in my situation (with limited ram) it's actually
better to have a smaller ring? Is that true?
Leveldb operates, aka consumes resources including ram, on a vnode basis.
> It likes to consume ram on the order of 300MB through 2.5GB per vnode
In my case it sounds like I need somewhere between 2GB and 17.5GB of
ram/node (with 7 vnodes per physical node) for leveldb. Can you explain
this 300MB-2.5GB ram/vnode requirement in conjunction with the 70% ram
allocation setting? How does the 70% setting affect ram usage?
Remember that I have a sparsely accessed dataset, so it is not necessary
that all keys be kept in ram since most of them will not be accessed
frequently and it's fine if it takes longer to access a key that has not
been accessed for a while. My constraints are mainly in hardware, so I'm
trying to get a configuration that will run acceptably on minimal hardware.
So far (using swap) I have acceptable performance, but I'd prefer to
eliminate swap and switch to leveldb backend if I can get to comparable
performance levels that way.
Thanks so much for your help!
~ Daniel
On Tue, Nov 22, 2016 at 12:27 PM, Alexander Sicular
wrote:
> Hi Daniel,
>
> Ya, I'm not surprised you're having issues. 4GB ram is woefully
> underspecd. 😔
>
> 🤓Stupid math:
>
> 3e7 x 3 (replication) / 9 = 1e7 minimum objects per node ( absolutely more
> due to obj > 1MB size )
>
> 1e7 x ~400 bytes per obj in ram = 4e9 ram per node just for bitcask. Aka 4
> GB.
>
> You already hit your limit. We can stop here. Done. End of. ☠️
>
> 🤔But let's continue for funzies😋.
>
> Assuming defaults:
>
> Default ring_size = 64 / 9 nodes ~ 7 virtual nodes per physical node.
>
> Default leveldb ram allocation = 70%
>
> Leveldb operates, aka consumes resources including ram, on a vnode basis.
> It likes to consume ram on the order of 300MB through 2.5GB per vnode,
> increasing in performance till it caps. Even if you did switch everything
> to level you'd still be redlined.
>
> Bottom line is that bitcask, leveldb and your OS are fighting for ram all
> day 'ery day😡. Why you hate them and make them fight like that?😩 Not
> nice! (Trumpisms!)🤓
>
> -Alexander
>
> ps. You probably want to bump to 128 ring size. More vnodes equals more
> parallelism, but also means more resource consumption. You prob want min 8
> (v)CPU and 16GB min ram. YMMV, check my math.
>
> pps. If you don't want to double your per VM cost (aws ec2, etc) you could
> add nodes to the cluster. Because Riak uniformly distributes data around
> the cluster adding nodes increase total resources to the cluster, reduces
> number of objects allocated to each node. The converse is also true, if you
> double your node size you could halve your node count. That said, systems
> like Riak like prefer more nodes. It's just a math game.
>
> @siculars
> http://siculars.posthaven.com
>
> Sent from my iRotaryPhone
>
> On Nov 22, 2016, at 08:51, Daniel Miller wrote:
>
> Hi Alexander,
>
> Thanks for responding.
>
> > How many nodes?
>
> We currently have 9 nodes in our cluster.
>
> > How much ram per node?
>
> Each node has 4GB of ram and 4GB of swap. The memory levels (ram + swap)
> on each node are currently between 4GB and 5.5GB.
>
> > How many objects (files)? What is the average file size?
>
> We currently have >30 million objects, and I analyzed the average object
> size before we migrated data into the cluster it was about 4KB/object, with
> some objects being much larger (multiple MB). Is there an easy way to get
> this information from a running cluster so I can give you more accurate
> information?
>
>
> On Tue, Nov 22, 2016 at 2:42 AM, Alexander Sicular
> wrote
