Good day, Yesterday we went through the exercise of doubling the size of our initial 3 node cluster. The rebalancing took four hours or so. Each node now has between 7 and 8 GB of data stored in the Leveldb backend with an n_val of 3.
During rebalancing the cluster became nearly useless. The CPUs where largely pegged, and IO was hit hard, as could be expected. Op throughput dropped to the floor and services depending on Riak become unresponsive, as they waited on Riak ops and then failed with timeouts. This was in EC2 m1.large instances with ephemeral drives. A few questions. Is this expected behavior? Granted, these are EC2 boxes and Leveldb depends heavily on disk, but I can't imagine folks using Riak on production his this type of performance hit resulting from rebalancing. Are client ops prioritized higher than the rebalancing work? If not, why not? I recall there was some mention that by using Riak EDS it was possible to fill in a new node slowly, and then have it join the cluster when its nearly in sync, thus limiting the stampede effect that regular rebalancing seems to bring on. Is that the case? If so, is there any documentation on this? I am also starting to suspect the timing statistics that Riak reports. We've been charting them, and comparing them before and after the doubling of the cluster size shows some odd results. Nearly all stats in the older nodes (mean, median, 95 percentile, 99 percentile, 100 percentile) are about the same before and after the addition of new nodes. This is the case even though there are now double as many nodes to handle requests, and each node has about half as much data as before. The only one that did change was the puts 100 percentile. It increased during the rebalancing, and then stayed at its higher value. How is this possible? In fact, I find the 100 percentile stat a bit incredible. One of the boxes is showing a 8,012 seconds 100 percentile stat (yes, I did divide by one million to convert from microseconds). Or are the time stats not rolling stats? Are they computed since the start of the node, meaning that the 100 percentile will never go down, only up? Elias
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
