Hi,
If you use SizeTieredCompactionStrategy, you should have x2 disk space
to be on the safe side. So if you want to store 2TB data, you need
partition size of 4TB at least. LeveledCompactionStrategy is available
in 1.x and supposed to require less free disk space (but comes at price
of I/O).
--
Rustam.
On 12/03/2012 09:23, Vanger wrote:
*We have cassandra 4 nodes cluster* with RF = 3 (nodes named from 'A'
to 'D', initial tokens:
*A (25%)*: 20543402371996174596346065790779111550, *
B (25%)*: 63454860067234500516210522518260948578,
*C (25%)*: 106715317233367107622067286720208938865,
*D (25%)*: 150141183460469231731687303715884105728),
*and want to add 5th node* ('E') with initial token =
164163260474281062972548100673162157075, then we want to rebalance A,
D, E nodes such way they'll own equal percentage of data. All nodes
have ~400 GB of data and around ~300GB disk free space.
What we did:
1. 'Join' new cassandra instance (node 'E') to cluster and wait 'till
it loads data for it tokens range.
2. Move node 'D' initial token down from 150... to 130...
Here we ran into a problem. When "move" started disk usage for node C
grows from 400 to 750GB, we saw running compactions on node 'D' but
some compactions failed with /"WARN [CompactionExecutor:580]
2012-03-11 16:57:56,036 CompactionTask.java (line 87) insufficient
space to compact all requested files SSTableReader"/ after that we
killed "move" process to avoid "out of disk space" error (when 5GB of
free space left). After restart it frees 100GB of space and now we
have total of 105GB free disk space on node 'D'. Also we noticed
increased disk usage by ~150GB at node 'B' but it stops growing before
we stopped "move token".
So now we have 5 nodes in cluster in status like this:
Node, Owns%, Load, Init. token
A: 16% 400GB 020...
B: 25% 520GB 063...
C: 25% 400GB 106...
D: 25% 640GB 150...
E: 9% 300GB 164...
We'll add disk space for all nodes and run some cleanups, but there's
still left some questions:
What is the best next step for us from this point?
What is correct procedure after all and what should we expect when
adding node to cassandra cluster?
We expected decrease of used disk space on node 'D' 'cause we shrink
token range for this node, but saw the opposite, why it happened and
is it normal behavior?
What if we'll have 2TB of data on 2.5TB disk and we wanted to add
another node and move tokens?
Is it possible to automate node addition to cluster and be sure we
won't run out of space?
Thank.
