On 5/9/11 9:49 PM, Jonathan Ellis wrote:
On Mon, May 9, 2011 at 5:58 PM, Alex Araujo<cassandra->> How many
replicas are you writing?
Replication factor is 3.
So you're actually spot on the predicted numbers: you're pushing
20k*3=60k "raw" rows/s across your 4 machines.
You might get another 10% or so from increasing memtable thresholds,
but bottom line is you're right around what we'd expect to see.
Furthermore, CPU is the primary bottleneck which is what you want to
see on a pure write workload.
That makes a lot more sense. I upgraded the cluster to 4 m2.4xlarge
instances (68GB of RAM/8 CPU cores) in preparation for application
stress tests and the results were impressive @ 200 threads per client:
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| Server Nodes | Client Nodes | --keep-going | Columns |
Client | Total | Rep Factor | Test Rate | Cluster Rate |
| | | | |
Threads | Threads | | (writes/s) | (writes/s) |
+==============+==============+==============+==============+==============+==============+==============+==============+==============+
| 4 | 3 | N | 10000000 |
200 | 600 | 3 | 44644 | 133931 |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
The issue I'm seeing with app stress tests is that the rate will be
comparable/acceptable at first (~100k w/s) and will degrade considerably
(~48k w/s) until a flush and restart. CPU usage will correspondingly be
high at first (500-700%) and taper down to 50-200%. My data model is
pretty standard (<This> is pseudo-type information):
Users<Column>
"UserId<32CharHash>" : {
"email<String>": "a...@b.com",
"first_name<String>": "John",
"last_name<String>": "Doe"
}
UserGroups<SuperColumn>
"GroupId<UUID>": {
"UserId<32CharHash>": {
"date_joined<DateTime>": "2011-05-10 13:14.789",
"date_left<DateTime>": "2011-05-11 13:14.789",
"active<short>": "0|1"
}
}
UserGroupTimeline<Column>
"GroupId<UUID>": {
"date_joined<TimeUUID>": "UserId<32CharHash>"
}
UserGroupStatus<Column>
"CompositeId('GroupId<UUID>:UserId<32CharHash>')": {
"active<short>": "0|1"
}
Every new User has a row in Users and a ColumnOrSuperColumn in the other
3 CFs (total of 4 operations). One notable difference is that the RAID0
on this instance type (surprisingly) only contains two ephemeral volumes
and appear a bit more saturated in iostat, although not enough to
clearly stand out as the bottleneck. Is the bottleneck in this scenario
likely memtable flush and/or commitlog rotation settings?
RF = 2; ConsistencyLevel = One; -Xmx = 6GB; concurrent_writes: 64; all
other settings are the defaults. Thanks, Alex.
