Did the images have EBS storage or Instance Store storage? Typically EBS volumes aren't the best to be benchmarking against: http://www.mail-archive.com/user@cassandra.apache.org/msg11022.html
Joaquin Casares DataStax Software Engineer/Support On Wed, Apr 20, 2011 at 5:12 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > A few months ago I was seeing 12k writes/s on a single EC2 XL. So > something is wrong. > > My first suspicion is that your client node may be the bottleneck. > > On Wed, Apr 20, 2011 at 2:56 PM, Alex Araujo > <cassandra-us...@alex.otherinbox.com> wrote: > > Does anyone have any Ec2 benchmarks/experiences they can share? I am > trying > > to get a sense for what to expect from a production cluster on Ec2 so > that I > > can compare my application's performance against a sane baseline. What I > > have done so far is: > > > > 1. Lunched a 4 node cluster of m1.xlarge instances in the same > availability > > zone using PyStratus (https://github.com/digitalreasoning/PyStratus). > Each > > node has the following specs (according to Amazon): > > 15 GB memory > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) > > 1,690 GB instance storage > > 64-bit platform > > > > 2. Changed the default PyStratus directories in order to have commit logs > on > > the root partition and data files on ephemeral storage: > > commitlog_directory: /var/cassandra-logs > > data_file_directories: [/mnt/cassandra-data] > > > > 2. Gave each node 10GB of MAX_HEAP; 1GB HEAP_NEWSIZE in > > conf/cassandra-env.sh > > > > 3. Ran `contrib/stress/bin/stress -d node1,..,node4 -n 10000000 -t 100` > on a > > separate m1.large instance: > > total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time > > ... > > 9832712,7120,7120,0.004948514851485148,842 > > 9907616,7490,7490,0.0043189949802413755,852 > > 9978357,7074,7074,0.004560353967289125,863 > > 10000000,2164,2164,0.004065933558194335,867 > > > > 4. Truncated Keyspace1.Standard1: > > # /usr/local/apache-cassandra/bin/cassandra-cli -host localhost -port > 9160 > > Connected to: "Test Cluster" on x.x.x.x/9160 > > Welcome to cassandra CLI. > > > > Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit. > > [default@unknown] use Keyspace1; > > Authenticated to keyspace: Keyspace1 > > [default@Keyspace1] truncate Standard1; > > null > > > > 5. Expanded the cluster to 8 nodes using PyStratus and sanity checked > using > > nodetool: > > # /usr/local/apache-cassandra/bin/nodetool -h localhost ring > > Address Status State Load Owns > > Token > > x.x.x.x Up Normal 1.3 GB 12.50% > > 21267647932558653966460912964485513216 > > x.x.x.x Up Normal 3.06 GB 12.50% > > 42535295865117307932921825928971026432 > > x.x.x.x Up Normal 1.16 GB 12.50% > > 63802943797675961899382738893456539648 > > x.x.x.x Up Normal 2.43 GB 12.50% > > 85070591730234615865843651857942052864 > > x.x.x.x Up Normal 1.22 GB 12.50% > > 106338239662793269832304564822427566080 > > x.x.x.x Up Normal 2.74 GB 12.50% > > 127605887595351923798765477786913079296 > > x.x.x.x Up Normal 1.22 GB 12.50% > > 148873535527910577765226390751398592512 > > x.x.x.x Up Normal 2.57 GB 12.50% > > 170141183460469231731687303715884105728 > > > > 6. Ran `contrib/stress/bin/stress -d node1,..,node8 -n 10000000 -t 100` > on a > > separate m1.large instance again: > > total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time > > ... > > 9880360,9649,9649,0.003210443956226165,720 > > 9942718,6235,6235,0.003206934154398794,731 > > 9997035,5431,5431,0.0032615939761032457,741 > > 10000000,296,296,0.002660033726812816,742 > > > > In a nutshell, 4 nodes inserted at 11,534 writes/sec and 8 nodes inserted > at > > 13,477 writes/sec. > > > > Those numbers seem a little low to me, but I don't have anything to > compare > > to. I'd like to hear others' opinions before I spin my wheels with with > > number of nodes, threads, memtable, memory, and/or GC settings. Cheers, > > Alex. > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
