The node(s) you are connecting to have disappeared, look at the server side logs for errors. It's probably running out of memory, if so turn off things like the key cache and key_cache_save_period until you have a stable system, then gradually turn them back on.
You may also want to have a read of this http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
As for the throughput, it depends on the HW, how you are doing the inserts, how many clients you have, how big the columns are and a bunch of other things. But 2GB RAM and 1GB Heap is at the low end of the scale.
Hope that helps.
Aaron
We are making some tests using 3 nodes: A, B, C. We are bulk inserting
87500 keys, and for each of them 1 super column with 768 columns.
We are using hector 0.7.0-18 to insert the data, and at some point an
exception is raised, and sometimes cassandra deamon stop running in
one of the nodes.
The nodes has 2gb of RAM, so the JVM heap is 1gb, and the CPU load
goes up to 80%.
It is possible to insert that ammount of data every 10 minutes? That
would be our use case scenario We are newbies in cassandra, so maybe
we must take a different approach. What do you suggest?
The keyspace configuration is:
- name: TestKeyspace
replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
replication_factor: 3
column_families:
- name: TestFamily
column_type: Super
compare_with: LongType
compare_subcolumns_with: UTF8Type
keys_cached: 200000
rows_cached: 0
key_cache_save_period_in_seconds: 3600
row_cache_save_period_in_seconds: 0
memtable_flush_after_mins: 3600
memtable_throughput_in_mb: 80
memtable_operations_in_millions: 0.10
the seeds configuration for each node in each machine:
seeds:
- A
and the exception that is raised is:
Exception in thread "main"
me.prettyprint.hector.api.exceptions.HUnavailableException:
UnavailableException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:36)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:88)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:81)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(FailoverOperator.java:388)
at me.prettyprint.cassandra.service.FailoverOperator.operateSingleIteration(FailoverOperator.java:194)
at me.prettyprint.cassandra.service.FailoverOperator.operate(FailoverOperator.java:99)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:123)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:93)
at me.prettyprintcassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:99)
at me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:142)
at me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:139)
at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:58)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:139)
at com.popego.benchmarks.InsertSegmentsHits.batchInsert(InsertSegmentsHits.java:154)
at com.popego.benchmarks.InsertSegmentsHits.insertData(InsertSegmentsHits.java:131)
at com.popego.benchmarks.InsertSegmentsHits.main(InsertSegmentsHits.java:177)
Caused by: UnavailableException()
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:16633)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:935)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:909)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:86)
... 15 more
