Hi everyone,
I seem to have hit a problem in which writing to cassandra through a python
script fails and also occasionally causes cassandra node to crash. Here are
the details of my problem.
I have a python based streaming application that reads data from kafka at a
high rate and pushes it to cassandra through datastax's cassandra driver
for python. My cassandra setup consists of 3 nodes and a replication factor
of 2. Problem is that my python application crashes after writing ~12000
records with the following error:
Exception: Error from server: code=1100 [Coordinator node timed out
waiting for replica nodes' responses] message="Operation timed out -
received only 0 responses." info={'received_responses':
0, 'consistency': 'LOCAL_ONE', 'required_responses': 1}
Sometimes the python application crashes with this traceback:
cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request
timeout. See Session.execute[_async](timeout)'}, last_host=10.128.1.1
With the error above, one of the cassandra node crashes as well. When I
look at cassandra system logs (/var/log/cassandra/system.log), I see the
following exception:
https://gist.github.com/farazmateen/e7aa5749f963ad2293f8be0ca1ccdc22/
e3fd274af32c20eb9f534849a31734dcd33745b4
According to the suggestion in post linked below, I have set my JVM Heap
size to 8GB but the problem still persists.:
https://dzone.com/articles/diagnosing-and-fixing-cassandra-timeouts
*Cluster:*
- Cassandra version 3.9
- 3 nodes, with 8 cores and 30GB of RAM each.
- Keyspace has a replication factor of 2.
- Write consistency is LOCAL_ONE
- MAX HEAP SIZE is set to 8GB.
Any help will be greatly appreciated.
--
Faraz
