The following won't address any server performance issues, but will allow
your application to continue to run even if there are client or server
timeouts:
Your python code should wrap all Cassandra statement execution calls in
a try/except block to catch any errors and handle them appropriately.
For timeouts, you might consider re-trying the statement.
You may also want to consider proactively setting your client and/or
server timeouts so your application sees fewer failures.
Any production code should include proper error handling and during initial
development and testing, it may be helpful to allow your application to
continue running
so you get a better idea of if or when different timeouts occur.
see:
cassandra.Timeout
cassandra.WriteTimeout
cassandra.ReadTimeout
also:
https://datastax.github.io/python-driver/api/cassandra.html
On Tue, Mar 13, 2018 at 5:17 PM, Goutham reddy <goutham.chiru...@gmail.com>
wrote:
> Faraz,
> Can you share your code snippet, how you are trying to save the entity
> objects into cassandra.
>
> Thanks and Regards,
> Goutham Reddy Aenugu.
>
> Regards
> Goutham Reddy
>
> On Tue, Mar 13, 2018 at 3:42 PM, Faraz Mateen <fmat...@an10.io> wrote:
>
>> Hi everyone,
>>
>> I seem to have hit a problem in which writing to cassandra through a
>> python script fails and also occasionally causes cassandra node to crash.
>> Here are the details of my problem.
>>
>> I have a python based streaming application that reads data from kafka at
>> a high rate and pushes it to cassandra through datastax's cassandra driver
>> for python. My cassandra setup consists of 3 nodes and a replication factor
>> of 2. Problem is that my python application crashes after writing ~12000
>> records with the following error:
>>
>> Exception: Error from server: code=1100 [Coordinator node timed out waiting
>> for replica nodes' responses] message="Operation timed out - received only 0
>> responses." info={'received_responses':
>> 0, 'consistency': 'LOCAL_ONE', 'required_responses': 1}
>>
>> Sometimes the python application crashes with this traceback:
>>
>> cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request timeout.
>> See Session.execute[_async](timeout)'}, last_host=10.128.1.1
>>
>> With the error above, one of the cassandra node crashes as well. When I
>> look at cassandra system logs (/var/log/cassandra/system.log), I see the
>> following exception:
>>
>> https://gist.github.com/farazmateen/e7aa5749f963ad2293f8be0c
>> a1ccdc22/e3fd274af32c20eb9f534849a31734dcd33745b4
>>
>> According to the suggestion in post linked below, I have set my JVM Heap
>> size to 8GB but the problem still persists.:
>> https://dzone.com/articles/diagnosing-and-fixing-cassandra-timeouts
>>
>> *Cluster:*
>>
>> - Cassandra version 3.9
>> - 3 nodes, with 8 cores and 30GB of RAM each.
>> - Keyspace has a replication factor of 2.
>> - Write consistency is LOCAL_ONE
>> - MAX HEAP SIZE is set to 8GB.
>>
>> Any help will be greatly appreciated.
>>
>> --
>> Faraz
>>
>
>
