Thanks for the response. Here is the output of "DESCRIBE" on my table
https://gist.github.com/farazmateen/1c88f6ae4fb0b9f1619a2a1b28ae58c4 I am getting two errors from the python script that I mentioned above. First one does not show any error or exception in server logs. Second error: *"cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=10.128.1.1"* shows JAVA HEAP Exception in server logs. You can look at the exception here: https://gist.githubusercontent.com/farazmateen/ e7aa5749f963ad2293f8be0ca1ccdc22/raw/e3fd274af32c20eb9f534849a31734 dcd33745b4/JVM-HEAP-EXCEPTION.txt My python code snippet can be viewed at the following link: https://gist.github.com/farazmateen/02be8bb59cdb205d6a35e8e3f93e27d5 <https://gist.github.com/farazmateen/02be8bb59cdb205d6a35e8e3f93e27d5> H <https://gist.github.com/farazmateen/02be8bb59cdb205d6a35e8e3f93e27d5>ere are timeout related arguments from (*/etc/cassandra/cassandra.yaml*) read_request_timeout_in_ms: 5000 range_request_timeout_in_ms: 10000 write_request_timeout_in_ms: 10000 counter_write_request_timeout_in_ms: 5000 cas_contention_timeout_in_ms: 1000 truncate_request_timeout_in_ms: 60000 request_timeout_in_ms: 10000 cross_node_timeout: false On Wed, Mar 14, 2018 at 4:22 AM, Bruce Tietjen <bruce.tietjen@imatsolutions. com> wrote: > The following won't address any server performance issues, but will allow > your application to continue to run even if there are client or server > timeouts: > > Your python code should wrap all Cassandra statement execution calls > in a try/except block to catch any errors and handle them appropriately. > For timeouts, you might consider re-trying the statement. > > You may also want to consider proactively setting your client and/or > server timeouts so your application sees fewer failures. > > > Any production code should include proper error handling and during > initial development and testing, it may be helpful to allow your > application to continue running > so you get a better idea of if or when different timeouts occur. > > see: > cassandra.Timeout > cassandra.WriteTimeout > cassandra.ReadTimeout > > also: > https://datastax.github.io/python-driver/api/cassandra.html > > > > > > On Tue, Mar 13, 2018 at 5:17 PM, Goutham reddy <goutham.chiru...@gmail.com > > wrote: > >> Faraz, >> Can you share your code snippet, how you are trying to save the entity >> objects into cassandra. >> >> Thanks and Regards, >> Goutham Reddy Aenugu. >> >> Regards >> Goutham Reddy >> >> On Tue, Mar 13, 2018 at 3:42 PM, Faraz Mateen <fmat...@an10.io> wrote: >> >>> Hi everyone, >>> >>> I seem to have hit a problem in which writing to cassandra through a >>> python script fails and also occasionally causes cassandra node to crash. >>> Here are the details of my problem. >>> >>> I have a python based streaming application that reads data from kafka >>> at a high rate and pushes it to cassandra through datastax's cassandra >>> driver for python. My cassandra setup consists of 3 nodes and a replication >>> factor of 2. Problem is that my python application crashes after writing >>> ~12000 records with the following error: >>> >>> Exception: Error from server: code=1100 [Coordinator node timed out waiting >>> for replica nodes' responses] message="Operation timed out - received only >>> 0 responses." info={'received_responses': >>> 0, 'consistency': 'LOCAL_ONE', 'required_responses': 1} >>> >>> Sometimes the python application crashes with this traceback: >>> >>> cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request timeout. >>> See Session.execute[_async](timeout)'}, last_host=10.128.1.1 >>> >>> With the error above, one of the cassandra node crashes as well. When I >>> look at cassandra system logs (/var/log/cassandra/system.log), I see >>> the following exception: >>> >>> https://gist.github.com/farazmateen/e7aa5749f963ad2293f8be0c >>> a1ccdc22/e3fd274af32c20eb9f534849a31734dcd33745b4 >>> >>> According to the suggestion in post linked below, I have set my JVM Heap >>> size to 8GB but the problem still persists.: >>> https://dzone.com/articles/diagnosing-and-fixing-cassandra-timeouts >>> >>> *Cluster:* >>> >>> - Cassandra version 3.9 >>> - 3 nodes, with 8 cores and 30GB of RAM each. >>> - Keyspace has a replication factor of 2. >>> - Write consistency is LOCAL_ONE >>> - MAX HEAP SIZE is set to 8GB. >>> >>> Any help will be greatly appreciated. >>> >>> -- >>> Faraz >>> >> >> > -- Faraz Mateen
