The problem was likely more with the fact that it can’t flush in 128ms so you backup on flush
-- Jeff Jirsa > On Mar 14, 2018, at 12:07 PM, Faraz Mateen <fmat...@an10.io> wrote: > > I was able to overcome the timeout error by setting > memtable_flush_period_in_ms to 0 for all my tables. Initially it was set to > 128. > Now I able to write ~40000 records/min in cassandra and the script has been > running for around 12 hours now. > > However, I am still curious that why was cassandra unable to hold data in the > memory for 128 ms considering that I have 30 GB of RAM for each node. > >> On Wed, Mar 14, 2018 at 2:24 PM, Faraz Mateen <fmat...@an10.io> wrote: >> Thanks for the response. >> >> Here is the output of "DESCRIBE" on my table >> >> https://gist.github.com/farazmateen/1c88f6ae4fb0b9f1619a2a1b28ae58c4 >> >> I am getting two errors from the python script that I mentioned above. First >> one does not show any error or exception in server logs. Second error: >> >> "cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request timeout. >> See Session.execute[_async](timeout)'}, last_host=10.128.1.1" >> >> shows JAVA HEAP Exception in server logs. You can look at the exception here: >> >> https://gist.githubusercontent.com/farazmateen/e7aa5749f963ad2293f8be0ca1ccdc22/raw/e3fd274af32c20eb9f534849a31734dcd33745b4/JVM-HEAP-EXCEPTION.txt >> >> My python code snippet can be viewed at the following link: >> https://gist.github.com/farazmateen/02be8bb59cdb205d6a35e8e3f93e27d5 >> >> Here are timeout related arguments from (/etc/cassandra/cassandra.yaml) >> >> read_request_timeout_in_ms: 5000 >> range_request_timeout_in_ms: 10000 >> write_request_timeout_in_ms: 10000 >> counter_write_request_timeout_in_ms: 5000 >> cas_contention_timeout_in_ms: 1000 >> truncate_request_timeout_in_ms: 60000 >> request_timeout_in_ms: 10000 >> cross_node_timeout: false >> >> >>> On Wed, Mar 14, 2018 at 4:22 AM, Bruce Tietjen >>> <bruce.tiet...@imatsolutions.com> wrote: >>> The following won't address any server performance issues, but will allow >>> your application to continue to run even if there are client or server >>> timeouts: >>> >>> Your python code should wrap all Cassandra statement execution calls in >>> a try/except block to catch any errors and handle them appropriately. >>> For timeouts, you might consider re-trying the statement. >>> >>> You may also want to consider proactively setting your client and/or >>> server timeouts so your application sees fewer failures. >>> >>> >>> Any production code should include proper error handling and during initial >>> development and testing, it may be helpful to allow your application to >>> continue running >>> so you get a better idea of if or when different timeouts occur. >>> >>> see: >>> cassandra.Timeout >>> cassandra.WriteTimeout >>> cassandra.ReadTimeout >>> >>> also: >>> https://datastax.github.io/python-driver/api/cassandra.html >>> >>> >>> >>> >>> >>>> On Tue, Mar 13, 2018 at 5:17 PM, Goutham reddy >>>> <goutham.chiru...@gmail.com> wrote: >>>> Faraz, >>>> Can you share your code snippet, how you are trying to save the entity >>>> objects into cassandra. >>>> >>>> Thanks and Regards, >>>> Goutham Reddy Aenugu. >>>> >>>> Regards >>>> Goutham Reddy >>>> >>>>> On Tue, Mar 13, 2018 at 3:42 PM, Faraz Mateen <fmat...@an10.io> wrote: >>>>> Hi everyone, >>>>> >>>>> I seem to have hit a problem in which writing to cassandra through a >>>>> python script fails and also occasionally causes cassandra node to crash. >>>>> Here are the details of my problem. >>>>> >>>>> I have a python based streaming application that reads data from kafka at >>>>> a high rate and pushes it to cassandra through datastax's cassandra >>>>> driver for python. My cassandra setup consists of 3 nodes and a >>>>> replication factor of 2. Problem is that my python application crashes >>>>> after writing ~12000 records with the following error: >>>>> >>>>> Exception: Error from server: code=1100 [Coordinator node timed out >>>>> waiting for replica nodes' responses] message="Operation timed out - >>>>> received only 0 responses." info={'received_responses': >>>>> 0, 'consistency': 'LOCAL_ONE', 'required_responses': 1} >>>>> Sometimes the python application crashes with this traceback: >>>>> >>>>> cassandra.OperationTimedOut: errors={'10.128.1.1': 'Client request >>>>> timeout. See Session.execute[_async](timeout)'}, last_host=10.128.1.1 >>>>> With the error above, one of the cassandra node crashes as well. When I >>>>> look at cassandra system logs (/var/log/cassandra/system.log), I see the >>>>> following exception: >>>>> >>>>> https://gist.github.com/farazmateen/e7aa5749f963ad2293f8be0ca1ccdc22/e3fd274af32c20eb9f534849a31734dcd33745b4 >>>>> >>>>> According to the suggestion in post linked below, I have set my JVM Heap >>>>> size to 8GB but the problem still persists.: >>>>> https://dzone.com/articles/diagnosing-and-fixing-cassandra-timeouts >>>>> >>>>> Cluster: >>>>> Cassandra version 3.9 >>>>> 3 nodes, with 8 cores and 30GB of RAM each. >>>>> Keyspace has a replication factor of 2. >>>>> Write consistency is LOCAL_ONE >>>>> MAX HEAP SIZE is set to 8GB. >>>>> Any help will be greatly appreciated. >>>>> >>>>> -- >>>>> Faraz >>>> >>> >> >> >> >> -- >> Faraz Mateen > > > > -- > Faraz Mateen
