HI Bo; you raised 2 questions: 20% system utilization Hints
20% system utilization: For a node or a cluster to have 20% utilization is Normal during peak write operation. Hints: hints are written when a node is unreachable; C* 3.0 has a complete over haul in the way hints have been implemented. Recommend reading up this blog article: http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery hope this helps Jan/ -------------------------------------------- On Thu, 4/21/16, Jens Rantil <jens.ran...@tink.se> wrote: Subject: Re: When are hints written? To: user@cassandra.apache.org Date: Thursday, April 21, 2016, 8:57 AM Hi again Bo, I assume this is the piece of documentation you are referring to? http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance > If a replica node is overloaded or unavailable, and the failure detector has not yet marked it down, then expect most or all writes to that node to fail after the timeout triggered by write_request_timeout_in_ms, which defaults to 10 seconds. During that time, Cassandra writes the hint when the timeout is reached. I'm not an expert on this, but the way I've seen is that hints are written stored as soon as there is _any_ issues writing a mutation (insert/update/delete) to a node. By "issue", that essentially means that a node hasn't acknowledged back to the coordinator that the write succeeded within write_request_timeout_in_ms. This includes TCP/socket timeouts, connection issues or that the node is down. The hints are stored for a maximum timespan defaulting to 3 hours. Cheers, Jens On Thu, Apr 21, 2016 at 8:06 AM Bo Finnerup Madsen <bo.gunder...@gmail.com> wrote: Hi Jens, Thank you for the tip!ALL would definitely cure our hints issue, but as you note, it is not optimal as we are unable to take down nodes without clients failing. I am most probably overlooking something in the documentation, but I cannot see any description of when hints are written other than when a node is marked as being down. And since none of our nodes have been marked as being down (at least according to the logs), I suspect that there is some timeout that governs when hints are written? Regarding your other post: Yes, 3.0.3 is pretty new. But we are new to this cassandra game, and our schema-fu is not strong enough for us to create a schema without using materialized views :) ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil <jens.ran...@tink.se>: Hi Bo, > In our case, I would like for the cluster to wait for the write to be persisted on the relevant nodes before returning an ok to the client. But I don't know which knobs to turn to accomplish this? or if it is even possible :) This is what write consistency option is for. Have a look at https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html. Note, however that if you use ALL, your clients will fail (throw exception, depending on language) as soon as a single partition can't be written. This means you can't do online maintenance of a Cassandra node (such as upgrading it etc.) without experiencing write issues. Cheers,Jens On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen <bo.gunder...@gmail.com> wrote: Hi, We have a small 5 node cluster of m4.xlarge clients that receives writes from ~20 clients. The clients will write as fast as they can, and the whole process is limited by the write performance of the cassandra cluster.After we have tweaked our schema to avoid large partitions, the load is going ok and we don't see any warnings or errors in the cassandra logs. But we do see quite a lot of hint handoff activity. During the load, the cassandra nodes are quite loaded, with linux reporting a load as high as 20. I have read the available documentation on how hints works, and to my understanding hints should only be written if a node is down. But as far as I can see, none of the nodes are marked as down during the load. So I suspect I am missing something :)We have configured the servers with write_request_timeout_in_ms: 120000 and the clients with a timeout of 130000, but still get hints stored. In our case, I would like for the cluster to wait for the write to be persisted on the relevant nodes before returning an ok to the client. But I don't know which knobs to turn to accomplish this? or if it is even possible :) We are running cassandra 3.0.3, with 8Gb heap and a replication factor of 3. Thank you in advance! Yours sincerely, Bo Madsen -- Jens Rantil Backend Developer @ Tink Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden For urgent matters you can reach me at +46-708-84 18 32. -- Jens Rantil Backend Developer @ Tink Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden For urgent matters you can reach me at +46-708-84 18 32.
