Hi Jan, I don't think I expressed myself clearly :) The load I mention is as reported by linux. It is my understanding that a load of 20 means that the computer have enough work to saturate 20 CPU's. Seeing as we only have 2 vCPUs in our AWS machines, it is quite a bit more than 20% :)
Regarding that blog, I have read it...several times by now :) Unfortunately it does not answer the general question of exactly when a hint is written. fre. 22. apr. 2016 kl. 04.33 skrev Jan <cne...@yahoo.com>: > HI Bo; > > you raised 2 questions: > 20% system utilization > Hints > > 20% system utilization: For a node or a cluster to have 20% utilization > is Normal during peak write operation. > Hints: hints are written when a node is unreachable; C* 3.0 has > a complete over haul in the way hints have been implemented. > > Recommend reading up this blog article: > > http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery > > hope this helps > Jan/ > > > -------------------------------------------- > On Thu, 4/21/16, Jens Rantil <jens.ran...@tink.se> wrote: > > Subject: Re: When are hints written? > To: user@cassandra.apache.org > Date: Thursday, April 21, 2016, 8:57 AM > > Hi again > Bo, > I assume this is the piece of > documentation you are referring to? > http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance > > > If a > replica node is overloaded or unavailable, and the failure > detector has not yet marked it down, then expect most or all > writes to that node to fail after the timeout triggered by > write_request_timeout_in_ms, > which defaults to 10 seconds. During that time, Cassandra > writes the hint when the timeout is reached. > I'm not an expert on this, but > the way I've seen is that hints are written stored as > soon as there is _any_ issues writing a mutation > (insert/update/delete) to a node. By "issue", that > essentially means that a node hasn't acknowledged back > to the coordinator that the write succeeded within > write_request_timeout_in_ms. This includes TCP/socket > timeouts, connection issues or that the node is down. The > hints are stored for a maximum timespan defaulting to 3 > hours. > > Cheers, > Jens > On Thu, Apr > 21, 2016 at 8:06 AM Bo Finnerup Madsen <bo.gunder...@gmail.com> > wrote: > Hi Jens, > Thank you > for the tip!ALL would definitely cure our hints > issue, but as you note, it is not optimal as we are unable > to take down nodes without clients failing. > I am most probably overlooking > something in the documentation, but I cannot see any > description of when hints are written other than when a node > is marked as being down. And since none of our nodes have > been marked as being down (at least according to the logs), > I suspect that there is some timeout that governs when hints > are written? > Regarding > your other post: Yes, 3.0.3 is pretty new. But we are new to > this cassandra game, and our schema-fu is not strong enough > for us to create a schema without using materialized views > :) > > ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil > <jens.ran...@tink.se>: > Hi Bo, > > In our case, I would like for the > cluster to wait for the write to be persisted on the > relevant nodes before returning an ok to the client. But I don't know > which > knobs to turn to accomplish this? or if it is even possible > :) > This is what write consistency > option is for. Have a look at > https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html > . > Note, however that if you use ALL, your clients will fail > (throw exception, depending on language) as soon as a single > partition can't be written. This means you can't do > online maintenance of a Cassandra node (such as upgrading it > etc.) without experiencing write issues. > Cheers,Jens > On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen > <bo.gunder...@gmail.com> > wrote: > Hi, > We have a > small 5 node cluster of m4.xlarge clients that receives > writes from ~20 clients. The clients will write as fast as > they can, and the whole process is limited by the write > performance of the cassandra cluster.After we have tweaked our schema to > avoid large partitions, the load is going ok and we > don't see any warnings or errors in the cassandra logs. > But we do see quite a lot of hint handoff activity. During > the load, the cassandra nodes are quite loaded, with linux > reporting a load as high as 20. > I have read the available > documentation on how hints works, and to my understanding > hints should only be written if a node is down. But as far > as I can see, none of the nodes are marked as down during > the load. So I suspect I am missing something > :)We have configured the servers > with write_request_timeout_in_ms: 120000 and the clients > with a timeout of 130000, but still get hints > stored. > In our case, I > would like for the cluster to wait for the write to be > persisted on the relevant nodes before returning an ok to > the client. But I don't know which knobs to turn to > accomplish this? or if it is even possible :) > We are running cassandra 3.0.3, with > 8Gb heap and a replication factor of 3. > Thank you in advance! > Yours sincerely, Bo > Madsen > -- > > > > > > > > > Jens Rantil > Backend Developer @ Tink > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden > For urgent matters you can reach me at > +46-708-84 18 32. > -- > > > > > > > > > Jens Rantil > Backend Developer @ Tink > Tink AB, Wallingatan 5, 111 60 > Stockholm, Sweden > For urgent matters you can > reach me at +46-708-84 18 32. >
