Everyone really should move off of the 2.x versions just like you are doing.
Tell me about it... But since there are a lot of groups involved, these things take time unfortunately. Thanks for your assistance Kenneth On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman <kenbrot...@yahoo.com.invalid> wrote: > Since you are in the process of upgrading, I’d do nothing on the settings > right now. But if you wanted to do something on the settings in the > meantime, based on my read of the information available, I’d maybe double > the default settings. The upgrade will help a lot of things as you know. > > > > Everyone really should move off of the 2.x versions just like you are > doing. > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 12:34 PM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > See my comments inline. > > > > Do the 8 nodes clusters have the problem too? > > Yes > > > > To the same extent? > > It depends on the throughput, but basically the smaller clusters get low > throughput, so the problem is naturally smaller. > > > > Is it any cluster across multi-DC’s? > > Yes > > > > Do all the clusters use nodes with similar specs? > > All nodes have similar specs within a cluster but different specs on > different clusters. > > > > The version of Cassandra you are on can make a difference. What version > are you on? > > Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the > process of upgrading to 3.11.4 > > > > Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video > at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions > you can get into trouble if you go to fast or two slow? > > I guess you can say it about almost any parameter you change :) > > > > BTW, I thought the comments at the end of the article you mentioned were > really good. > > The entire article is very good, but I wonder if it's still valid since it > was created around 4 years ago. > > > > Thanks! > > > > > > > > > > On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman <kenbrotman@yahoocom.invalid> > wrote: > > Makes sense If you have time and don’t mind, could you answer the > following: > > Do the 8 nodes clusters have the problem too? > > To the same extent? > > Is it just the clusters with the large node count? > > Is it any cluster across multi-DC’s? > > Do all the clusters use nodes with similar specs? > > > > The version of Cassandra you are on can make a difference. What version > are you on? > > > > Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video > at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions > you can get into trouble if you go to fast or two slow? > > BTW, I thought the comments at the end of the article you mentioned were > really good. > > > > > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 11:04 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > It varies... > > Some clusters have 48 nodes, others 24 nodes and some 8 nodes. > > Both settings are on default. > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > That's the plan, but I thought I might first get some valuable information > from someone in the community that has already experienced in this type of > change. > > > > Thanks! > > > > On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman > <kenbrot...@yahoo.com.invalid> wrote: > > It sounds like your use case might be appropriate for tuning those two > settings some. > > > > How many nodes are in the cluster? > > Are both settings definitely on the default values currently? > > > > I’d try making a single conservative change to one or the other, measure > and reassess. Then do same to other setting. > > > > Then of course share your results with us. > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 9:54 AM > *To:* user@cassandra.apache.org > *Subject:* Re: A Question About Hints > > > > Hi Kenneth, > > > > The concern is that in some cases, hints accumulate on nodes, and it takes > a while until they are delivered (multi DCs). > > I see that whenever there are a lot of hints in play,like after a rolling > restart, the cluster works harder. That's why I want to decrease the hints > delivery time. > > I didn't want to change the configuration blindly and thought the > community might have some experience on this subject. > > > > I went over the cassandra.yaml file but didn't find any information on > optimizing these attributes, just that the max_throttle is divided between > nodes in the cluster and that I should increase the > max_hints_delivery_threads because I have multi-dc deployments. > > > > # Maximum throttle in KBs per second, per delivery thread. This will be > # reduced proportionally to the number of nodes in the cluster. (If there > # are two nodes in the cluster, each delivery thread will use the maximum > # rate; if there are three, each will throttle to half of the maximum, > # since we expect two nodes to be delivering hints simultaneously) > hinted_handoff_throttle_in_kb: 1024 > > # Number of threads with which to deliver hints; > # Consider increasing this number when you have multi-dc deployments, since > # cross-dc handoff tends to be slower > max_hints_delivery_threads: 2 > > > > > > Thanks for your help! > > > > > > On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman > <kenbrot...@yahoo.com.invalid> wrote: > > What is the concern? Why are you looking there? The casssandra.yml file > has some notes about it. Did you read them? > > > > *From:* shalom sagges [mailto:shalomsag...@gmail.com] > *Sent:* Monday, March 04, 2019 7:22 AM > *To:* user@cassandra.apache.org > *Subject:* A Question About Hints > > > > Hi All, > > > > Does anyone know what is the most optimal hints configuration (multiple > DCs) in terms of > > max_hints_delivery_threads and hinted_handoff_throttle_in_kb? > > If it's different for various use cases, is there a rule of thumb I can > work with? > > > > I found this post but it's quite old: > > http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ > > > > Thanks! > >
