Hello again, Back to this after a while...
As far as I can tell whenever DC2 is unavailable, there is one node from DC1 that acts as a coordinator. When DC2 is available again, this one node sends the hints to only one node at DC2, which then sends any replicas to the other nodes in the local DC (DC2). This ensures efficient cross-DC bandwidth usage. I was watching "system.hints" on all nodes during this test and this is the conclusion I came to. Two things: 1. If the above is correct, does the same apply when performing anti-entropy repair (without specifying a particular DC)? I'm just hoping the answer to this is going to be YES, otherwise the VPN is not going to be very happy in our case and we would prefer to not saturate it whenever running nodetool repair. I suppose we could have a traffic limiter on the firewalls worst case scenario but I would appreciate your input if you know more on this. 2. As I described earlier, in order to test this I was watching the "system.hints" CF in order to monitor any hints. I was looking to add a Nagios check for this purpose. For that reason I was looking into JMX Concole. I noticed that when a node stores hints, "MBean org.apache.cassandra.db:type=ColumnFamilies,keyspace=system,columnfamily=hints", attribute "MemtableColumnsCount" goes up (although I would expect it to be MemtableRowCount or something?). This attribute will retain its value, until the other node becomes available and ready to receive the hints. I was looking for another attribute somewhere to monitor the active hints. I checked: "MBean org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=hints,name=PendingTasks", "MBean org.apache.cassandra.metrics:type=Storage,name=TotalHints", "MBean org.apache.cassandra.metrics:type=Storage,name=TotalHintsInProgress", "MBean org.apache.cassandra.metrics:type=ThreadPools,path=internal,scope=HintedHandoff,name=ActiveTasks" and even "MBean org.apache.cassandra.metrics:type=HintedHandOffManager,name=Hints_not_stored-/ 10.2.1.100" (this one will never go back to zero). All of them would not increase whenever any hints are being sent (or at least I didn't catch it because it was too fast or whatever?). Does anyone know what all these attributes represent? It looks like there are more specific hint attributes on a per CF basis, but I was looking for a more generic one to begin with. Any help would be much appreciated. Thanks in advance, Vasilis On Wed, Jun 4, 2014 at 1:42 PM, Vasileios Vlachos < vasileiosvlac...@gmail.com> wrote: > Hello Matt, > > nodetool status: > > Datacenter: MAN > =============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Owns (effective) Host ID Token Rack > UN 10.2.1.103 89.34 KB 99.2% b7f8bc93-bf39-475c-a251-8fbe2c7f7239 > -9211685935328163899 RAC1 > UN 10.2.1.102 86.32 KB 0.7% 1f8937e1-9ecb-4e59-896e-6d6ac42dc16d > -3511707179720619260 RAC1 > Datacenter: DER > =============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Owns (effective) Host ID Token Rack > UN 10.2.1.101 75.43 KB 0.2% e71c7ee7-d852-4819-81c0-e993ca87dd5c > -1277931707251349874 RAC1 > UN 10.2.1.100 104.53 KB 99.8% 7333b664-ce2d-40cf-986f-d4b4d4023726 > -9204412570946850701 RAC1 > > I do not know why the cluster is not balanced at the moment, but it holds > almost no data. I will populate it soon and see how that goes. The output > of 'nodetool ring' just lists all the tokens assigned to each individual > node, and as you can imagine it would be pointless to paste it here. I just > did 'nodetool ring | awk ... | unique | wc -l' and it works out to be 1024 > as expected (4 nodes x 256 tokens each). > > Still have not got the answers to the other questions though... > > Thanks, > > Vasilis > > > On Wed, Jun 4, 2014 at 12:28 AM, Matthew Allen <matthew.j.al...@gmail.com> > wrote: > >> Thanks Vasileios. I think I need to make a call as to whether to switch >> to vnodes or stick with tokens for my Multi-DC cluster. >> >> Would you be able to show a nodetool ring/status from your cluster to see >> what the token assignment looks like ? >> >> Thanks >> >> Matt >> >> >> On Wed, Jun 4, 2014 at 8:31 AM, Vasileios Vlachos < >> vasileiosvlac...@gmail.com> wrote: >> >>> I should have said that earlier really... I am using 1.2.16 and Vnodes >>> are enabled. >>> >>> Thanks, >>> >>> Vasilis >>> >>> -- >>> Kind Regards, >>> >>> Vasileios Vlachos >>> >>> >> >
