Re: Measuring WAN replication latency

2014-07-30 Thread Rahul Neelakantan
Agreed... This is what we are trying right now. Rahul Neelakantan > On Jul 30, 2014, at 1:43 PM, Jeremy Jongsma wrote: > > Yes, the results should definitely not be relied on as a future performance > indicator for key app functionality. but knowing roughly what your current > replication lat

Re: Measuring WAN replication latency

2014-07-30 Thread Jeremy Jongsma
Yes, the results should definitely not be relied on as a future performance indicator for key app functionality. but knowing roughly what your current replication latency is (and whether it's outside of the normal average) can inform client failover policies, debug data consistency issues, warn of

Re: Measuring WAN replication latency

2014-07-30 Thread Robert Coli
On Wed, Jul 30, 2014 at 6:59 AM, Rahul Neelakantan wrote: > Any ideas you can provide on how to do this will be appreciated, we would > like to build a latency monitoring tool/dashboard that shows how long it > takes for data to get sent across various DCs. > The brute force method described dow

Re: Measuring WAN replication latency

2014-07-30 Thread Jeremy Jongsma
The brute force way would be: 1) Make client connections to a node in each datacenter from your monitoring tool. 2) Periodically write a row to one datacenter (at whatever consistency level your application typically uses.) 3) Immediately query the other datacenter nodes for the same row key with

Re: Measuring WAN replication latency

2014-07-30 Thread Rahul Neelakantan
Rob, Any ideas you can provide on how to do this will be appreciated, we would like to build a latency monitoring tool/dashboard that shows how long it takes for data to get sent across various DCs. Rahul Neelakantan > On Jul 29, 2014, at 8:53 PM, Robert Coli wrote: > >> On Tue, Jul 29, 2014

Re: Measuring WAN replication latency

2014-07-29 Thread Laing, Michael
I saw this awhile back: With requests possibly coming in from either US region, we need to make > sure that the replication of data happens within an acceptable time > threshold. This lead us to perform an experiment where we wrote 1 million > records in one region of a multi-region cluster. We th

Re: Measuring WAN replication latency

2014-07-29 Thread Rahul Neelakantan
Here is the use case ... The Cassandra ring is going to span 6 data centers. The client can write in any data center (the client is also in 6 DCs) and turn back around and read from any other data center for this I assume you will tell me to use each quorum for the writes, however that is going

Re: Measuring WAN replication latency

2014-07-29 Thread Robert Coli
On Tue, Jul 29, 2014 at 3:15 PM, Rahul Neelakantan wrote: > Does anyone know of a way to measure/monitor WAN replication latency for > Cassandra? No. [1] =Rob [1] There are ways to do something like this task, but you probably don't actually want to do them. Trying to do them suggests that yo

Measuring WAN replication latency

2014-07-29 Thread Rahul Neelakantan
Does anyone know of a way to measure/monitor WAN replication latency for Cassandra? For example I make a write to DC 1 of a multi DC setup using local_quorum , how long before that write is available for a read either locally in DC1 or remotely in DC2 using local_quorum. Rahul Neelakantan