Hi All

I have a Cassandra 4.1.4 cluster with two data centers, each having 3
nodes. The configuration is: listen_address = private IP, broadcast_address
= public IP, listen_on_broadcast_address = true, prefer_local = true

*Issue Observed:*

   - We execute a multi-partition batch query with LOCAL_QUORUM
   consistency, and it succeeds.
   - We bring down the private IP of one node in the DC where queries are
   executed.
   - Batch queries start timing out, but simple INSERT and SELECT queries
   work fine.
   - Stopping the affected node (where IP was down) resolves the issue, and
   batch queries succeed again.

*Analysis So Far:*

   -

   From the Cassandra source code, it looks like the coordinator picks two
   other nodes (nodes other than coordinator) for writing batch logs.
   -

   The failed node (private IP down) gets selected for batch logs, but it
   never responds, causing the timeout.
   -

   The node is not marked down in nodetool status but is unreachable in
   nodetool describecluster till node is not brought down. After restarting
   the node, nodetool status also shows the problematic node as Down.

Is this expected behaviour that nodes in the datacenter are not able to
mark the node down as soon as private interface of one of the node goes
down in a multi dc setup where both private and public interfaces are open?

I am seeing the same behaviour in Cassandra 3.11.2

Regards

Manish

Reply via email to