Mark,
Here you go:
*NodeTool status:*
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.10.20.15 1.62 TB 256 8.1%
01a01f07-4df2-4c87-98e9-8dd38b3e4aee rack1
UN 10.10.20.19 1.66 TB 256 8.3%
30ddf003-4d59-4a3e-85fa-e94e4adba1cb rack1
UN 10.10.20.35 1.62 TB 256 9.0%
17cb8772-2444-46ff-8525-33746514727d rack1
UN 10.10.20.31 1.64 TB 256 8.3%
1435acf9-c64d-4bcd-b6a4-abcec209815e rack1
UN 10.10.20.52 1.59 TB 256 9.1%
6b5aca07-1b14-4bc2-a7ba-96f026fa0e4e rack1
UN 10.10.20.27 1.66 TB 256 7.7%
76023cdd-c42d-4068-8b53-ae94584b8b04 rack1
UN 10.10.20.22 1.66 TB 256 8.9%
46af9664-8975-4c91-847f-3f7b8f8d5ce2 rack1
UN 10.10.20.39 1.68 TB 256 8.0%
b7d44c26-4d75-4d36-a779-b7e7bdaecbc9 rack1
UN 10.10.20.45 1.49 TB 256 7.7%
8d6bce33-8179-4660-8443-2cf822074ca4 rack1
UN 10.10.20.47 1.64 TB 256 7.9%
bcd51a92-3150-41ae-9c51-104ea154f6fa rack1
UN 10.10.20.62 1.59 TB 256 8.2%
84b47313-da75-4519-94f3-3951d554a3e5 rack1
UN 10.10.20.51 1.66 TB 256 8.9%
0343cd58-3686-465f-8280-56fb72d161e2 rack1
*Astyanax Connection Settings:*
seeds :12
maxConns :16
maxConnsPerHost :16
connectTimeout :2000
socketTimeout :60000
maxTimeoutCount :16
maxBlockedThreadsPerHost:16
maxOperationsPerConnection:16
DiscoveryType: RING_DESCRIBE
ConnectionPoolType: TOKEN_AWARE
DefaultReadConsistencyLevel: CL_QUORUM
DefaultWriteConsistencyLevel: CL_QUORUM
On Fri, Jul 11, 2014 at 5:04 PM, Mark Reddy <[email protected]> wrote:
> Can you post the output of nodetool status and your Astyanax connection
> settings?
>
>
> On Fri, Jul 11, 2014 at 9:06 PM, Ruchir Jha <[email protected]> wrote:
>
>> This is how we create our keyspace. We just ran this command once through
>> a cqlsh session on one of the nodes, so don't quite understand what you
>> mean by "check that your DC names match up"
>>
>> CREATE KEYSPACE prod WITH replication = {
>> 'class': 'NetworkTopologyStrategy',
>> 'datacenter1': '3'
>> };
>>
>>
>>
>> On Fri, Jul 11, 2014 at 3:48 PM, Chris Lohfink <[email protected]>
>> wrote:
>>
>>> What replication strategy are you using? if using NetworkTopolgyStrategy
>>> double check that your DC names match up (case sensitive)
>>>
>>> Chris
>>>
>>> On Jul 11, 2014, at 9:38 AM, Ruchir Jha <[email protected]> wrote:
>>>
>>> Here's the complete stack trace:
>>>
>>> com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException:
>>> TokenRangeOfflineException:
>>> [host=ny4lpcas5.fusionts.corp(10.10.20.47):9160, latency=22784(42874),
>>> attempts=3]UnavailableException()
>>> at
>>> com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165)
>>> at
>>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
>>> at
>>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
>>> at
>>> com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
>>> at
>>> com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
>>> at
>>> com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
>>> at
>>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:485)
>>> at
>>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:79)
>>> at
>>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:123)
>>> Caused by: UnavailableException()
>>> at
>>> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20841)
>>> at
>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>>> at
>>> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
>>> at
>>> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
>>> at
>>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:129)
>>> at
>>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:126)
>>> at
>>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
>>> ... 12 more
>>>
>>>
>>>
>>> On Fri, Jul 11, 2014 at 9:11 AM, Prem Yadav <[email protected]>
>>> wrote:
>>>
>>>> Please post the full exception.
>>>>
>>>>
>>>> On Fri, Jul 11, 2014 at 1:50 PM, Ruchir Jha <[email protected]>
>>>> wrote:
>>>>
>>>>> We have a 12 node cluster and we are consistently seeing this
>>>>> exception being thrown during peak write traffic. We have a replication
>>>>> factor of 3 and a write consistency level of QUORUM. Also note there is no
>>>>> unusual Or Full GC activity during this time. Appreciate any help.
>>>>>
>>>>> Sent from my iPhone
>>>>
>>>>
>>>>
>>>
>>>
>>
>
