OK, attached are the logs.
At time 15:55 I performed a write with CL = ONE, a subsequent read with CL =
ONE and received correct results. Similar results occur with any read CL other
than LOCAL_QUORUM.
At time 15:58 I performed a write with CL = ONE, a subsequent read with CL =
LOCAL_QUORUM and received a TimedOutException.
System.log and cassandra.log are from server dlc05-sea to corresponds with the
hector error message "Closing client CassandraClient<dlc05-sea:9160-50>". I
haven't found anything in the logs that indicates the problem but maybe others
with more discerning eyes can.
Please let me know if I can provide any additional information.
Thanks,
-Thor
On Dec 28, 2010, at 12:21 PM, Jonathan Ellis wrote:
> Turn on debug logging to see what nodes the request gets sent to, and
> which ones replied.
>
> [dev to BCC]
>
> On Tue, Dec 28, 2010 at 1:59 PM, Thor Carpenter
> <tcarpen...@mediosystems.com> wrote:
>> Hi All,
>>
>> I am a bit stumped. When reading at a CL of local_quorum I get
>> TimedOutExceptions however when reading at any other CL, including ALL,
>> everything works as expected. Any write CL, including local_quorum, also
>> works as expected.
>>
>> I am running a 4 node 0.7 RC1 cluster in datacenter "DC1". "DC2" is planned
>> for the future but currently has no nodes. I am using Hector 0.7.0-20.
>>
>> Here is my testing schema:
>>
>> create keyspace GlobalCacheKeyspaceDC1RF4 with
>> replication_factor = 4 and
>> placement_strategy =
>> 'org.apache.cassandra.locator.NetworkTopologyStrategy' and
>> strategy_options = [{DC1:4, DC2:0}];
>>
>> use GlobalCacheKeyspaceDC1RF4;
>>
>> create column family GlobalCacheCF with
>> comment = 'General purpose cache' and
>> column_type = 'Standard' and
>> comparator = 'BytesType' and
>> keys_cached = 200000 and
>> rows_cached = 2000 and
>> gc_grace = 0 and
>> read_repair_chance = 0.0;
>>
>>
>> Here is my cassandra-topology.properties:
>>
>> 10.5.64.15=DC1:R1
>> 10.5.64.28=DC1:R1
>> 10.5.64.56=DC1:R1
>> 10.5.64.39=DC1:R1
>>
>> # default for unknown nodes
>> default=DC2:R1
>>
>>
>> Here is the exception I get when reading at a CL of local_quorum:
>>
>> 1:44:14.176 [main] ERROR m.p.c.connection.HConnectionManager - Could not
>> fullfill request on this host CassandraClient<dlc05-sea:9160-50>
>> 11:44:14.182 [main] ERROR m.p.c.connection.HConnectionManager - Exception:
>> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
>> at
>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:27)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$15.execute(KeyspaceServiceImpl.java:598)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$15.execute(KeyspaceServiceImpl.java:587)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:89)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:101)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceServiceImpl.java:604)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:57)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:52)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:65)
>> [hector-0.7.0-20.jar:na]
>> at
>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery.execute(ThriftColumnQuery.java:51)
>> [hector-0.7.0-20.jar:na]
>>
>>
>> There are no warnings or errors in cassandra.log.
>>
>> Any guidance would be appreciated.
>>
>> Thanks,
>> -Thor
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
