I just tested with stress tool, it is reproducible and timeout happens always.
./stress -d <mynode address> -e QUORUM -l 3 -o RANGE_SLICE total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time 0,0,0,NaN,10 0,0,0,NaN,20 0,0,0,NaN,30 0,0,0,NaN,40 0,0,0,NaN,50 0,0,0,NaN,60 0,0,0,NaN,70 0,0,0,NaN,80 0,0,0,NaN,91 0,0,0,NaN,101 Operation [2] retried 10 times - error on calling get_indexed_slices for range offset 0000002 ((TimedOutException)) Operation [46] retried 10 times - error on calling get_indexed_slices for range offset 0000046 ((TimedOutException)) Operation [47] retried 10 times - error on calling get_indexed_slices for range offset 0000047 ((TimedOutException)) Operation [24] retried 10 times - error on calling get_indexed_slices for range offset 0000024 ((TimedOutException)) Operation [45] retried 10 times - error on calling get_indexed_slices for range offset 0000045 ((TimedOutException)) Operation [48] retried 10 times - error on calling get_indexed_slices for range offset 0000048 ((TimedOutException)) Operation [5] retried 10 times - error on calling get_indexed_slices for range offset 0000005 ((TimedOutException)) Operation [28] retried 10 times - error on calling get_indexed_slices for range offset 0000028 ((TimedOutException)) Operation [0] retried 10 times - error on calling get_indexed_slices for range offset 0000000 ((TimedOutException)) Operation [23] retried 10 times - error on calling get_indexed_slices for range offset 0000023 ((TimedOutException)) Operation [32] retried 10 times - error on calling get_indexed_slices for range offset 0000032 ((TimedOutException)) Operation [36] retried 10 times - error on calling get_indexed_slices for range offset 0000036 ((TimedOutException)) Operation [16] retried 10 times - error on calling get_indexed_slices for range offset 0000016 ((TimedOutException)) Operation [6] retried 10 times - error on calling get_indexed_slices for range offset 0000006 ((TimedOutException)) Operation [9] retried 10 times - error on calling get_indexed_slices for range offset 0000009 ((TimedOutException)) Operation [11] retried 10 times - error on calling get_indexed_slices for range offset 0000011 ((TimedOutException)) Operation [25] retried 10 times - error on calling get_indexed_slices for range offset 0000025 ((TimedOutException)) Operation [26] retried 10 times - error on calling get_indexed_slices for range offset 0000026 ((TimedOutException)) Operation [43] retried 10 times - error on calling get_indexed_slices for range offset 0000043 ((TimedOutException)) Operation [27] retried 10 times - error on calling get_indexed_slices for range offset 0000027 ((TimedOutException)) Operation [33] retried 10 times - error on calling get_indexed_slices for range offset 0000033 ((TimedOutException)) Operation [44] retried 10 times - error on calling get_indexed_slices for range offset 0000044 ((TimedOutException)) Operation [19] retried 10 times - error on calling get_indexed_slices for range offset 0000019 ((TimedOutException)) Operation [35] retried 10 times - error on calling get_indexed_slices for range offset 0000035 ((TimedOutException)) Operation [34] retried 10 times - error on calling get_indexed_slices for range offset 0000034 ((TimedOutException)) Operation [30] retried 10 times - error on calling get_indexed_slices for range offset 0000030 ((TimedOutException)) Operation [14] retried 10 times - error on calling get_indexed_slices for range offset 0000014 ((TimedOutException)) Operation [7] retried 10 times - error on calling get_indexed_slices for range offset 0000007 ((TimedOutException)) Operation [20] retried 10 times - error on calling get_indexed_slices for range offset 0000020 ((TimedOutException)) Operation [3] retried 10 times - error on calling get_indexed_slices for range offset 0000003 ((TimedOutException)) Operation [8] retried 10 times - error on calling get_indexed_slices for range offset 0000008 ((TimedOutException)) Operation [15] retried 10 times - error on calling get_indexed_slices for range offset 0000015 ((TimedOutException)) Operation [29] retried 10 times - error on calling get_indexed_slices for range offset 0000029 ((TimedOutException)) Operation [12] retried 10 times - error on calling get_indexed_slices for range offset 0000012 ((TimedOutException)) Operation [40] retried 10 times - error on calling get_indexed_slices for range offset 0000040 ((TimedOutException)) Operation [38] retried 10 times - error on calling get_indexed_slices for range offset 0000038 ((TimedOutException)) Operation [18] retried 10 times - error on calling get_indexed_slices for range offset 0000018 ((TimedOutException)) Operation [31] retried 10 times - error on calling get_indexed_slices for range offset 0000031 ((TimedOutException)) Operation [4] retried 10 times - error on calling get_indexed_slices for range offset 0000004 ((TimedOutException)) Operation [22] retried 10 times - error on calling get_indexed_slices for range offset 0000022 ((TimedOutException)) Operation [49] retried 10 times - error on calling get_indexed_slices for range offset 0000049 ((TimedOutException)) Operation [41] retried 10 times - error on calling get_indexed_slices for range offset 0000041 ((TimedOutException)) Operation [37] retried 10 times - error on calling get_indexed_slices for range offset 0000037 ((TimedOutException)) Operation [13] retried 10 times - error on calling get_indexed_slices for range offset 0000013 ((TimedOutException)) Operation [39] retried 10 times - error on calling get_indexed_slices for range offset 0000039 ((TimedOutException)) Operation [21] retried 10 times - error on calling get_indexed_slices for range offset 0000021 ((TimedOutException)) Operation [10] retried 10 times - error on calling get_indexed_slices for range offset 0000010 ((TimedOutException)) Operation [17] retried 10 times - error on calling get_indexed_slices for range offset 0000017 ((TimedOutException)) Operation [1] retried 10 times - error on calling get_indexed_slices for range offset 0000001 ((TimedOutException)) Operation [42] retried 10 times - error on calling get_indexed_slices for range offset 0000042 ((TimedOutException)) 0,0,0,NaN,106 END On Dec 2, 2011, at 2:45 PM, Janne Jalkanen wrote: > > Would be glad to be of any help; it's kind of annoying. > > * Nothing unusual on any nodes that I can see > * Cannot reproduce on a single-node cluster; I see it only on our prod > cluster which was running 0.6.13 until this point (cluster conf is attached > to the JIRA issue mentioned below). > > Let me know of anything that I can try, short of taking my production cluster > offline :-P > > /Janne > > On Dec 2, 2011, at 20:42 , Jonathan Ellis wrote: > >> The first step towards determining how serious it is, is showing us >> how to reproduce it or otherwise narrowing down what could be causing >> it, because timeouts can be caused by a lot of non-bug scenarios. >> Does it occur for every query or just some? Is there anything unusual >> on the coordinator or replica nodes, like high CPU? Can you reproduce >> with the stress tool? Can you reproduce on a single-node-cluster? >> That kind of thing. >> >> On Fri, Dec 2, 2011 at 12:18 PM, Pierre Belanger >> <pierre.belan...@xobni.com> wrote: >>> Hello, >>> >>> Is this bug serious enough for 1.0.6 to come out shortly or not? >>> >>> Thank you, >>> PBR >>> >>> >>> >>> On Thu, Dec 1, 2011 at 6:05 PM, Zhong Li <z...@voxeo.com> wrote: >>>> >>>> After upgrade to 1.0.5 RangeSlice got timeout. Ticket >>>> https://issues.apache.org/jira/browse/CASSANDRA-3551 >>>> >>>> On Dec 1, 2011, at 5:43 PM, Evgeniy Ryabitskiy wrote: >>>> >>>>> +1 >>>>> After upgrade to 1.0.5 also have Timeout exception on Secondary Index >>>>> search (get_indexed_slices API) . >>>> >>> >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com >
