really bad select performance

[Chris Hart]

Hi,

I have the following cluster:

136112946768375385385349842972707284580     
<ip address>  MountainViewRAC1        Up     Normal  1.86 GB         20.00%  0  
                                         
<ip address>  MountainViewRAC1        Up     Normal  2.17 GB         33.33%  
56713727820156410577229101238628035242      
<ip address>  MountainViewRAC1        Up     Normal  2.41 GB         33.33%  
113427455640312821154458202477256070485     
<ip address>     Rackspace   RAC1        Up     Normal  3.9 GB          13.33%  
136112946768375385385349842972707284580    

The following query runs quickly on all nodes except 1 MountainView node:

 select * from Access_Log where row_loaded = 0 limit 1;

There is a secondary index on row_loaded.  The query usually doesn't complete 
(but sometimes does) on the bad node and returns very quickly on all other 
nodes.  I've upping the rpc timeout to a full minute (rpc_timeout_in_ms: 60000) 
in the yaml, but it still often doesn't complete in a minute.  It seems just as 
likely to complete and takes about the same amount of time whether the limit is 
1, 100 or 1000.


Thanks for any help,
Chris