Client side (it is just a 5th instance in the same EC2 zone, having
stress.py installed on it) gives the following error:
Process Inserter-4:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-1:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-3:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-8:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-2:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-6:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-5:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-7:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-9:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-10:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py", line 238, in run
self.cclient.batch_mutate(cfmap, consistency)
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 873, in batch_mutate
self.recv_batch_mutate()
File
"/home/ec2-user/cassandra/interface/thrift/gen-py/cassandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
The related server side errors look like:
DEBUG 22:29:04,407 Deleting CommitLog-1299623301883.log.header
DEBUG 22:29:04,412 Deleting CommitLog-1299623301883.log
DEBUG 22:29:04,443 Deleting CommitLog-1299623318627.log.header
DEBUG 22:29:04,443 Deleting CommitLog-1299623318627.log
DEBUG 22:29:09,202 ... timed out
DEBUG 22:29:09,426 ... timed out
DEBUG 22:29:10,318 ... timed out
DEBUG 22:29:11,354 logged out: #<User allow_all groups=[]>
DEBUG 22:29:11,354 logged out: #<User allow_all groups=[]>
DEBUG 22:29:11,354 logged out: #<User allow_all groups=[]>
DEBUG 22:29:12,442 Processing response on a callback from 784@/10.253.203.224
DEBUG 22:29:12,443 Processing response on a callback from 786@/10.253.203.224
DEBUG 22:29:12,443 Processing response on a callback from 791@/10.253.203.224
On Tue, Mar 8, 2011 at 3:22 PM, aaron morton <aa...@thelastpickle.com> wrote:
> Is this a client side time out or a server side one? What does the error
> stack look like ?
> Also check the server side logs for errors. The thrift API will raise a
> timeout when less the CL level of nodes return in rpc_timeout.
> Good luck
> Aaron
> On 9/03/2011, at 7:37 AM, ruslan usifov wrote:
>
>
> 2011/3/8 A J <s5a...@gmail.com>
>>
>> Trying out stress.py on AWS EC2 environment (4 Large instances. Each
>> of 2-cores and 7.5GB RAM. All in the same region/zone.)
>>
>> python stress.py -o insert -d
>> 10.253.203.224,10.220.203.48,10.220.17.84,10.124.89.81 -l 2 -e ALL -t
>> 10 -n 500 -S 1000000 -k
>>
>> (I want to try with column size of about 1MB. I am assuming the above
>> gives me 10 parallel threads each executing 50 inserts sequentially
>> (500/10) ).
>>
>> Getting several timeout errors.TimedOutException(). With just 10
>> concurrent writes spread across 4 nodes, kind of surprised to get so
>> many timeouts. Any suggestions ?
>>
>
>
> It may by EC2 disc speed degradation (io speed of EC2 instances doesnt
> const, also can vary in greater limits)
>
>
