Known bug, fixed in latest 0.6 release.
On Tue, Jun 15, 2010 at 3:29 PM, aaron <aa...@thelastpickle.com> wrote: > hello, > > I have a 4 node cassandra cluster with 0.6.1 installed. We've been running > a mixed read / write workload test how it works in our environment, we run > about 4M bath mutations and 40M get_range_slice requests over 6 to 8 hours > that load about 10 to 15 GB of data. > > Yesterday while there was no activity I noticed 2 nodes sitting at 200% > CPU on 8 Core machine. Thought nothing of it. Checked again this morning > and they are still sitting at that level of activity with no requests going > into them. Checking the streams using node tool I see node 3 is streaming > to node 0 and 2, and appears to have stalled. The information in the JMX > console for streams matches the info below. > > I cannot see any errors in the logs. > > This is just a test system, so am happy to bounce the JVM's. Before I do > is there anything else I should be looking for to understand why this > happened? > > Also, sorry for the previous empty email. > > Node 0 > Mode: Normal > Nothing streaming to /192.168.34.27 > Nothing streaming to /192.168.34.28 > Nothing streaming to /192.168.34.29 > Streaming from: /192.168.34.29 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db > 0/22765 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db > 0/10750717 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Index.db > 0/58 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Filter.db > 0/325 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Data.db > 0/695 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Index.db > 0/58 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Filter.db > 0/325 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Data.db > 0/695 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Index.db > 0/587164 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Filter.db > 0/22765 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Data.db > 0/5159652 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-124-Data.db > 22765/4966927 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Index.db > 22765/1223053 > > Node 1 > Mode: Normal > Nothing streaming to /192.168.34.26 > Nothing streaming to /192.168.34.28 > Nothing streaming to /192.168.34.29 > Not receiving any streams. > > Node 2 > Mode: Normal > Nothing streaming to /192.168.34.26 > Nothing streaming to /192.168.34.27 > Nothing streaming to /192.168.34.29 > Streaming from: /192.168.34.29 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db > 0/22765 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db > 0/2161660 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Index.db > 0/787524 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Filter.db > 0/22765 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Data.db > 0/6917064 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Index.db > 0/58 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Filter.db > 0/565 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Data.db > 0/695 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Index.db > 0/581779 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Filter.db > 0/22765 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Data.db > 0/5111887 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Data.db > 361367/3173057 > junkbox.mycompany: > /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Index.db > 695/361367 > > Node 3 > ode: Normal > Streaming to: /192.168.34.26 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Filter.db > 22765/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Data.db > 0/4966927 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Index.db > 0/58 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Filter.db > 0/325 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Data.db > 0/695 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Index.db > 0/1223053 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Filter.db > 0/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Data.db > 0/10750717 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Index.db > 0/58 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Filter.db > 0/325 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Data.db > 0/695 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Index.db > 0/58 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Filter.db > 0/325 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Data.db > 0/695 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Index.db > 0/587164 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Filter.db > 0/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Data.db > 0/5159652 > Nothing streaming to /192.168.34.27 > Streaming to: /192.168.34.28 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Index.db > 361367/361367 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Filter.db > 0/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Data.db > 0/3173057 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Index.db > 0/58 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Filter.db > 0/325 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Data.db > 0/695 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Index.db > 0/246014 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Filter.db > 0/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Data.db > 0/2161660 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Index.db > 0/787524 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Filter.db > 0/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Data.db > 0/6917064 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Index.db > 0/58 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Filter.db > 0/565 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Data.db > 0/695 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Index.db > 0/581779 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Filter.db > 0/22765 > > /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Data.db > 0/5111887 > Not receiving any streams. > > > >
