sounds like http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
On Sun, Apr 18, 2010 at 3:05 AM, Schubert Zhang <zson...@gmail.com> wrote: > Thanks Jonathan and Chris, > > I am using 6u19 in above test: > > [cassan...@nd1-rack0-cloud cassandra]$ java -version > java version "1.6.0_19" > Java(TM) SE Runtime Environment (build 1.6.0_19-b04) > Java HotSpot(TM) 64-Bit Server VM (build 16.2-b04, mixed mode) > > > On Sun, Apr 18, 2010 at 12:56 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> Ugh, I mean the fix for LBQ, and it was actually in u19 not 20. >> >> http://java.sun.com/javase/6/webnotes/6u19.html >> >> On Sat, Apr 17, 2010 at 11:38 PM, Jonathan Ellis <jbel...@gmail.com> >> wrote: >> > jdk 1.6 b20 has the fix for the CLQ gc problem, btw. >> > >> > On Sat, Apr 17, 2010 at 11:31 PM, Chris Goffinet <goffi...@digg.com> >> > wrote: >> >> I wonder if that might be related to this: >> >> https://issues.apache.org/jira/browse/CASSANDRA-896 >> >> We switched from a Concurrent structure to LinkedBlockingQueue in 0.6. >> >> -Chris >> >> On Apr 17, 2010, at 9:26 PM, Schubert Zhang wrote: >> >> >> >> We are testing 0.6.0, compares with 0.5.1, and it seems: >> >> >> >> 1. 0.6.0 need more memory/heap. >> >> 2. after inserted billions of columns, tens-million of keys, the >> >> inseting >> >> operation become very slow and jamed. >> >> Exceptions TimeoutException and UnavailableException are throwed >> >> sometimes. >> >> >> >> I add more log, such as : >> >> >> >> WARN [pool-1-thread-4] 2010-04-18 00:00:00,534 CassandraServer.java >> >> (line >> >> 460) UnavailableException() >> >> UnavailableException() >> >> at >> >> >> >> org.apache.cassandra.service.StorageProxy.assureSufficientLiveNodes(StorageProxy.java:298) >> >> at >> >> >> >> org.apache.cassandra.service.StorageProxy.mutateBlocking(StorageProxy.java:208) >> >> at >> >> >> >> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:452) >> >> at >> >> >> >> org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:362) >> >> at >> >> >> >> org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:1484) >> >> at >> >> >> >> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1125) >> >> at >> >> >> >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> at java.lang.Thread.run(Thread.java:619) >> >> >> >> WARN [pool-1-thread-5] 2010-04-18 12:20:03,614 CassandraServer.java >> >> (line >> >> 456) java.util.concurrent.TimeoutException: Operation tim >> >> ed out - received only 00 responses >> >> java.util.concurrent.TimeoutException: Operation timed out - received >> >> only >> >> 00 responses >> >> at >> >> >> >> org.apache.cassandra.service.WriteResponseHandler.get(WriteResponseHandler.java:77) >> >> at >> >> >> >> org.apache.cassandra.service.StorageProxy.mutateBlocking(StorageProxy.java:262) >> >> at >> >> >> >> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:452) >> >> at >> >> >> >> org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:362) >> >> at >> >> >> >> org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:1484) >> >> at >> >> >> >> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1125) >> >> at >> >> >> >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> at java.lang.Thread.run(Thread.java:619) >> >> >> >> and >> >> INFO [Timer-1] 2010-04-18 11:09:13,928 Gossiper.java (line 179) >> >> InetAddress >> >> /10.24.1.16 is now dead. >> >> INFO [Timer-1] 2010-04-18 11:09:14,930 Gossiper.java (line 179) >> >> InetAddress >> >> /10.24.1.14 is now dead. >> >> INFO [Timer-1] 2010-04-18 11:09:14,930 Gossiper.java (line 179) >> >> InetAddress >> >> /10.24.1.18 is now dead. >> >> >> >> In fact, these node are alive. >> >> >> >> >> >> >> >> >> >> 2010/4/15 Ted Zlatanov <t...@lifelogs.com> >> >>> >> >>> On Wed, 14 Apr 2010 12:23:19 -0500 Eric Evans <eev...@rackspace.com> >> >>> wrote: >> >>> >> >>> EE> On Wed, 2010-04-14 at 10:16 -0500, Ted Zlatanov wrote: >> >>> >> Can it support a non-root user through /etc/default/cassandra? >> >>> >> I've >> >>> >> been patching the init script myself but was hoping this would be >> >>> >> standard. >> >>> >> >>> EE> It's the first item on debian/TODO, but, you know, patches welcome >> >>> and >> >>> EE> all that. >> >>> >> >>> The appended patch has been sufficient for me. I have to override the >> >>> PIDFILE too, but that's a system issue. So my /etc/default/cassandra, >> >>> for example, is: >> >>> >> >>> JAVA_HOME="/usr/lib/jvm/java-6-sun" >> >>> USER=cassandra >> >>> PIDFILE=/var/tmp/$NAME.pid >> >>> >> >>> Ted >> >>> >> >>> --- debian/init 2010-04-14 12:57:30.000000000 -0500 >> >>> +++ /etc/init.d/cassandra 2010-04-14 13:00:25.000000000 -0500 >> >>> @@ -21,6 +21,7 @@ >> >>> JSVC=/usr/bin/jsvc >> >>> JVM_MAX_MEM="1G" >> >>> JVM_START_MEM="128M" >> >>> +USER=root >> >>> >> >>> [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0 >> >>> [ -e /etc/cassandra/storage-conf.xml ] || exit 0 >> >>> @@ -75,6 +76,7 @@ >> >>> is_running && return 1 >> >>> >> >>> $JSVC \ >> >>> + -user $USER \ >> >>> -home $JAVA_HOME \ >> >>> -pidfile $PIDFILE \ >> >>> -errfile "&1" \ >> >>> >> >> >> >> >> >> >> > > >
