This is what I do to shutdown. Disabling thrift and gossip will stop incoming requests, but it wont stop existing streams. However these do not go through the commit log.
echo "Disabling thrift and gossip..." nodetool -h localhost disablethrift; nodetool -h localhost disablegossip; echo "Sleeping for 10..." sleep 10; echo "Drain..." nodetool -h localhost drain; echo "Sleeping for 10..." sleep 10; echo "Stopping..." sudo monit stop cassandra; A ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/01/2013, at 9:02 AM, Edward Capriolo <edlinuxg...@gmail.com> wrote: > The only true drain is > 1) turn on ip tables to stop all incoming traffic > 2) flush > 3) wait > 4) delete files > 5) upgrade > 6) restart > > > On Thu, Jan 3, 2013 at 2:59 PM, Michael Kjellman <mkjell...@barracuda.com> > wrote: > That's why I didn’t create a ticket as I knew there was one. But, I thought > this had been fixed in 1.1.7 ?? > > From: Edward Capriolo <edlinuxg...@gmail.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Thursday, January 3, 2013 11:57 AM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Error after 1.2.0 upgrade > > There is a bug on this, drain has been in a weird state for a long time. In > 1.0 it did not work labeled as a known limitation. > > https://issues.apache.org/jira/browse/CASSANDRA-4446 > > > > On Thu, Jan 3, 2013 at 2:49 PM, Michael Kjellman <mkjell...@barracuda.com> > wrote: > Another thing: for those that use counters this might be a problem. > > I always do a nodetool drain before upgrading a node (as is good practice > btw). However, in every case on every one of my nodes, the commit log was > replayed on each node and mutations were created. Could lead to double > counting of counters… > > No bug for that yet > > Best, > Micahel > > From: Michael Kjellman <mkjell...@barracuda.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Thursday, January 3, 2013 11:42 AM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Error after 1.2.0 upgrade > > Tracking Issues: > > https://issues.apache.org/jira/browse/CASSANDRA-5101 > https://issues.apache.org/jira/browse/CASSANDRA-5104 which was created > because of https://issues.apache.org/jira/browse/CASSANDRA-5103 > https://issues.apache.org/jira/browse/CASSANDRA-5102 > > Also friendly reminder to all that cql2 created indexes will not work with > cql3. You need to drop them and recreate in cql3, otherwise you'll see > rpc_timeout issues. > > I'll update with more issues as I see them. The fun bugs never happen in your > dev environment do they :) > > From: aaron morton <aa...@thelastpickle.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Thursday, January 3, 2013 11:38 AM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Error after 1.2.0 upgrade > > Michael, > Could you share some of your problems ? May be of help for others. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 4/01/2013, at 5:45 AM, Michael Kjellman <mkjell...@barracuda.com> wrote: > >> I'm having huge upgrade issues from 1.1.7 -> 1.2.0 atm but in a 12 node >> cluster which I am slowly massaging into a good state I haven't seen this in >> 15+ hours of operation… >> >> This looks related to JNA? >> >> From: Alain RODRIGUEZ <arodr...@gmail.com> >> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Date: Thursday, January 3, 2013 8:42 AM >> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Subject: Error after 1.2.0 upgrade >> >> In a dev env, C* 1.1.7 -> 1.2.0, 1 node. >> >> I run Cassandra in a 8GB memory environment. >> >> The upgrade went well, but I sometimes have the following error: >> >> INFO 17:31:04,143 Node /192.168.100.201 state jump to normal >> INFO 17:31:04,149 Enqueuing flush of Memtable-local@1654799672(32/32 >> serialized/live bytes, 2 ops) >> INFO 17:31:04,149 Writing Memtable-local@1654799672(32/32 serialized/live >> bytes, 2 ops) >> INFO 17:31:04,371 Completed flushing >> /home/stockage/cassandra/data/system/local/system-local-ia-12-Data.db (91 >> bytes) for commitlog position ReplayPosition(segmentId=1357230649515, >> position=49584) >> INFO 17:31:04,376 Startup completed! Now serving reads. >> INFO 17:31:04,798 Compacted to >> [/var/lib/cassandra/data/system/local/system-local-ia-13-Data.db,]. 950 to >> 471 (~49% of original) bytes for 1 keys at 0,000507MB/s. Time: 886ms. >> INFO 17:31:04,889 mx4j successfuly loaded >> HttpAdaptor version 3.0.2 started on port 8081 >> INFO 17:31:04,967 Not starting native transport as requested. Use JMX >> (StorageService->startNativeTransport()) to start it >> INFO 17:31:04,980 Binding thrift service to /0.0.0.0:9160 >> INFO 17:31:05,007 Using TFramedTransport with a max frame size of 15728640 >> bytes. >> INFO 17:31:09,964 Using synchronous/threadpool thrift server on 0.0.0.0 : >> 9160 >> INFO 17:31:09,965 Listening for thrift clients... >> *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with >> message transform method call failed at >> ../../../src/share/instrument/JPLISAgent.c line: 806 >> ERROR 17:33:56,002 Exception in thread Thread[Thrift:1702,5,main] >> java.lang.StackOverflowError >> at java.net.SocketInputStream.socketRead0(Native Method) >> at java.net.SocketInputStream.read(Unknown Source) >> at java.io.BufferedInputStream.fill(Unknown Source) >> at java.io.BufferedInputStream.read1(Unknown Source) >> at java.io.BufferedInputStream.read(Unknown Source) >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) >> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >> at >> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) >> at >> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) >> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199) >> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown >> Source) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> at java.lang.Thread.run(Unknown Source) >> >> ---------------------------------- >> Join Barracuda Networks in the fight against hunger. >> To learn how you can help in your community, please visit: >> http://on.fb.me/UAdL4f >> > > > ---------------------------------- > Join Barracuda Networks in the fight against hunger. > To learn how you can help in your community, please visit: > http://on.fb.me/UAdL4f > > > ---------------------------------- > Join Barracuda Networks in the fight against hunger. > To learn how you can help in your community, please visit: > http://on.fb.me/UAdL4f > > > > ---------------------------------- > Join Barracuda Networks in the fight against hunger. > To learn how you can help in your community, please visit: > http://on.fb.me/UAdL4f > >
