Ah, that sounds like https://issues.apache.org/jira/browse/CASSANDRA-1198. That it happened after removetoken is just that that happened to change your ring topology enough to make your queries start hitting it.
On Tue, Jun 22, 2010 at 10:39 AM, Joost Ouwerkerk <jo...@openplaces.org> wrote: > I don't mind missing data for a few hours, it's the weird behaviour of > get_range_slices that's bothering me. I added some logging to > ColumnFamilyRecordReader to see what's going on: > > Split startToken=67160993471237854630929198835217410155, > endToken=68643623863384825230116928934887817211 > > ... > > Getting batch for range: 67965855060996012099315582648654139032 to > 68643623863384825230116928934887817211 > > Token for last row is: 50448492574454416067449808504057295946 > > Getting batch for range: 50448492574454416067449808504057295946 to > 68643623863384825230116928934887817211 > > ... > > Notice how the get_range_slices response is invalid since it returns an > out-of-range row. This poisons the batching loop and causes the task to > spin out of control. > /joost > > On Tue, Jun 22, 2010 at 9:09 AM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> What I would expect to have happen is for the removed node to >> disappear from the ring and for nodes that are supposed to get more >> data to start streaming it over. I would expect it to be hours before >> any new data started appearing anywhere when you are anticompacting >> 80+GB prior to the streaming part. >> http://wiki.apache.org/cassandra/Streaming >> >> On Tue, Jun 22, 2010 at 12:57 AM, Joost Ouwerkerk <jo...@openplaces.org> >> wrote: >> > Yes, although "forget" implies that we once knew we were supposed to do >> > so. >> > Given the following before-and-after states, on which nodes are we >> > supposed >> > to run repair? Should the cluster be restarted? Is there anything else >> > we >> > should be doing, or not doing? >> > >> > 1. Node is down due to hardware failure >> > >> > 192.168.1.104 Up 111.75 GB >> > 8954799129498380617457226511362321354 | ^ >> > 192.168.1.106 Up 113.25 GB >> > 17909598258996761234914453022724642708 v | >> > 192.168.1.107 Up 75.65 GB >> > 22386997823745951543643066278405803385 | ^ >> > 192.168.1.108 Down 75.77 GB >> > 26864397388495141852371679534086964062 v | >> > 192.168.1.109 Up 76.14 GB >> > 35819196517993522469828906045449285416 | ^ >> > 192.168.1.110 Up 75.9 GB >> > 40296596082742712778557519301130446093 v | >> > 192.168.1.111 Up 95.21 GB >> > 49251395212241093396014745812492767447 | ^ >> > >> > 2. nodetool removetoken 26864397388495141852371679534086964062 >> > >> > 192.168.1.104 Up 111.75 GB >> > 8954799129498380617457226511362321354 | ^ >> > 192.168.1.106 Up 113.25 GB >> > 17909598258996761234914453022724642708 v | >> > 192.168.1.107 Up 75.65 GB >> > 22386997823745951543643066278405803385 | ^ >> > 192.168.1.109 Up 76.14 GB >> > 35819196517993522469828906045449285416 | ^ >> > 192.168.1.110 Up 75.9 GB >> > 40296596082742712778557519301130446093 v | >> > 192.168.1.111 Up 95.21 GB >> > 49251395212241093396014745812492767447 | ^ >> > >> > At this point we're expecting 192.168.1.107 to pick up the slack for the >> > removed token, and for 192.168.1.109 and/or 192.168.1.110 to start >> > streaming >> > data to 192.168.1.107 since they are holding the replicated data for >> > that >> > range. >> > >> > 3. nodetool repair ? >> > >> > On Tue, Jun 22, 2010 at 12:03 AM, Benjamin Black <b...@b3k.us> wrote: >> >> >> >> Did you forget to run repair? >> >> >> >> On Mon, Jun 21, 2010 at 7:02 PM, Joost Ouwerkerk <jo...@openplaces.org> >> >> wrote: >> >> > I believe we did nodetool removetoken on nodes that were already down >> >> > (due >> >> > to hardware failure), but I will check to make sure. We're running >> >> > Cassandra >> >> > 0.6.2. >> >> > >> >> > On Mon, Jun 21, 2010 at 9:59 PM, Joost Ouwerkerk >> >> > <jo...@openplaces.org> >> >> > wrote: >> >> >> >> >> >> Greg, can you describe the steps we took to decommission the nodes? >> >> >> >> >> >> ---------- Forwarded message ---------- >> >> >> From: Rob Coli <rc...@digg.com> >> >> >> Date: Mon, Jun 21, 2010 at 8:08 PM >> >> >> Subject: Re: get_range_slices confused about token ranges after >> >> >> decommissioning a node >> >> >> To: user@cassandra.apache.org >> >> >> >> >> >> >> >> >> On 6/21/10 4:57 PM, Joost Ouwerkerk wrote: >> >> >>> >> >> >>> We're seeing very strange behaviour after decommissioning a node: >> >> >>> when >> >> >>> requesting a get_range_slices with a KeyRange by token, we are >> >> >>> getting >> >> >>> back tokens that are out of range. >> >> >> >> >> >> What sequence of actions did you take to "decommission" the node? >> >> >> What >> >> >> version of Cassandra are you running? >> >> >> >> >> >> =Rob >> >> >> >> >> > >> >> > >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
