Hi Michael, Upgrading from 2.0.2 to 2.0.3 seems to have done the trick! Thank you for the recommendation!
Best regards, -David Laube On Dec 9, 2013, at 3:57 PM, "Laing, Michael" <michael.la...@nytimes.com> wrote: > My experience is that you must upgrade to 2.0.3 ASAP to fix this. > > Michael > > > On Mon, Dec 9, 2013 at 6:39 PM, David Laube <d...@stormpath.com> wrote: > Hi All, > > We are running Cassandra 2.0.2 and have recently stumbled upon an issue with > nodetool repair. Upon running nodetool repair on each of the 5 nodes in the > ring (one at a time) we observe the following exceptions returned to standard > out; > > > [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f > for range (1246984843639507027,1266616572749926276] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, > (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 > [2013-12-08 11:04:02,063] Repair session 284c8b40-5ff8-11e3-97c0-5973cc397f8f > for range (-109256956528331396,-89316884701275697] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, > (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.103 > [2013-12-08 11:04:02,070] Repair session 399e7160-5ff8-11e3-97c0-5973cc397f8f > for range (8901153810410866970,8915879751739915956] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, > (8901153810410866970,8915879751739915956]] Validation failed in /10.x.x.103 > [2013-12-08 11:04:02,072] Repair session 3ea73340-5ff8-11e3-97c0-5973cc397f8f > for range (1149084504576970235,1190026362216198862] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, > (1149084504576970235,1190026362216198862]] Validation failed in /10.x.x.103 > [2013-12-08 11:04:02,091] Repair session 6f0da460-5ff8-11e3-97c0-5973cc397f8f > for range (-5407189524618266750,-5389231566389960750] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, > (-5407189524618266750,-5389231566389960750]] Validation failed in /10.x.x.103 > [2013-12-09 23:16:36,962] Repair session 7efc2740-6127-11e3-97c0-5973cc397f8f > for range (1246984843639507027,1266616572749926276] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, > (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 > [2013-12-09 23:16:36,986] Repair session a8c44260-6127-11e3-97c0-5973cc397f8f > for range (-109256956528331396,-89316884701275697] failed with error > org.apache.cassandra.exceptions.RepairException: [repair > #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, > (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.210 > > The /var/log/cassandra/system.log shows similar info as above with no real > explanation as to the root cause behind the exception(s). There also does > not appear to be any additional info in /var/log/cassandra/cassandra.log. We > have tried restoring a recent snapshot of the keyespace in question to a > separate staging ring and the repair runs successfully and without exception > there. This is even after we tried insert/delete on the keyspace in the > separate staging ring. Has anyone seen this behavior before and what can we > do to resolve this? Any assistance would be greatly appreciated. > > Best regards, > -Dave >
