Hi Alex,
I now ran nodetool repair –full –pr keyspace cfs on all nodes in parallel and
this may pop up now:
0.176.38.128 (progress: 1%)
[2017-09-18 07:59:17,145] Some repair failed
[2017-09-18 07:59:17,151] Repair command #3 finished in 0 seconds
error: Repair job has failed with the error message: [2017-09-18 07:59:17,145]
Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-09-18 07:59:17,145] Some repair failed
at
org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
2017-09-18 07:59:17 repair finished
If running the above nodetool call sequentially on all nodes, repair finishes
without printing a stack trace.
The error message and stack trace isn’t really useful here. Any further
ideas/experiences?
Thanks,
Thomas
From: Alexander Dejanovski [mailto:[email protected]]
Sent: Freitag, 15. September 2017 11:30
To: [email protected]
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
Right, you should indeed add the "--full" flag to perform full repairs, and you
can then keep the "-pr" flag.
I'd advise to monitor the status of your SSTables as you'll probably end up
with a pool of SSTables marked as repaired, and another pool marked as
unrepaired which won't be compacted together (hence the suggestion of running
subrange repairs).
Use sstablemetadata to check on the "Repaired at" value for each. 0 means
unrepaired and any other value (a timestamp) means the SSTable has been
repaired.
I've had behaviors in the past where running "-pr" on the whole cluster would
still not mark all SSTables as repaired, but I can't say if that behavior has
changed in latest versions.
Having separate pools of SStables that cannot be compacted means that you might
have tombstones that don't get evicted due to partitions living in both states
(repaired/unrepaired).
To sum up the recommendations :
- Run a full repair with both "--full" and "-pr" and check that SSTables are
properly marked as repaired
- Use a tight repair schedule to avoid keeping partitions for too long in both
repaired and unrepaired state
- Switch to subrange repair if you want to fully avoid marking SSTables as
repaired (which you don't need anyway since you're not using incremental
repairs). If you wish to do this, you'll have to mark back all your sstables to
unrepaired, using nodetool
sstablerepairedset<https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSSTableRepairedSet.html>.
Cheers,
On Fri, Sep 15, 2017 at 10:27 AM Steinmaurer, Thomas
<[email protected]<mailto:[email protected]>>
wrote:
Hi Alex,
thanks a lot. Somehow missed that incremental repairs are the default now.
We have been happy with full repair so far, cause data what we currently
manually invoke for being prepared is a small (~1GB or even smaller).
So I guess with full repairs across all nodes, we still can stick with the
partition range (-pr) option, but with 3.0 we additionally have to provide the
–full option, right?
Thanks again,
Thomas
From: Alexander Dejanovski
[mailto:[email protected]<mailto:[email protected]>]
Sent: Freitag, 15. September 2017 09:45
To: [email protected]<mailto:[email protected]>
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
Hi Thomas,
in 2.1.18, the default repair mode was full repair while since 2.2 it is
incremental repair.
So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't trigger
the same operation.
Incremental repair cannot run on more than one node at a time on a cluster,
because you risk to have conflicts with sessions trying to anticompact and run
validation compactions on the same SSTables (which will make the validation
phase fail, like your logs are showing).
Furthermore, you should never use "-pr" with incremental repair because it is
useless in that mode, and won't properly perform anticompaction on all nodes.
If you were happy with full repairs in 2.1.18, I'd suggest to stick with those
in 3.0.14 as well because there are still too many caveats with incremental
repairs that should hopefully be fixed in 4.0+.
Note that full repair will also trigger anticompaction and mark SSTables as
repaired in your release of Cassandra, and only full subrange repairs are the
only flavor that will skip anticompaction.
You will need some tooling to help with subrange repairs though, and I'd
recommend to use Reaper which handles automation for you :
http://cassandra-reaper.io/
If you decide to stick with incremental repairs, first perform a rolling
restart of your cluster to make sure no repair session still runs, and run
"nodetool repair" on a single node at a time. Move on to the next node only
when nodetool or the logs show that repair is over (which will include the
anticompaction phase).
Cheers,
On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas
<[email protected]<mailto:[email protected]>>
wrote:
Hello,
we are currently in the process of upgrading from 2.1.18 to 3.0.14. After
upgrading a few test environments, we start to see some suspicious log entries
regarding repair issues.
We have a cron job on all nodes basically executing the following repair call
on a daily basis:
nodetool repair –pr <list of CFs>
This gets started on all nodes at the same time. While this has worked with
2.1.18 (at least we haven’t seen anything suspicious in Cassandra log), with
3.0.14 we get something similar like that on all nodes (see below; IP addresses
and KS/CF faked).
Any pointers are appreciated. Thanks.
Thomas
INFO [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 - [repair
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
/FAKE.35.153, /FAKE.34.171 on range [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045],
(-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908],
(3732113975108886193,3743105867152786342],
(4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372],
(-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230],
(-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102],
(-4650650128572424858,-4650260492494258461],
(1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115],
(8114903118676615937,8132992506844206601],
(6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136],
(-167838948111369123,-141862581573028594],
(481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723],
(1659668187498418295,1679582585970705122],
(-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292],
(-79866884352549492,-77667207866300333],
(8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189],
(3767057277953758442,3783780844370292025],
(-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524],
(-9005441616028750657,-9002220258513351832],
(8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932],
(4976122621177738811,4987871287137312689],
(6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158],
(6895951547735922309,6899152466544114890],
(-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485],
(3824272999898372667,3829315045986248983],
(8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597],
(-956691623200349709,-946602525018301692],
(-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936],
(8154760186218662205,8157079055586089583],
(3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260],
(-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222],
(4964519403172053705,4970446606512414858],
(-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885],
(-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912],
(3466596314109623830,3468144326691723333],
(-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ]
INFO [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Requesting merkle trees for YYY (to
[/FAKE.35.153, /FAKE.34.171, /FAKE.33.64])
INFO [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 - [repair
#075d2720-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
/FAKE.35.57, /FAKE.34.171 on range
[(-5410955131843184047,-5390722609201388849],
(-2429793939970389370,-2402273315769352748],
(8085575576842594575,8086965740279021106],
(-8802193901675845653,-8790472027607832351],
(-3900412470120874591,-3892641480459306647],
(5455804264750818305,5465037357825542970],
(4930767198829659527,4939587074207662799],
(8086965740279021106,8087442741329154201],
(-8933201045321260661,-8926445549049070674],
(-4841328524165418854,-4838895482794593338],
(628107265570603622,682509946926464280],
(7043245467621414187,7055126022831789025],
(624871765540463735,627374995781897409],
(9219228482330263660,9221294940422311559],
(-2335215188301493066,-2315034243278984017],
(-6216599212198827632,-6211460136507414133],
(-3276490559558850323,-3273110814046238767],
(7204991007334459472,7214826985711309418],
(1815809811279373566,1846961604192445001],
(8743912118048160970,8751518028513315549],
(-9204701745739426439,-9200185935622985719],
(7926527126882050773,7941554683778488797],
(-1307707180308444994,-1274682085495751899],
(8354147540115782875,8358523989614737607],
(-5418282332713406631,-5415077779309282099],
(2436459402559272117,2441988676982099299],
(3718536403335860823,3722612994041485455],
(613072756805503750,624871765540463735],
(3348338086790700840,3352211222405547568],
(7941554683778488797,7949493423692701952],
(-319434281654804937,-312029192727845573],
(6560852500852746428,6577389024948220270],
(3468872066705970665,3469611600245714638],
(-4691666557374653357,-4691655196509961176],
(1517867403379415135,1524487506534693145],
(-7422802504145504981,-7402342710717945820],
(1766173970230627147,1798177111825664424],
(-4402244320098942276,-4399211415161403124],
(-6024004998426219783,-6013031695922198269],
(-6930256177103656090,-6926607739769548590],
(-3578196517591043633,-3561058036097453011],
(-7923695406605667073,-7885733750925254770],
(-8063817427378208986,-8061632046535266293],
(-7341733690813664665,-7287847431444491429],
(-8327456680139422800,-8322944736494135341],
(885662136143255295,898069608029365525],
(-5555349213881020405,-5555145874347997903],
(-4895525453201761515,-4892963646364151947],
(4925572085497096406,4926057814710187377],
(7785700473508314561,7813617525484553901],
(-1598850926292630432,-1593204162764888475],
(-2583030464997858536,-2581717714589673598],
(7780476570449984013,7785700473508314561],
(-3558045406807923433,-3550343462935144866],
(7571721681971063633,7578715430118769516],
(8865905084309486257,8879414598472867800],
(-4984170160639031182,-4973560491294201055],
(-5469249030580263062,-5432196900285376733],
(-340636581410346966,-324449230085298889],
(9120471287581648922,9127588925404102522],
(8366987539912250171,8368772126617619299],
(-9125573241533654210,-9123870950381500195],
(1193377366975272040,1205407211929215537],
(8879414598472867800,8900714457425949896],
(7055126022831789025,7065799572034111661],
(-1588707445511226686,-1539910209795101605],
(6990922382177634221,7007948980474566617]] for XXX.[YYY, ZZZ]
INFO [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 - [repair
#0791a4a0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
/FAKE.35.57, /FAKE.34.90 on range [(-3078252446668567055,-3069777768104388887],
(3648099278098819362,3665843214181043507],
(5797678959887947223,5820452778888079981],
(-2642545928835445644,-2631494184435884123],
(-1612062874902452641,-1610692114449831135],
(-1707971415501754602,-1692411707392102715],
(291977720343160936,302780457185309086],
(-8155767539645304082,-8140296327332660530],
(5553475010776119070,5565820507856584918],
(3560286576304809625,3566372473455883512],
(5226123540640728306,5228400303590993685],
(3421729237821036215,3428685863718017921],
(1227377289103207735,1240348980976161342],
(3246531324407742399,3247845316596819400],
(5304633131413824443,5306083993155890840],
(-8619514398407319973,-8606439320645645548],
(5911017071945481872,5912872298340550901],
(-5637986723275926091,-5632042588860715397],
(-7132684921710202809,-7132365563419969085],
(8246342224027692193,8283424185055442516],
(-4044458382788374703,-4036186919446747968],
(-5613445605139257263,-5596969946550242654],
(1083135773045625799,1084813234330856986],
(-8140296327332660530,-8138240730730282332],
(2996936679493130945,3002752531484603869],
(-4490677154018906600,-4487029511866186581],
(8456154651893349894,8463606226270547769],
(-2656313914110802764,-2642545928835445644],
(7193156006022031982,7197068598281426786],
(8318481021259042556,8345872746994439720],
(-1692411707392102715,-1673006435564531865],
(1097191643473406518,1104630053604629743],
(-3613177001569108681,-3599878627600515791],
(-1662308889719802294,-1656482847672279831],
(-6811080679276087557,-6804940650782637386],
(7392275851743093948,7393304112887579707],
(4240513765329086484,4257716503015670881],
(-5036265422304136579,-5029426061972234042],
(-4230577892781670592,-4228053239714848467],
(-8159151100170546488,-8155767539645304082],
(-5492523660879132140,-5478464594772026508],
(3421125671176312710,3421729237821036215],
(5262611881402902860,5275876653180871852],
(1577722798587790899,1613215617842269344],
(6666156845317387896,6676801654642938276],
(5155814764198615180,5168412530555049843],
(-6824229014637940118,-6819748162411093540],
(-3666684985038092211,-3613177001569108681],
(-8874623295866980549,-8872063750661463000],
(-3599878627600515791,-3599733923716673512],
(-5960723066215554398,-5949787783010678970],
(258158907638948329,259127284829404565],
(6661762711488840391,6666156845317387896],
(-6367100347601878936,-6363911697345955296],
(-6250542414915159381,-6229550501272299394],
(-6917261891898915460,-6904807338337882298],
(-7449919126757571330,-7438527202807402113],
(-6904807338337882298,-6881332882762615208],
(-7942161058117423218,-7938222136384107977],
(781504901963705642,791302021433124535],
(6527741860207715339,6537050054260012793],
(7186200312153798494,7187207897161667549]] for XXX.[YYY, ZZZ]
INFO [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 - [repair
#07aaaae0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
/FAKE.35.153, /FAKE.34.90 on range [(3075097238958732453,3095612227388458535],
(-3791204388712221856,-3785200705970537637],
(8410714452405602873,8417924634749091961],
(5114137550939144346,5121098642749542165],
(2740032807677652551,2746972622905345611],
(2197312222134575375,2220347677927119914],
(4864988928009437565,4869276508135102594],
(-3451050463660365008,-3450091362870187260],
(7424782039380072835,7472858232992173041],
(-5338516343760774634,-5325637469358550877],
(2731359217194343571,2734544646630861913],
(382200642783712616,390184326306122373],
(430251086609083397,431821763463501369],
(-5338895526845396808,-5338516343760774634],
(-2677272836643942252,-2671813555202977248],
(-2132202487362637132,-2128425620095084071],
(2165546227633605752,2174360971273026818],
(2763720958753148061,2764848409944217454],
(3538741049060169597,3547917038270802802],
(-6741608677619654195,-6731111079555719429],
(2764848409944217454,2779298419000989854],
(374883901669627753,382200642783712616],
(-2706148319144852162,-2694312255222799319],
(7132646356350938192,7136352706932176819],
(194821142331236531,228318513249291325],
(-2047674613927860008,-2032759017860426681],
(-2032759017860426681,-2011077831196488705],
(7472858232992173041,7477981949405760270],
(431821763463501369,449911816694531926],
(4347942912351042626,4349624658068191606],
(311672118330363382,320380777495445406],
(3099877433733294763,3112419968835610433],
(-7182182373680571317,-7177421545891675761],
(-6742074278675667310,-6741608677619654195],
(-2770159943814078393,-2757633164368231752],
(3095612227388458535,3099877433733294763],
(-3741816028443553809,-3741630281022069399],
(-738951315633674297,-727109613092919188],
(4851195187346961355,4864988928009437565],
(-769229207399684543,-744901822064683937],
(-4143548408671617625,-4141029975015318194],
(-7957562451432593113,-7952882742232241654],
(-511680378144663402,-494227476671052554],
(-832115128813031544,-819699210830328959],
(6784361874104123161,6792435444931864417],
(-8284973647739186524,-8256379601422884439],
(-5321596150219749918,-5309942391653520721],
(4723799060589319973,4734570102349898748],
(5077208816402461873,5091797538025862046],
(7487792874019513006,7489186250794409815],
(-1647123563569159073,-1620470309759457440],
(2779298419000989854,2784870878309140225],
(320380777495445406,344020839234233467],
(3112419968835610433,3119057615014897057],
(-8506504972343997648,-8502433630545238869],
(-2810980092222625726,-2810271001960433176],
(344020839234233467,347093216302465566],
(-8545779729891984163,-8519832061775919840],
(2633165392330429557,2680549740474631393],
(6391794517057168460,6431615673158447973],
(6440766036098716417,6458083094779720657],
(-8735147357564473077,-8730522574908969951],
(7484355688408489651,7487792874019513006],
(4325444310827485194,4347942912351042626],
(2220347677927119914,2223568835084606157],
(6802149418704156481,6819713717509862346],
(4827439895247379322,4851195187346961355],
(-4016529712384875309,-4009393333251988044],
(2018021481024026660,2049270980733207626]] for XXX.[YYY, ZZZ]
ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471
ActiveRepairService.java:554 - Cannot start multiple repair sessions over the
same sstables
ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 -
Failed creating a merkle tree for [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0
on XXX/YYY, [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045],
(-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908],
(3732113975108886193,3743105867152786342],
(4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372],
(-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230],
(-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102],
(-4650650128572424858,-4650260492494258461],
(1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115],
(8114903118676615937,8132992506844206601],
(6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136],
(-167838948111369123,-141862581573028594],
(481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723],
(1659668187498418295,1679582585970705122],
(-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292],
(-79866884352549492,-77667207866300333],
(8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189],
(3767057277953758442,3783780844370292025],
(-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524],
(-9005441616028750657,-9002220258513351832],
(8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932],
(4976122621177738811,4987871287137312689],
(6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158],
(6895951547735922309,6899152466544114890],
(-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485],
(3824272999898372667,3829315045986248983],
(8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597],
(-956691623200349709,-946602525018301692],
(-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936],
(8154760186218662205,8157079055586089583],
(3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260],
(-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222],
(4964519403172053705,4970446606512414858],
(-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885],
(-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912],
(3466596314109623830,3468144326691723333],
(-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for details)
INFO [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176 -
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Received merkle tree for YYY
from /FAKE.33.64
ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 - Exception
in thread Thread[Repair#1:1,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.cassandra.exceptions.RepairException: [repair
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
[(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045],
(-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908],
(3732113975108886193,3743105867152786342],
(4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372],
(-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230],
(-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102],
(-4650650128572424858,-4650260492494258461],
(1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115],
(8114903118676615937,8132992506844206601],
(6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136],
(-167838948111369123,-141862581573028594],
(481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723],
(1659668187498418295,1679582585970705122],
(-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292],
(-79866884352549492,-77667207866300333],
(8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189],
(3767057277953758442,3783780844370292025],
(-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524],
(-9005441616028750657,-9002220258513351832],
(8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932],
(4976122621177738811,4987871287137312689],
(6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158],
(6895951547735922309,6899152466544114890],
(-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485],
(3824272999898372667,3829315045986248983],
(8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597],
(-956691623200349709,-946602525018301692],
(-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936],
(8154760186218662205,8157079055586089583],
(3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260],
(-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222],
(4964519403172053705,4970446606512414858],
(-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885],
(-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912],
(3466596314109623830,3468144326691723333],
(-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
at
com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
~[guava-18.0.jar:na]
at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_102]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
~[apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
[(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045],
(-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908],
(3732113975108886193,3743105867152786342],
(4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372],
(-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230],
(-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102],
(-4650650128572424858,-4650260492494258461],
(1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115],
(8114903118676615937,8132992506844206601],
(6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136],
(-167838948111369123,-141862581573028594],
(481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723],
(1659668187498418295,1679582585970705122],
(-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292],
(-79866884352549492,-77667207866300333],
(8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189],
(3767057277953758442,3783780844370292025],
(-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524],
(-9005441616028750657,-9002220258513351832],
(8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932],
(4976122621177738811,4987871287137312689],
(6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158],
(6895951547735922309,6899152466544114890],
(-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485],
(3824272999898372667,3829315045986248983],
(8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597],
(-956691623200349709,-946602525018301692],
(-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936],
(8154760186218662205,8157079055586089583],
(3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260],
(-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222],
(4964519403172053705,4970446606512414858],
(-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885],
(-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912],
(3466596314109623830,3468144326691723333],
(-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
at
org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_102]
... 4 common frames omitted
WARN [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 - [repair
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0] YYY sync failed
ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 -
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Session completed with the
following error
org.apache.cassandra.exceptions.RepairException: [repair
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
[(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045],
(-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908],
(3732113975108886193,3743105867152786342],
(4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372],
(-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230],
(-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102],
(-4650650128572424858,-4650260492494258461],
(1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115],
(8114903118676615937,8132992506844206601],
(6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136],
(-167838948111369123,-141862581573028594],
(481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723],
(1659668187498418295,1679582585970705122],
(-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292],
(-79866884352549492,-77667207866300333],
(8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189],
(3767057277953758442,3783780844370292025],
(-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524],
(-9005441616028750657,-9002220258513351832],
(8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932],
(4976122621177738811,4987871287137312689],
(6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158],
(6895951547735922309,6899152466544114890],
(-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485],
(3824272999898372667,3829315045986248983],
(8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597],
(-956691623200349709,-946602525018301692],
(-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936],
(8154760186218662205,8157079055586089583],
(3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260],
(-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222],
(4964519403172053705,4970446606512414858],
(-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885],
(-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912],
(3466596314109623830,3468144326691723333],
(-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
at
org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.14.jar:3.0.14]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose it
to anyone else. If you received it in error please notify us immediately and
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a
company registered in Linz whose registered office is at 4040 Linz, Austria,
Freist<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>ädterstra<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>ße
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>
--
-----------------
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com<http://www.thelastpickle.com/>
The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose it
to anyone else. If you received it in error please notify us immediately and
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a
company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>
--
-----------------
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com<http://www.thelastpickle.com/>
The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose it
to anyone else. If you received it in error please notify us immediately and
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a
company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
