My colleague (Richard Ney) has already been in touch with you on a couple of
other issues we’ve seen in the past. Our development been trying to track down
some new issues we’ve been seeing on one of our pre-prod environments where
we’ve been having consistent failures very often (every day or every 2-3 days),
even if load/number of transactions are very light. We’re running a 2 data
center deployment with 3 nodes in each data center. Our tables are setup with
replication factor = 2 and we have 16G dedicated to the heap with the G1GC for
garbage collection. Our systems are AWS M4.2xlarge with 8 CPUs and 32GB of RAM
and we have 2 general purpose EBS volumes on each node of 500GB each. Once we
hit this it seems like the only way to recover is to shutdown the cluster and
restart. Running repairs after the restart often results in failures and we
pretty much end up having to truncate the tables before starting up clean
again. We are not sure if the two are inter-related. We pretty much see the
same issue on all the nodes. If anyone has any tips or any suggestions on how
to diagnose this further, it will help a great deal! The issues are:
Issue 1: Once the errors occur they just repeat for a bit followed by the
errors in issue 2.
INFO [CompactionExecutor:165] 2017-01-08 08:32:39,915 AutoSavingCache.java:386
- Saved KeyCache (63 items) in 5 ms
INFO [IndexSummaryManager:1] 2017-01-08 08:32:41,438
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [HANDSHAKE-ahldataslave4.bos.manhattan.aspect-cloud.net/10.184.8.224]
2017-01-08 09:30:03,988 OutboundTcpConnection.java:505 - Handshaking version
with ahldataslave4.bos.manhattan.aspect-cloud.net/10.184.8.224
INFO [IndexSummaryManager:1] 2017-01-08 09:32:41,440
IndexSummaryRedistribution.java:74 - Redistributing index summaries
WARN [SharedPool-Worker-9] 2017-01-08 10:30:00,116 BatchStatement.java:289 -
Batch of prepared statements for [manhattan.rcmessages] is of size 9264,
exceeding specified threshold of 5120 by 4144.
INFO [IndexSummaryManager:1] 2017-01-08 10:32:41,442
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 11:32:41,443
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [CompactionExecutor:162] 2017-01-08 12:32:39,914 AutoSavingCache.java:386
- Saved KeyCache (108 items) in 4 ms
INFO [IndexSummaryManager:1] 2017-01-08 12:32:41,446
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 13:32:41,448
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 14:32:41,450
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 15:32:41,451
IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [CompactionExecutor:170] 2017-01-08 16:32:39,915 AutoSavingCache.java:386
- Saved KeyCache (109 items) in 4 ms
INFO [IndexSummaryManager:1] 2017-01-08 16:32:41,453
IndexSummaryRedistribution.java:74 - Redistributing index summaries
WARN [SharedPool-Worker-4] 2017-01-08 17:30:45,048
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-4,5,main]: {}
java.lang.AssertionError: null
at org.apache.cassandra.db.rows.BufferCell.<init>(BufferCell.java:49)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:88)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:83)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.purge(BufferCell.java:175)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.ComplexColumnData.lambda$purge$107(ComplexColumnData.java:165)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.ComplexColumnData.transformAndFilter(ComplexColumnData.java:170)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:165)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:43)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.lambda$purge$102(BTreeRow.java:333)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:338)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:333)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:116)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:133)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Issue 2:
WARN [SharedPool-Worker-2] 2017-01-09 05:18:58,880
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.NullPointerException: null
WARN [SharedPool-Worker-1] 2017-01-09 05:19:14,220
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.NullPointerException: null
WARN [SharedPool-Worker-1] 2017-01-09 05:19:29,876
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
WARN [SharedPool-Worker-1] 2017-01-09 05:19:45,217
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
WARN [SharedPool-Worker-2] 2017-01-09 05:20:00,241
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
Issue 3: Periodic warnings about leaks (this was during a repair) - we had an
OOM occurrence about a week ago:
INFO [Thread-9] 2017-01-09 15:01:11,828 RepairSession.java:237 - [repair
#7593ad41-d67c-11e6-a76f-f1d6fd321f0d] new session: will sync
ahldataslave1.bos.manhattan.aspect-cloud.net/10.184.8.151, /10.184.8.137,
/10.184.8.219, /10.184.8.224 on range
[(4815877001221178276,4825317823493984218],
(3375739423664266689,3378070249246945856],
(-7393599445401300009,-7393031105803218575],
(-8130357422229977058,-8127934975152946543],
(-4611683212167667055,-4597307445147354114],
(4829191785668845783,4847928037092763493],
(5522544388079333104,5528633388915935404],
(1021118078158195547,1032730403399741255],
(-20739833830226207,-14719054523073734],
(-6205449048248449637,-6203730539323274527],
(45435932670861622,69554714149031706], (148647262653106287,181904829526383862],
(-854190757691833472,-854087983203765538],
(-7714687510919982336,-7700463340346726648],
(-360739232930366185,-339249720009847821],
(524595904743716879,537371267474489752],
(8185882494697143291,8187345181917158585],
(181904829526383862,190096166859570216],
(-1152328644109499972,-1134711681717758961],
(-586016578220869954,-566687033455187734],
(-4597307445147354114,-4541034774723552824],
(-4488655272471923458,-4481233629800439836],
(3111053603069111732,3121096557789915147],
(-1167831376731724655,-1152328644109499972],
(-6302582962631272172,-6290099341099600242],
(8082429902489189721,8084233962135014968],
(-2781993705200227829,-2777492754784519178],
(5996816756578981045,6022011570976925279],
(4404919629779433734,4422232637329318322],
(-643317786131863042,-611582394998065454],
(4514099138058135901,4514906323961126580],
(7273471227296986256,7316794306200362675],
(4462984376163219141,4464984714163480487],
(-2390509810947844616,-2382282465250439153],
(-7773729052257068390,-7772926086336061342],
(-161569209320145017,-149914302996366392],
(1392011488470148469,1403436343776848669],
(546940383302067460,574197777751440495],
(6624568645457217871,6629411218351717560],
(-1134711681717758961,-1133755953039077472],
(-7248281701751445296,-7243217407431962906],
(2374653705052569734,2377396978942737768],
(5619550734131192401,5628464353424177974],
(-192925818286153751,-181902834109073167],
(5896301416900157412,5911734690958039138],
(4422232637329318322,4462984376163219141],
(-4524564105581793254,-4504900851715014673],
(-2115528876758613050,-2112322288682500569],
(5504098260114685179,5515264579319185925],
(35614519826275404,45435932670861622],
(-855913784497695841,-854190757691833472],
(5736245318761313405,5738445441111493493],
(4807463771329802801,4815877001221178276],
(-2884278504117308670,-2881329894351532736],
(-207545241327995479,-207389332352346254],
(-181902834109073167,-172819893538312319],
(-7636862515759014032,-7626686102011136779],
(-8140870857704067579,-8130357422229977058],
(-2342122616740401439,-2335433185444036860],
(-2881329894351532736,-2880349360092682175],
(-611582394998065454,-587918354120388686],
(-7320730625612121443,-7317644215584737598],
(-13045270760172222,8260384715982984],
(-8668935926714005039,-8657580917928283146],
(8260384715982984,35614519826275404],
(2861067483327829799,2864748323206840859],
(3127114336475141934,3139163273099443478],
(-7657420721193956821,-7636862515759014032],
(1032730403399741255,1032951438204062821],
(-1133755953039077472,-1130095656542041882],
(5515264579319185925,5522544388079333104],
(6826479291574328528,6828593995008227175],
(658600138550738740,680809612853177633],
(2364271158745468459,2374653705052569734],
(2864748323206840859,2868871349367827459],
(-7741910906325473118,-7738712583959975589],
(2749562470526527976,2765502980561577063],
(4569656624441412194,4581277061087905130],
(-1130095656542041882,-1102822102812415509],
(2357686856375407577,2364271158745468459],
(2875146491882456097,2913694653712675567],
(-2365721528619247416,-2362417204623405805],
(-7700463340346726648,-7657420721193956821],
(-1545400958003324469,-1527145992709891047],
(6595468060656389384,6611122695943865425],
(-2362417204623405805,-2360989147149959339],
(680809612853177633,691564499046700936],
(1048818468740031469,1051848236340749987],
(-6290099341099600242,-6284139189426371408],
(-430511209136180745,-417631748808875679],
(-2127191227331961211,-2115528876758613050],
(-587918354120388686,-586016578220869954],
(5950659959556897324,5972536818860340742],
(7228681112293176899,7273471227296986256],
(-92115275177849392,-73078256610822906],
(-2150182737057335813,-2131084968707606517],
(-5931891237188071234,-5925084712702354100],
(1854063092500381970,1894128417137542926],
(3139163273099443478,3144353879892041737],
(2795130016444690338,2849989872521310320],
(69554714149031706,73259277905023853],
(-198884270246085916,-192925818286153751],
(2301867852127537258,2344138511230415627],
(-2360989147149959339,-2342122616740401439],
(2868871349367827459,2875146491882456097],
(-8520120040026037727,-8484758828657908805],
(1044175704807438060,1048818468740031469],
(3658700192210846721,3709106004630729567],
(-8715352510373593584,-8695484052773103051],
(4825317823493984218,4829191785668845783],
(5881177751245354064,5896301416900157412],
(-1557733846751863547,-1545400958003324469],
(3764340229927726801,3772352737530248438],
(5723087139361841478,5736245318761313405],
(-339249720009847821,-325063001525001753],
(4581277061087905130,4585591969590606261],
(2849989872521310320,2861067483327829799],
(-201641945389990466,-198884270246085916],
(3062119490734517386,3070339219117072282],
(6611989977644669113,6624568645457217871],
(-6303561532875621980,-6302582962631272172],
(-2880349360092682175,-2815907193866317595],
(-2210673969380244356,-2200655423201071981],
(6920406973111913689,6944396390330370813],
(3070339219117072282,3089918739923672617],
(1018479337556218583,1021118078158195547],
(-22373917471115570,-20802161777184740],
(2349224556257120691,2357686856375407577],
(1610291395978708348,1610350862682840167],
(-7774308296105089774,-7773729052257068390],
(-207389332352346254,-201641945389990466],
(-4504900851715014673,-4488655272471923458],
(-913362829471550684,-900550299179500948],
(574197777751440495,575687259162456966],
(-209852186312631406,-207545241327995479],
(3961560378473489131,4029122901768525705],
(1059898575782522398,1065913666232086022],
(6611122695943865425,6611989977644669113],
(-2815907193866317595,-2781993705200227829],
(-8149621014270042242,-8140870857704067579],
(-999997390934570392,-998678768267637866],
(2377396978942737768,2384267095612056863],
(-2267001694418802171,-2210871230482048364],
(-2131084968707606517,-2127191227331961211],
(3144353879892041737,3183343610940682057],
(-2210871230482048364,-2210673969380244356],
(4399306102077085676,4404919629779433734],
(537371267474489752,546940383302067460],
(-7722143244670988745,-7714687510919982336],
(3754196018068729018,3763367677110000435],
(-5925084712702354100,-5919412889821651249],
(4585591969590606261,4590124192625756967],
(-1189315862160445133,-1167831376731724655],
(-14719054523073734,-13045270760172222],
(3772352737530248438,3791853264732937441],
(-149914302996366392,-138596249554837825],
(-1459586834232833970,-1455782887662096513],
(-7626686102011136779,-7613607518487725198],
(-2370342707880863831,-2365721528619247416],
(1051848236340749987,1059898575782522398],
(575687259162456966,587747968172054843],
(1846675642540253784,1854063092500381970],
(5911734690958039138,5950659959556897324],
(3709106004630729567,3754196018068729018],
(2274687735800823626,2301867852127537258],
(3763367677110000435,3764340229927726801],
(2765502980561577063,2768302382460911776],
(-4835698441475193016,-4821740368643417588],
(-4840592682190099042,-4835698441475193016],
(-900550299179500948,-898509802837897450],
(5972536818860340742,5996816756578981045],
(7316794306200362675,7316948326538779302],
(-20802161777184740,-20739833830226207],
(-8695484052773103051,-8693825425156218047],
(-6228811049824135380,-6205449048248449637],
(-8522744988679262888,-8520120040026037727],
(-1527145992709891047,-1522580222898594256],
(1065913666232086022,1070185844169898505],
(3121096557789915147,3127114336475141934]] for manhattan.[config, messages,
rcmessages, csseedmessages, metadata, rqmessages]
ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK
DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@8520181)
to class
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@110489339:/data/manhattan/messages-5e11ec20d49411e6ac36cb64da3c5bcf/ma-1-big
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK
DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2aa6af1)
to class
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1202250944:/data/manhattan/rcmessages-5f65e360d49411e6ac36cb64da3c5bcf/ma-1-big
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@5feebbbd) to class
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1990550669:/data/manhattan/rcmessages-5f65e360d49411e6ac36cb64da3c5bcf/ma-2-big
was not released before the reference was garbage collected
Issue 4: Failure on repair (cannot complete repair) - validation failures.
INFO [Thread-9] 2017-01-09 15:01:12,496 RepairSession.java:237 - [repair
#75f99b00-d67c-11e6-a76f-f1d6fd321f0d] new session: will sync
ahldataslave1.bos.manhattan.aspect-cloud.net/10.184.8.151, /10.184.8.136,
/10.184.8.219, /10.184.8.206 on range
[(7070473513877450523,7071785512108503562],
(7540323476844137856,7551069853951512536],
(6496851087781126402,6506947825241172690],
(4956723654110177663,4997988272614120716],
(6159728418399742238,6161785046042182181],
(-6111201816165886930,-6109358652076682027],
(6765434292326999735,6786216263996459971],
(5670991529662923445,5678922736379606122],
(-8368452819419163190,-8352559832808839636],
(-3579114434972148656,-3552744642655983590],
(7574005965138357735,7575976736757457093],
(3443342880213529041,3448513378658973391],
(-6374441519731158925,-6368083975098827965],
(1506273267531550913,1508155162580364084],
(-3841814855559153943,-3835938678769454001],
(9170965416227969215,9181174655986386320],
(-6589650497263617394,-6578994961053045565],
(-7526495575912144496,-7518448902193774650],
(3448513378658973391,3466857105667177973],
(7553098986919382391,7574005965138357735],
(-7927877550785715833,-7927648953253692861],
(8525761079018902513,8533541215797136240],
(-9020129600263662082,-9010501329825003246],
(1090724998362773160,1097939939665235297],
(6495099300248039972,6496851087781126402],
(6161785046042182181,6230901034893815059],
(-7463338857564269606,-7459175745770455573],
(-5365113529531278290,-5358403398116793300],
(2249898065901245108,2256752991177466019],
(427458615961451939,456792456721828910],
(-7518448902193774650,-7476577165771266744],
(8496964666711176665,8503454084084785332],
(1508155162580364084,1531943799366406071],
(1623939448991457330,1628591069075100791],
(2500969373198505595,2510256322219581447],
(1487072216030226303,1506273267531550913],
(-8842784826526224555,-8840216173643212475],
(2233348668129647030,2247658708769712814],
(-8289206614243730701,-8287074180401539509],
(-7476577165771266744,-7470389682785988771],
(3509709684061378867,3512938839959595162],
(1296769759309759534,1302830049359124705],
(7575976736757457093,7612135532866611803],
(7539950122318008545,7540323476844137856],
(-1860975172751451123,-1853676643512030698],
(-8116786947316919605,-8107369675963369796],
(-5576866678991811211,-5548824138592035245],
(-9211881728677655622,-9209566963086044390],
(7489367377277174970,7511259639253940249],
(1302830049359124705,1310010318365119318],
(-8750733220711433842,-8741408722769214425],
(-6060376367842620341,-6049131555099770258],
(2476529278549364684,2476659930520260878],
(-4418742343319617797,-4384333951548488071],
(1930660747072472501,1946687557062422024],
(-3702380150896445622,-3691910222307497728],
(-4419703203609450987,-4418742343319617797],
(7551697213870628887,7553098986919382391],
(-7909869864636951454,-7904683595914140744],
(2492882420790179151,2500969373198505595],
(759685512451005610,773693246847159219],
(722689136592207691,730821328907957274],
(5219025897237634110,5229566135848567231],
(4149517035748854868,4162833917677490963],
(1326132699179277905,1349642653396670633],
(-3552744642655983590,-3549073980410969181],
(-4456698363145934960,-4419703203609450987],
(-5482989026069906323,-5481213659503324794],
(279535406932511121,297415597645494493],
(-9204266745720873498,-9161795161892156657],
(6786216263996459971,6786883290959946615],
(1635462812102810232,1638975594001298412],
(1531943799366406071,1546279755200334429],
(-3851094911406737954,-3841814855559153943],
(8488130353439482734,8496964666711176665],
(9146936609495586767,9148689706364420814],
(-3674452659583777301,-3669778494255064100],
(1546279755200334429,1554992050513055527],
(2969501640373218313,2975603144417796179],
(-3865994019611705472,-3851094911406737954],
(1119818899797211952,1121114266048498658],
(5555981883036126656,5559877922798642583],
(-9161795161892156657,-9148239227447645822],
(-1988480938545038453,-1972594091279542463],
(-8457443005098123276,-8426466451426413266],
(7527886006442742638,7539950122318008545],
(1482724986864293552,1487072216030226303],
(218814708185728764,222373457441562248],
(-1844742172330357357,-1834186452017037646],
(-7914786200548736052,-7909869864636951454],
(1554992050513055527,1556471756149638584],
(2247658708769712814,2249241162600622119],
(2249241162600622119,2249898065901245108],
(-6368083975098827965,-6357129727004451888],
(1349642653396670633,1372390693080139703],
(-5628093303198956105,-5602276528881256607],
(-5602276528881256607,-5576866678991811211],
(-1972594091279542463,-1932380490869749103],
(6746187069375198686,6759453964579921534],
(-9209566963086044390,-9204266745720873498],
(2969169700471726709,2969501640373218313],
(-1846725725884736711,-1844742172330357357],
(8466845312658849111,8478632986527402947],
(6786883290959946615,6786909234664957611],
(-8105207391882034800,-8058995460560987960],
(-5481213659503324794,-5480151738028046165],
(6152943806705127166,6159728418399742238],
(-7931416471770778308,-7927877550785715833],
(-7444305315017293180,-7438698090368547807],
(-5405765462772655927,-5365113529531278290],
(7468218535666030613,7489367377277174970],
(7370990959551251700,7384554095789173771],
(9181174655986386320,9197356813903415489],
(-7891619558235712999,-7886926665775314135],
(4863660993721847898,4876110806493712229],
(5546813895398001993,5555981883036126656],
(6793672076817085638,6817463007248949484],
(730821328907957274,741073317079317196],
(1285297621466628116,1296769759309759534],
(-6109358652076682027,-6097749005908966679],
(-3160359268440132710,-3150255443388600113],
(-6067028212437311851,-6060376367842620341],
(3512938839959595162,3531669213417260709],
(-1641724835017736571,-1630212592242213627],
(-5631109618795517413,-5628093303198956105],
(6759453964579921534,6765434292326999735],
(8027648385044762960,8030142266857918084],
(1264145341476479882,1285297621466628116],
(1116504555019555851,1119818899797211952],
(-3139664283152392978,-3138393885745606369],
(-1652241560457928053,-1641724835017736571],
(8503454084084785332,8525761079018902513],
(-1853676643512030698,-1846725725884736711],
(1628591069075100791,1635462812102810232],
(-8840216173643212475,-8817712049837887184],
(-6357129727004451888,-6355060010614218705],
(-6097749005908966679,-6088616433680186631],
(-7904683595914140744,-7891619558235712999],
(-4952793941753266051,-4937049631932742604],
(-3150255443388600113,-3139664283152392978],
(2512801900471921714,2516325796899709313],
(2510256322219581447,2512801900471921714],
(-5638226176228799616,-5631109618795517413],
(-7886926665775314135,-7882937045572193145],
(6786909234664957611,6793672076817085638],
(-3691910222307497728,-3674452659583777301],
(-4867940821711541762,-4866946150328928181],
(7071785512108503562,7081793743588959944],
(4876110806493712229,4879643371134562502],
(6138845154266790671,6152943806705127166],
(-6355060010614218705,-6332373169802347251],
(-6125214696594291714,-6111201816165886930],
(-7470389682785988771,-7463338857564269606],
(-3598204681868075295,-3579114434972148656],
(-8107369675963369796,-8105207391882034800],
(7612135532866611803,7641727005250566009],
(9148689706364420814,9170965416227969215],
(6506947825241172690,6515762919020447377],
(-6088616433680186631,-6067028212437311851],
(741073317079317196,759685512451005610],
(-8352559832808839636,-8344010281020219217],
(-6049131555099770258,-6043632332003790850],
(3494099643169044463,3509709684061378867],
(3466857105667177973,3494099643169044463],
(7551069853951512536,7551697213870628887],
(2476659930520260878,2492882420790179151],
(-8842911646480810622,-8842784826526224555],
(193808271419729511,198960416318160202],
(-4873803694032119901,-4867940821711541762],
(-7569775426219162591,-7569610743922997927],
(4730391117872973381,4732570096640666262],
(9137762516596245936,9146936609495586767],
(-9148239227447645822,-9137147760354721369],
(-8296203956052422690,-8289206614243730701],
(198960416318160202,218814708185728764]] for manhattan.[config, messages,
rcmessages, csseedmessages, metadata, rqmessages]
WARN [SharedPool-Worker-2] 2017-01-09 15:01:38,219
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.NullPointerException: null
WARN [SharedPool-Worker-2] 2017-01-09 15:04:41,678
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461)
~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.3.0.jar:3.3.0]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Regards,
DHRUVA GOPAL
sr. MANAGER, ENGINEERING
REPORTING, ANALYTICS AND BIG DATA
+1 408.325.2011 WORK
+1 408.219.1094 MOBILE
UNITED STATES
dhruva.go...@aspect.com<mailto:dhruva.go...@aspect.com>
aspect.com<http://www.aspect.com/>
[Description: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
This email (including any attachments) is proprietary to Aspect Software, Inc.
and may contain information that is confidential. If you have received this
message in error, please do not read, copy or forward this message. Please
notify the sender immediately, delete it from your system and destroy any
copies. You may not further disclose or distribute this email or its
attachments.
