That looks like a real problem, but it's not immediately clear to me what it is.
It's possible that it's a genuine disk failure (do you see other signs of that disk having read/write failures?), or more likely, some race in SASI that is causing the index file to be removed while it's being updated. If possible, I encourage you to open a JIRA with as much info as you can (the stacks are great, sanitized logs would be good too). JIRA is here: https://issues.apache.org/jira/browse/CASSANDRA You could also try upgrading to 3.11.1, there's always a chance it's been fixed (though I don't see anything in the 3.11.1 changelog that looks related). On 2017-12-09 00:41, Earl Lapus <earl.la...@gmail.com> wrote: > Hi, > > I have python program that inserts data into a Cassandra server, it runs > without reporting any problems and all inserts are successful. > However, I get recurring errors logs in the server every time I run my > python program for inserting data (see sample errors below). > > So, should I ignore these errors -OR- are these real problems that I should > be fixing? > > Cheers, > Earl > > > Error 1 - "Failed to build index segment..." > *** log snippet *** > ERROR [SASI-General:21] 2017-12-09 08:18:07,148 > PerSSTableIndexWriter.java:262 - Failed to build index segment > /var/lib/cassandra/data/rates_local/meta_raterow_buyer-14ffa180dcb811e79e82f58bd8d3766a/mc-21-big-SI_meta_raterow_buyer_effective_thru_idx.db > org.apache.cassandra.io.FSWriteError: java.io.IOException: Term - > '1970-01-01' belongs to more than 5 keys in SPARSE mode, which is not > allowed. at > org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:328) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:269) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:241) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:258) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$complete$1(PerSSTableIndexWriter.java:287) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_131] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_131] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_131] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > ~[na:1.8.0_131] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] > *** log snippet *** > > Error 2 - "Can't open index file at..." > *** log snippet *** > ERROR [CompactionExecutor:4] 2017-12-09 08:18:07,369 DataTracker.java:179 - > Can't open index file at > /var/lib/cassandra/data/rates_local/meta_raterow_buyer-14ffa180dcb811e79e82f58bd8[16/1905] > -21-big-SI_meta_raterow_buyer_effective_thru_idx.db, skipping. > java.lang.IllegalArgumentException: position: 455796228778837249, limit: 107 > at > org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:148) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:68) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.conf.DataTracker.getBuiltIndexes(DataTracker.java:159) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:68) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.index.sasi.SASIIndex.handleNotification(SASIIndex.java:319) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.lifecycle.Tracker.notifySSTablesChanged(Tracker.java:410) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.lifecycle.LifecycleTransaction.doCommit(LifecycleTransaction.java:227) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.io.sstable.SSTableRewriter.doCommit(SSTableRewriter.java:206) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doCommit(CompactionAwareWriter.java:105) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:200) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:185) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) > [apache-cassandra-3.11.0.jar:3.11.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_131] > *** log snippet *** > > Error 3 - "LEAK DETECTED..." > *** log snippet *** > ERROR [Reference-Reaper:1] 2017-12-09 08:18:08,783 Ref.java:224 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@126a6518) to class > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@750364681:/var/lib/cassandra/data/rates_local/meta_raterow_buyer-14ffa180dcb811e79e82f58bd8d3766a/mc-21-big > was not released before the reference was garbage collected > ERROR [Reference-Reaper:1] 2017-12-09 08:18:08,783 Ref.java:224 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@3a7af7f6) to class > org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@750364681:/var/lib/cassandra/data/rates_local/meta_raterow_buyer-14ffa180dcb811e79e82f58bd8d3766a/mc-21-big > was not released before the reference was garbage collected > *** log snippet *** > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
