Robert,
New development today:
FSReadError in
/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:95)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:41)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:323)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:185)
at
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer$1.runMayThrow(ParallelCompactionIterable.java:271)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:250)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:101)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87)
... 17 more
Followed by:
ERROR [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 StorageService.java (line 364) Stopping gossiper
WARN [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 StorageService.java (line 278) Stopping gossip by
operator request
INFO [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 Gossiper.java (line 1251) Announcing shutdown
After this the listeners are no longer available but the DB does not
officially die just hangs and needs a restart.
*From:* Robert Coli [mailto:rc...@eventbrite.com]
*Sent:* Monday, July 7, 2014 6:55 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Compaction causing listeners to stall
On Mon, Jul 7, 2014 at 5:20 AM, Bryon Spahn <bsp...@kitedesk.com> wrote:
I am experiencing a strange issue where we run a compaction job weekly and
as a result, the listeners stall. This is a single node cluster running on
an i2.2xl instance in AWS. We are getting the message:
There are almost no cases where it makes sense to run a single node of
Cassandra, especially in production.
*[StorageServiceShutdownHook]*
I bet you a donut that you're OOMing the JVM. Stop doing that, and your
Cassandra node will stop crashing.
https://issues.apache.org/jira/browse/CASSANDRA-7507
Is probably the case you have just hit.
Basically, in some pathological circumstances, the JVM will send Cassandra
a signal that it handles as if you were an operator attempting a clean
shutdown. This probably usually does not succeed, but may be worth a shot.
=Rob
