One of our nodes keeps crashing continuously with out of memory errors. I
see the following error in the logs -
INFO 21:03:54,007 Creating new commitlog segment
/local3/logs/cassandra/commitlog/CommitLog-1348016634007.log
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
pages failed.
INFO 21:03:54,017 Replaying
/local3/logs/cassandra/commitlog/CommitLog-1347950471775.log,
/local3/logs/cassandra/commitlog/CommitLog-1347952280335.log,
/local3/logs/cassandra/commitlog/CommitLog-
1347954861373.log,
/local3/logs/cassandra/commitlog/CommitLog-1347957791957.log,
/local3/logs/cassandra/commitlog/CommitLog-1347959168686.log,
/local3/logs/cassandra/commitlog/CommitLog-1347961014
948.log, /local3/logs/cassandra/commitlog/CommitLog-1347968405068.log,
/local3/logs/cassandra/commitlog/CommitLog-1347972420459.log,
/local3/logs/cassandra/commitlog/CommitLog-1347975084979.log, /
local3/logs/cassandra/commitlog/CommitLog-1347975538081.log,
/local3/logs/cassandra/commitlog/CommitLog-1347976033450.log,
/local3/logs/cassandra/commitlog/CommitLog-1347976685447.log, /local3/log
s/cassandra/commitlog/CommitLog-1347977204225.log,
/local3/logs/cassandra/commitlog/CommitLog-1347977904344.log,
/local3/logs/cassandra/commitlog/CommitLog-1347978791835.log,
/local3/logs/cassandr
a/commitlog/CommitLog-1347979595214.log,
/local3/logs/cassandra/commitlog/CommitLog-1347980280043.log,
/local3/logs/cassandra/commitlog/CommitLog-1347980822272.log,
/local3/logs/cassandra/commitlo
g/CommitLog-1347981376135.log,
/local3/logs/cassandra/commitlog/CommitLog-1347982023403.log,
/local3/logs/cassandra/commitlog/CommitLog-1347982906942.log,
/local3/logs/cassandra/commitlog/CommitLo
g-1347983999163.log,
/local3/logs/cassandra/commitlog/CommitLog-1347985475186.log,
/local3/logs/cassandra/commitlog/CommitLog-1347987026500.log,
/local3/logs/cassandra/commitlog/CommitLog-13479885
20038.log, /local3/logs/cassandra/commitlog/CommitLog-1347990191955.log,
/local3/logs/cassandra/commitlog/CommitLog-1347991899744.log,
/local3/logs/cassandra/commitlog/CommitLog-1347993488753.log,
/local3/logs/cassandra/commitlog/CommitLog-1347994801952.log,
/local3/logs/cassandra/commitlog/CommitLog-1347995807921.log,
/local3/logs/cassandra/commitlog/CommitLog-1347996959596.log, /local3/l
ogs/cassandra/commitlog/CommitLog-1347997961248.log,
/local3/logs/cassandra/commitlog/CommitLog-1348005812088.log,
/local3/logs/cassandra/commitlog/CommitLog-1348015418757.log
ERROR 21:03:54,030 Exception encountered during startup.
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:254)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:124)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:196)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:195)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:336)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
Exception encountered during startup.
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:254)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:124)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:196)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:195)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:336)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
pages failed.
Has anybody seen this before?
-Rajesh
