Thanks for the update :) A
----------------- Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 7/08/2013, at 7:03 AM, sankalp kohli <kohlisank...@gmail.com> wrote: > @Aaron > This problem happens when you drop and recreate a keyspace with the same name > and you do it very quickly. I have also filed a JIRA for it > > https://issues.apache.org/jira/browse/CASSANDRA-5843 > > > On Tue, Aug 6, 2013 at 10:31 AM, Keith Wright <kwri...@nanigans.com> wrote: > The file does not appear on disk and the permissions are definitely correct. > We have seen the file in snapshots. This is completely blocking us from > adding the new node. How can we recover? Just run repairs? > > Thanks > > From: Aaron Morton <aa...@thelastpickle.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Tuesday, August 6, 2013 4:06 AM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Unable to bootstrap node > >> Caused by: java.io.FileNotFoundException: >> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >> (No such file or directory) >> at java.io.RandomAccessFile.open(Native Method) >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >> at >> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) >> at >> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init> > This is somewhat serous, specially if it's from the a bug in dropping tables. > Though I would expect that would show up for a lot of people. > > Does the file exist on disk? > Are the permissions correct ? > > IMHO you need to address this issue on the existing nodes before worrying > about the new node. > > Cheers > > ----------------- > Aaron Morton > Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 6/08/2013, at 1:25 PM, sankalp kohli <kohlisank...@gmail.com> wrote: > >> Let me know if this fixes the problem? >> >> >> On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisank...@gmail.com> wrote: >> So the problem is that when you dropped and recreated the table with the >> same name, some how the old CFStore object was not purged. So now there were >> two objects which caused same sstable to have 2 SSTableReader object. >> >> The fix is to find all nodes which is emitting this FileNotFound Exception >> and restart them. >> >> In your case, restart the node which is serving the data and emitting >> FileNotFound exception. >> >> Once this is up, again restart the bootstrapping node with bootstrap >> argument. Now it will successfully stream the data. >> >> >> On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwri...@nanigans.com> wrote: >> >> Yes we likely dropped and recreated tables. If we stop the sending node, >> what will happen to the bootstrapping node? >> >> sankalp kohli <kohlisank...@gmail.com> wrote: >> >> Hi, >> The problem is that the node sending the stream is hitting this >> FileNotFound exception. You need to restart this node and it should fix the >> problem. >> >> Are you seeing lot of FileNotFoundExceptions? Did you do any schema change >> recently? >> >> Sankalp >> >> >> On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwri...@nanigans.com> wrote: >> Hi all, >> >> I have been trying to bootstrap a new node into my 7 node 1.2.4 C* >> cluster with Vnodes RF3 with no luck. It gets close to completing and then >> the streaming just stalls with streaming at 99% from 1 or 2 nodes. >> Nodetool netstats shows the items that have yet to stream but the logs on >> the new node do not show any errors. I tried shutting down then node, >> clearing all data/commit logs/caches, and re-boot strapping with no luck. >> The nodes that are hanging sending the data only have the error below but >> that's related to compactions (see below) although it is one of the files >> that is waiting to be sent. I tried nodetool scrub on the column family >> with the missing item but got an error indicating it could not get a hard >> link. Any ideas? We were able to bootstrap one of the new nodes with no >> issues but this other one has been a real pain. Note that when the new node >> is joining the cluster, it does not appear in nodetool status. Is that >> expected? >> >> Thanks all, my next step is to try getting a new IP for this machine, my >> thought being that the cluster doesn't like me continuing to attempt to >> bootstrap the node repeatedly each time getting a new host id. >> >> [kwright@lxpcas008 ~]$ nodetool netstats | grep >> rts-40301_feedProducts-ib-1-Data.db >> rts: >> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >> sections=73 progress=0/1884669 - 0% >> >> ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java (line >> 174) Exception in thread Thread[ReadStage:427,5,main] >> java.lang.RuntimeException: java.io.FileNotFoundException: >> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >> (No such file or directory) >> at >> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) >> at >> org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57) >> at >> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41) >> at >> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976) >> at >> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98) >> at >> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117) >> at >> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64) >> at >> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) >> at >> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) >> at >> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133) >> at >> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) >> at >> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) >> at >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) >> at >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) >> at org.apache.cassandra.db.Table.getRow(Table.java:347) >> at >> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) >> at >> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44) >> at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:722) >> Caused by: java.io.FileNotFoundException: >> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >> (No such file or directory) >> at java.io.RandomAccessFile.open(Native Method) >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >> at >> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) >> at >> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75) >> at >> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42) >> ... 20 more >> >> >> > >
