> Caused by: java.io.FileNotFoundException: > /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init> This is somewhat serous, specially if it's from the a bug in dropping tables. Though I would expect that would show up for a lot of people.
Does the file exist on disk? Are the permissions correct ? IMHO you need to address this issue on the existing nodes before worrying about the new node. Cheers ----------------- Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 6/08/2013, at 1:25 PM, sankalp kohli <kohlisank...@gmail.com> wrote: > Let me know if this fixes the problem? > > > On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisank...@gmail.com> wrote: > So the problem is that when you dropped and recreated the table with the same > name, some how the old CFStore object was not purged. So now there were two > objects which caused same sstable to have 2 SSTableReader object. > > The fix is to find all nodes which is emitting this FileNotFound Exception > and restart them. > > In your case, restart the node which is serving the data and emitting > FileNotFound exception. > > Once this is up, again restart the bootstrapping node with bootstrap > argument. Now it will successfully stream the data. > > > On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwri...@nanigans.com> wrote: > Yes we likely dropped and recreated tables. If we stop the sending node, > what will happen to the bootstrapping node? > > sankalp kohli <kohlisank...@gmail.com> wrote: > > Hi, > The problem is that the node sending the stream is hitting this > FileNotFound exception. You need to restart this node and it should fix the > problem. > > Are you seeing lot of FileNotFoundExceptions? Did you do any schema change > recently? > > Sankalp > > > On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwri...@nanigans.com> wrote: > Hi all, > > I have been trying to bootstrap a new node into my 7 node 1.2.4 C* cluster > with Vnodes RF3 with no luck. It gets close to completing and then the > streaming just stalls with streaming at 99% from 1 or 2 nodes. Nodetool > netstats shows the items that have yet to stream but the logs on the new node > do not show any errors. I tried shutting down then node, clearing all > data/commit logs/caches, and re-boot strapping with no luck. The nodes that > are hanging sending the data only have the error below but that's related to > compactions (see below) although it is one of the files that is waiting to be > sent. I tried nodetool scrub on the column family with the missing item but > got an error indicating it could not get a hard link. Any ideas? We were > able to bootstrap one of the new nodes with no issues but this other one has > been a real pain. Note that when the new node is joining the cluster, it > does not appear in nodetool status. Is that expected? > > Thanks all, my next step is to try getting a new IP for this machine, my > thought being that the cluster doesn't like me continuing to attempt to > bootstrap the node repeatedly each time getting a new host id. > > [kwright@lxpcas008 ~]$ nodetool netstats | grep > rts-40301_feedProducts-ib-1-Data.db > rts: > /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db > sections=73 progress=0/1884669 - 0% > > ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java (line 174) > Exception in thread Thread[ReadStage:427,5,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db > (No such file or directory) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) > at > org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57) > at > org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41) > at > org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) > at > org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) > at org.apache.cassandra.db.Table.getRow(Table.java:347) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.FileNotFoundException: > /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42) > ... 20 more > > >
