Let me know if this fixes the problem?
On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisank...@gmail.com>wrote: > So the problem is that when you dropped and recreated the table with the > same name, some how the old CFStore object was not purged. So now there > were two objects which caused same sstable to have 2 SSTableReader object. > > The fix is to find all nodes which is emitting this FileNotFound Exception > and restart them. > > In your case, restart the node which is serving the data and emitting > FileNotFound exception. > > Once this is up, again restart the bootstrapping node with bootstrap > argument. Now it will successfully stream the data. > > > On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwri...@nanigans.com>wrote: > >> Yes we likely dropped and recreated tables. If we stop the sending node, >> what will happen to the bootstrapping node? >> >> sankalp kohli <kohlisank...@gmail.com> wrote: >> >> >> Hi, >> The problem is that the node sending the stream is hitting this >> FileNotFound exception. You need to restart this node and it should fix the >> problem. >> >> Are you seeing lot of FileNotFoundExceptions? Did you do any schema >> change recently? >> >> Sankalp >> >> >> On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwri...@nanigans.com>wrote: >> >>> Hi all, >>> >>> I have been trying to bootstrap a new node into my 7 node 1.2.4 C* >>> cluster with Vnodes RF3 with no luck. It gets close to completing and then >>> the streaming just stalls with streaming at 99% from 1 or 2 nodes. >>> Nodetool netstats shows the items that have yet to stream but the logs on >>> the new node do not show any errors. I tried shutting down then node, >>> clearing all data/commit logs/caches, and re-boot strapping with no luck. >>> The nodes that are hanging sending the data only have the error below but >>> that's related to compactions (see below) although it is one of the files >>> that is waiting to be sent. I tried nodetool scrub on the column family >>> with the missing item but got an error indicating it could not get a hard >>> link. Any ideas? We were able to bootstrap one of the new nodes with no >>> issues but this other one has been a real pain. Note that when the new >>> node is joining the cluster, it does not appear in nodetool status. Is >>> that expected? >>> >>> Thanks all, my next step is to try getting a new IP for this machine, >>> my thought being that the cluster doesn't like me continuing to attempt to >>> bootstrap the node repeatedly each time getting a new host id. >>> >>> [kwright@lxpcas008 ~]$ nodetool netstats | grep >>> rts-40301_feedProducts-ib-1-Data.db >>> rts: >>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >>> sections=73 progress=0/1884669 - 0% >>> >>> ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java >>> (line 174) Exception in thread Thread[ReadStage:427,5,main] >>> java.lang.RuntimeException: java.io.FileNotFoundException: >>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >>> (No such file or directory) >>> at >>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) >>> at >>> org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57) >>> at >>> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41) >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976) >>> at >>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98) >>> at >>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117) >>> at >>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64) >>> at >>> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) >>> at >>> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) >>> at >>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133) >>> at >>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) >>> at org.apache.cassandra.db.Table.getRow(Table.java:347) >>> at >>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) >>> at >>> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44) >>> at >>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:722) >>> Caused by: java.io.FileNotFoundException: >>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >>> (No such file or directory) >>> at java.io.RandomAccessFile.open(Native Method) >>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >>> at >>> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) >>> at >>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75) >>> at >>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42) >>> ... 20 more >>> >> >> >
