The hard drives have plenty of space left and working properly as far as I know. We’ve been running Cassandra for almost a year now. It’s the first time we’re seeing this problem. I see a JIRA for v2.1 but that is with NFS.. We’re not using NFS.
From: Carlos Rolo [mailto:r...@pythian.com] Sent: Thursday, July 16, 2015 8:59 AM To: user@cassandra.apache.org Subject: Re: cassandra repair error Hello, It seems that is a problem with Cassandra trying to delete a directory that is not empty. First time seeing this error. Are your harddrives with enough space and working correctly? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com<http://www.pythian.com/> On Thu, Jul 16, 2015 at 2:45 PM, Modha, Digant <digant.mo...@tdsecurities.com<mailto:digant.mo...@tdsecurities.com>> wrote: Hi, We’re using Cassandra 2.0.10 ( 2 DC, 3 Nodes each RF=3 for each DC). During one of the weekly repairs, we received the following error: ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 Validator.java (line 242) Failed creating a merkle tree for [repair #d2178ba0-2902-11e5-bd95-f14c61d86b85 on dmds/curve_dates, (-1942303675502999131,-1890400428284965630]], / (see log for details) ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 CassandraDaemon.java (line 199) Exception in thread Thread[ValidationExecutor:1280,1,main] FSWriteError in /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85 at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:122) at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:384) at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:488) at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1877) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:811) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63) at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:398) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.file.DirectoryNotEmptyException: /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85 at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) at java.nio.file.Files.delete(Files.java:1126) at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:118) ... 10 more ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,993 StorageService.java (line 364) Stopping gossiper WARN [ValidationExecutor:1280] 2015-07-12 22:18:10,993 StorageService.java (line 278) Stopping gossip by operator request INFO [ValidationExecutor:1280] 2015-07-12 22:18:10,993 Gossiper.java (line 1279) Announcing shutdown Has anybody seen this error? The drives are local. Once this happened, the other node performing repair maxed out the CPU and cluster became unresponsive. Thanks, dm TD Securities disclaims any liability or losses either direct or consequential caused by the use of this information. This communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. TD Securities is neither making any investment recommendation nor providing any professional or advisory services relating to the activities described herein. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice Any products described herein are (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured depository institution and (iii) subject to investment risks, including possible loss of the principal amount invested. The information shall not be further distributed or duplicated in whole or in part by any means without the prior written consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain investment banking activities of The Toronto-Dominion Bank and its subsidiaries. -- TD Securities disclaims any liability or losses either direct or consequential caused by the use of this information. This communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. TD Securities is neither making any investment recommendation nor providing any professional or advisory services relating to the activities described herein. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice Any products described herein are (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured depository institution and (iii) subject to investment risks, including possible loss of the principal amount invested. The information shall not be further distributed or duplicated in whole or in part by any means without the prior written consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain investment banking activities of The Toronto-Dominion Bank and its subsidiaries.
