Hi, There are no piece of code in Cassandra that would remove this folder. You should start looking elsewhere, like other people mentioned (chef, ansible and so on), good luck
On 8 February 2018 at 22:54, test user <casst...@gmail.com> wrote: > Does anyone have more inputs on the missing hints folder, rather why it > gets deleted. > > Has anyone run into this scenario before? > > Regards, > Cassandra User > > On Wed, Feb 7, 2018 at 9:21 PM, test user <casst...@gmail.com> wrote: > >> The problem is even though, I get that O_RDONLY WARN message, if I try to >> navigate to the path where the hints folder is stored, the folder is not >> present. >> I cannot check the permissions on that folder, its already missing, got >> deleted somehow. >> >> I believe everything runs as the root user. >> >> I do see a lot of sstable activity performed by MemTableFlushWriter >> (ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable) >> before and after this WARN message. >> >> It is not a space issue, I checked that already. >> >> >> >> On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall <n...@thelastpickle.com> >> wrote: >> >>> >>> The environment is built using established images for Cassandra 3.10. >>>> Unfortunately the debug log does not indicate any errors before I start >>>> seeing the WARN for missing hints folder. I understand that hints file will >>>> be deleted after replay is complete, but not sure of the root cause of why >>>> the hints folder is getting deleted. >>>> When I look at the nodetool status or nodetool ring - it indicates that >>>> all nodes are up and running in normal state, no node went down. Also, I do >>>> not see anything the debug logs indicating that a node went down. In such a >>>> scenario, I am not sure why would HintsWriterExecutor would get triggered. >>>> >>>> >>> That error code (O_RDONLY) in the log message indicates that the hints >>> folder has had its permission bits set to read only. >>> >>> We've had several issues with some of the tools doing this type of thing >>> when they are run as the root user. Is this specific node one on which you >>> use any of the tools like sstableloader or similar? If so, are you running >>> them as root? >>> >>> Another thought - if it is on a different partition than the data >>> directory, is there free space left on the underlying device holding: >>> /var/lib/cassandra/hints? >>> >>> >>> -- >>> ----------------- >>> Nate McCall >>> Wellington, NZ >>> @zznate >>> >>> CTO >>> Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >> >> >
