Does anyone have more inputs on the missing hints folder, rather why it gets deleted.
Has anyone run into this scenario before? Regards, Cassandra User On Wed, Feb 7, 2018 at 9:21 PM, test user <casst...@gmail.com> wrote: > The problem is even though, I get that O_RDONLY WARN message, if I try to > navigate to the path where the hints folder is stored, the folder is not > present. > I cannot check the permissions on that folder, its already missing, got > deleted somehow. > > I believe everything runs as the root user. > > I do see a lot of sstable activity performed by MemTableFlushWriter > (ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable) > before and after this WARN message. > > It is not a space issue, I checked that already. > > > > On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall <n...@thelastpickle.com> > wrote: > >> >> The environment is built using established images for Cassandra 3.10. >>> Unfortunately the debug log does not indicate any errors before I start >>> seeing the WARN for missing hints folder. I understand that hints file will >>> be deleted after replay is complete, but not sure of the root cause of why >>> the hints folder is getting deleted. >>> When I look at the nodetool status or nodetool ring - it indicates that >>> all nodes are up and running in normal state, no node went down. Also, I do >>> not see anything the debug logs indicating that a node went down. In such a >>> scenario, I am not sure why would HintsWriterExecutor would get triggered. >>> >>> >> That error code (O_RDONLY) in the log message indicates that the hints >> folder has had its permission bits set to read only. >> >> We've had several issues with some of the tools doing this type of thing >> when they are run as the root user. Is this specific node one on which you >> use any of the tools like sstableloader or similar? If so, are you running >> them as root? >> >> Another thought - if it is on a different partition than the data >> directory, is there free space left on the underlying device holding: >> /var/lib/cassandra/hints? >> >> >> -- >> ----------------- >> Nate McCall >> Wellington, NZ >> @zznate >> >> CTO >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> > >
