The problem is even though, I get that O_RDONLY WARN message, if I try to navigate to the path where the hints folder is stored, the folder is not present. I cannot check the permissions on that folder, its already missing, got deleted somehow.
I believe everything runs as the root user. I do see a lot of sstable activity performed by MemTableFlushWriter (ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable) before and after this WARN message. It is not a space issue, I checked that already. On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall <n...@thelastpickle.com> wrote: > > The environment is built using established images for Cassandra 3.10. >> Unfortunately the debug log does not indicate any errors before I start >> seeing the WARN for missing hints folder. I understand that hints file will >> be deleted after replay is complete, but not sure of the root cause of why >> the hints folder is getting deleted. >> When I look at the nodetool status or nodetool ring - it indicates that >> all nodes are up and running in normal state, no node went down. Also, I do >> not see anything the debug logs indicating that a node went down. In such a >> scenario, I am not sure why would HintsWriterExecutor would get triggered. >> >> > That error code (O_RDONLY) in the log message indicates that the hints > folder has had its permission bits set to read only. > > We've had several issues with some of the tools doing this type of thing > when they are run as the root user. Is this specific node one on which you > use any of the tools like sstableloader or similar? If so, are you running > them as root? > > Another thought - if it is on a different partition than the data > directory, is there free space left on the underlying device holding: > /var/lib/cassandra/hints? > > > -- > ----------------- > Nate McCall > Wellington, NZ > @zznate > > CTO > Apache Cassandra Consulting > http://www.thelastpickle.com >
