Prateek created CASSANDRA-6392:
----------------------------------
Summary: Cassandra 2.0.1 leaking open file handles for deleted
system hints
Key: CASSANDRA-6392
URL: https://issues.apache.org/jira/browse/CASSANDRA-6392
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 2.0.1
Cassandra datastax java driver 1.0.3 (cassandra-driver-core)
Reporter: Prateek
The system was running for a few days with a heavy write workload. The system
crashed with too many open file handles. It seems the deleted system hint file
handles are still open even though the file has been deleted on disk. We use a
3-node cluster with replication factor 3 on ec2 us-east-1 using
NetworkTopologyStrategy. Default consistency level for write is ANY.
This is blocking our production deployment and causing instability in
production.
Appreciate your quick response.
Many thanks,
Prateek
See logs below:
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |wc
-l
201499
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568
|grep deleted|wc -l
186261
java 13568 ubuntu *676r REG 9,0 356 1610775839
/mnt/cassandra/data/system/hints/system-hints-jb-848-Data.db (deleted)
java 13568 ubuntu *677r REG 9,0 309 1610770742
/mnt/cassandra/data/system/hints/system-hints-jb-845-Data.db (deleted)
java 13568 ubuntu *678r REG 9,0 506 1610775703
/mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db (deleted)
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# ls -l
/mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
ls: cannot access /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db:
No such file or directory
--
This message was sent by Atlassian JIRA
(v6.1#6144)
