yep, so far it looks like a file descriptor leak. Not sure if gc or some other event like compaction would close these files..
[root@CAP-VM-1 ~]# ls -al /proc/31134/fd | grep MSA | wc -l 540 [root@CAP-VM-1 ~]# ls -al /proc/31134/fd | grep MSA | wc -l 542 [root@CAP-VM-1 ~]# ls -al /proc/31134/fd | grep MSA | wc -l 554 [root@CAP-VM-1 ~]# ls -al /proc/31134/fd | grep MSA | wc -l 558 On Wed, Dec 14, 2011 at 8:28 PM, Bryce Godfrey <bryce.godf...@azaleos.com> wrote: > I'm seeing this also, and my nodes have started crashing with "too many open > file errors". Running lsof I see lots of these open tmp files. > > java 8185 root 911u REG 8,32 38 > 129108266 > /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268721-CompressionInfo.db > java 8185 root 912u REG 8,32 0 > 155320741 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1092-Data.db > java 8185 root 913u REG 8,32 0 > 155320742 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1097-Index.db > java 8185 root 914u REG 8,32 0 > 155320743 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1097-Data.db > java 8185 root 916u REG 8,32 0 > 155320754 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1113-Data.db > java 8185 root 918u REG 8,32 0 > 155320744 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1102-Index.db > java 8185 root 919u REG 8,32 0 > 155320745 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1102-Data.db > java 8185 root 920u REG 8,32 0 > 155320755 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1118-Index.db > java 8185 root 921u REG 8,32 0 > 129108272 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268781-Data.db > java 8185 root 922u REG 8,32 38 > 129108273 > /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268781-CompressionInfo.db > java 8185 root 923u REG 8,32 0 > 155320756 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1118-Data.db > java 8185 root 929u REG 8,32 38 > 129108262 > /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268822-CompressionInfo.db > java 8185 root 947u REG 8,32 0 > 129108284 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268854-Data.db > java 8185 root 948u REG 8,32 38 > 129108285 > /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268854-CompressionInfo.db > java 8185 root 954u REG 8,32 0 > 155320746 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1107-Index.db > java 8185 root 955u REG 8,32 0 > 155320747 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1107-Data.db > > Going to try rolling back to 1.0.5 for the time being even though I was > hoping to use one of the fixes in 1.0.6 > > -----Original Message----- > From: Ramesh Natarajan [mailto:rames...@gmail.com] > Sent: Wednesday, December 14, 2011 6:03 PM > To: user@cassandra.apache.org > Subject: tmp files in /var/lib/cassandra/data > > We are using leveled compaction running cassandra 1.0.6. I checked the data > directory (/var/lib/cassandra/data) and i see these 0 bytes tmp files. > What are these files? > > thanks > Ramesh > > -rw-r--r-- 1 root root 0 Dec 14 17:15 uid-tmp-hc-106-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:15 uid-tmp-hc-106-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:23 uid-tmp-hc-117-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:23 uid-tmp-hc-117-Index.db > -rw-r--r-- 1 root root 0 Dec 14 15:51 uid-tmp-hc-11-Data.db > -rw-r--r-- 1 root root 0 Dec 14 15:51 uid-tmp-hc-11-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:31 uid-tmp-hc-129-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:31 uid-tmp-hc-129-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:40 uid-tmp-hc-142-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:40 uid-tmp-hc-142-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:40 uid-tmp-hc-145-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:40 uid-tmp-hc-145-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:47 uid-tmp-hc-158-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:47 uid-tmp-hc-158-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:47 uid-tmp-hc-162-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:47 uid-tmp-hc-162-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:55 uid-tmp-hc-175-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:55 uid-tmp-hc-175-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:55 uid-tmp-hc-179-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:55 uid-tmp-hc-179-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:03 uid-tmp-hc-193-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:03 uid-tmp-hc-193-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:03 uid-tmp-hc-197-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:03 uid-tmp-hc-197-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:02 uid-tmp-hc-19-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:02 uid-tmp-hc-19-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:03 uid-tmp-hc-200-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:03 uid-tmp-hc-200-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:11 uid-tmp-hc-213-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:11 uid-tmp-hc-213-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:11 uid-tmp-hc-217-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:11 uid-tmp-hc-217-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:19 uid-tmp-hc-230-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:19 uid-tmp-hc-230-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:19 uid-tmp-hc-235-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:19 uid-tmp-hc-235-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:27 uid-tmp-hc-249-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:27 uid-tmp-hc-249-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:27 uid-tmp-hc-253-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:27 uid-tmp-hc-253-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:28 uid-tmp-hc-257-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:28 uid-tmp-hc-257-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:35 uid-tmp-hc-270-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:35 uid-tmp-hc-270-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:36 uid-tmp-hc-275-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:36 uid-tmp-hc-275-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:44 uid-tmp-hc-288-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:44 uid-tmp-hc-288-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:10 uid-tmp-hc-28-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:10 uid-tmp-hc-28-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:44 uid-tmp-hc-293-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:44 uid-tmp-hc-293-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:52 uid-tmp-hc-307-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:52 uid-tmp-hc-307-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:52 uid-tmp-hc-310-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:52 uid-tmp-hc-310-Index.db > -rw-r--r-- 1 root root 0 Dec 14 18:52 uid-tmp-hc-315-Data.db > -rw-r--r-- 1 root root 0 Dec 14 18:52 uid-tmp-hc-315-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:00 uid-tmp-hc-328-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:00 uid-tmp-hc-328-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:00 uid-tmp-hc-333-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:00 uid-tmp-hc-333-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:08 uid-tmp-hc-347-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:08 uid-tmp-hc-347-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:08 uid-tmp-hc-353-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:08 uid-tmp-hc-353-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:09 uid-tmp-hc-357-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:09 uid-tmp-hc-357-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:17 uid-tmp-hc-370-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:17 uid-tmp-hc-370-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:17 uid-tmp-hc-375-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:17 uid-tmp-hc-375-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:25 uid-tmp-hc-389-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:25 uid-tmp-hc-389-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:18 uid-tmp-hc-38-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:18 uid-tmp-hc-38-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:25 uid-tmp-hc-393-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:25 uid-tmp-hc-393-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:25 uid-tmp-hc-399-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:25 uid-tmp-hc-399-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:33 uid-tmp-hc-412-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:33 uid-tmp-hc-412-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:34 uid-tmp-hc-418-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:34 uid-tmp-hc-418-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:42 uid-tmp-hc-432-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:42 uid-tmp-hc-432-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:42 uid-tmp-hc-438-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:42 uid-tmp-hc-438-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:42 uid-tmp-hc-442-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:42 uid-tmp-hc-442-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:50 uid-tmp-hc-455-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:50 uid-tmp-hc-455-Index.db > -rw-r--r-- 1 root root 0 Dec 14 19:50 uid-tmp-hc-460-Data.db > -rw-r--r-- 1 root root 0 Dec 14 19:50 uid-tmp-hc-460-Index.db > -rw-r--r-- 1 root root 10486534 Dec 14 19:58 uid-tmp-hc-470-Data.db > -rw-r--r-- 1 root root 35176 Dec 14 19:58 uid-tmp-hc-470-Filter.db > -rw-r--r-- 1 root root 255860 Dec 14 19:58 uid-tmp-hc-470-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:26 uid-tmp-hc-49-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:26 uid-tmp-hc-49-Index.db > -rw-r--r-- 1 root root 0 Dec 14 15:38 uid-tmp-hc-5-Data.db > -rw-r--r-- 1 root root 0 Dec 14 15:38 uid-tmp-hc-5-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:35 uid-tmp-hc-61-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:35 uid-tmp-hc-61-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:43 uid-tmp-hc-74-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:43 uid-tmp-hc-74-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:50 uid-tmp-hc-80-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:50 uid-tmp-hc-80-Index.db > -rw-r--r-- 1 root root 0 Dec 14 16:59 uid-tmp-hc-87-Data.db > -rw-r--r-- 1 root root 0 Dec 14 16:59 uid-tmp-hc-87-Index.db > -rw-r--r-- 1 root root 0 Dec 14 17:07 uid-tmp-hc-96-Data.db > -rw-r--r-- 1 root root 0 Dec 14 17:07 uid-tmp-hc-96-Index.db
