This looks like a prime candidate for a bug. There might be time to get it
in before 4.0 goes out!
On 9/14/12 9:54 AM, "Caleb Call" <calebc...@me.com> wrote:
>We came across an interesting issue yesterday in one of our clusters. We
>ran out of inodes on all of our cluster members (since when does this
>happen in 2012?). When this happened, it in turn made the / filesystem a
>read-only filesystem which in turn made all the hosts go in to emergency
>maintenance mode and as a result get marked down by Cloudstack. We found
>that it was caused by hundreds of thousands of stale socket files in /tmp
>named "stream-unix.####.######". To resolve the issue, we had to delete
>those stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm
>-v {} \;), then kill and restart xapi, then correct the emergency
>maintenance mode. These hosts had only been up for 45 days before this
>issue occurred.
>
>In our scouring of the interwebs, the only other instance we've been able
>to find of this (or similar) happening is in the same setup we are
>currently running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix
>sockets have anything to do with Cloudstack? I would think if this was a
>Xenserver issue (bug), there would be a lot more on the internet about
>this happening. For a temporary workaround, we've added a cronjob to
>cleanup these files but we'd really like to address the actual issue
>that's causing these sockets to become stale and not get cleaned-up.
>
>Thoughts?
>
>Thanks,
>Caleb
>
--
Æ
