[
https://issues.apache.org/jira/browse/CLOUDSTACK-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462811#comment-13462811
]
Devdeep Singh commented on CLOUDSTACK-105:
------------------------------------------
One way I could trigger creation of stale scoket files in a Xenserver host was
to have host part of a zone that uses basic networking and CSP packages
installed. Open vswitch is disabled. If the host is part of a pool and a new
master is designated (probably triggered by cloudstack), stale socket files are
left behind.
Caleb, can you look up the management server logs and see if CloudStack is
sending commands to designate a new master? If this is the case, the issue
would be with Xenserver leaving behind stale files. We can follow up
accordingly.
> /tmp/stream-unix.####.###### stale sockets causing inodes to run out on
> Xenserver
> ---------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-105
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-105
> Project: CloudStack
> Issue Type: Bug
> Components: XenServer
> Affects Versions: pre-4.0.0
> Environment: Xenserver 6.0.2
> Cloudstack 3.0.2
> Reporter: Caleb Call
> Assignee: Devdeep Singh
> Fix For: pre-4.0.0
>
>
> We came across an interesting issue in one of our clusters. We ran out of
> inodes on all of our cluster members (since when does this happen in 2012?).
> When this happened, it in turn made the / filesystem a read-only filesystem
> which in turn made all the hosts go in to emergency maintenance mode and as a
> result get marked down by Cloudstack. We found that it was caused by
> hundreds of thousands of stale socket files in /tmp named
> "stream-unix.####.######". To resolve the issue, we had to delete those
> stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm -v {} \;),
> then kill and restart xapi, then correct the emergency maintenance mode.
> These hosts had only been up for 45 days before this issue occurred.
> In our scouring of the interwebs, the only other instance we've been able to
> find of this (or similar) happening is in the same setup we are currently
> running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix sockets have
> anything to do with Cloudstack? I would think if this was a Xenserver issue
> (bug), there would be a lot more on the internet about this happening. For a
> temporary workaround, we've added a cronjob to cleanup these files but we'd
> really like to address the actual issue that's causing these sockets to
> become stale and not get cleaned-up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
