Public bug reported:
Glance servers refers to bin/teller-server.py and bin/parallax-server.py
which crash with a Bad File Descriptor exception when started on a
system that uses poll or epoll (Ubuntu Maverick) but succeeds on a
system that doesn't (Mac OS X).
After stracing, I've narrowed down the issue to an interaction between
eventlet and python's daemon module.
In both glance servers we import glance.common.(wsgi|controller) at the
top of the module. Both of these modules in turn import eventlet. The
key thing to note is that eventlet is imported BEFORE daemonization.
On systems that support epoll, merely importing eventlet causes it to
fire off an epoll_create syscall and create an epoll_file descriptor
which is then used to perform subsequent operations relating to epoll.
The problem is that when we subsequently daemonize, for safety, the
daemonize module closes all file descriptors not explicitly whitelisted.
Since we don't explicitly whitelist the epoll file descriptor (we're not
suppose to know about it, eventlet is supposed to hide that
implementation detail!), it is closed, which causes subsequent calls to
add fd's to the poll list to fail with a bad file descriptor error.
Proposed fix (short-term):
A quick fix is to move any imports that in turn import eventlet (wsgi
and controller above) into code that is executed *after* daemonization.
Essentially move global import into the body of main. This is how nova
does it which is why nova-api still works.
Proposed fix (long-term):
The long-term fix maybe to fix eventlet itself by moving the
epoll_create call to somewhere that isn't called during an import. This
will take considerable research to get right. Relevant files are:
green/socket.py (get_hub call), hub/epolls.py and hub.poll.py.
** Affects: glance
Importance: High
Status: New
--
You received this bug notification because you are a member of Registry
Administrators, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/687661
Title:
Glance servers crash when using epoll
_______________________________________________
Mailing list: https://launchpad.net/~registry
Post to : registry@lists.launchpad.net
Unsubscribe : https://launchpad.net/~registry
More help : https://help.launchpad.net/ListHelp
