On 05/11/2015 09:02 PM, Dieterly, Deklan wrote:
Given Python’s inherent inability to scale (GIL) relative to other
languages/platforms, have there been any serious discussions on
allowing other more scalable languages into the OpenStack ecosystem
when concurrency/scalability is paramount?
Robert has already responded to your lack of specificity in the above
statement. I'd like to add a couple things:
a) The architecture of our OpenStack projects -- including the systems
we use as backing data stores and the coarse-grained locking techniques
we have used to date -- is a bigger problem than the language most
OpenStack components are written in.
b) The speed of development and the familiarity with the Python language
of the folks involved in our CI, testing, and infra/build platforms and
the inherent economies of scale we get from that represent a far greater
long-term cost reduction than trying to rewrite existing systems in
"faster or more scalable" platforms. Developer and operator time costs
way more than the tiny amount of additional costs that comes from buying
a few more and faster processors to put controller services on.
c) Many OpenStack services, including Nova, Cinder, and Neutron, when
looked at from a thousand-foot level, are little more than glue code
that pipes out to a shell to execute system commands (sorry, but it's
true). If you look at the time spent in the database and message queue
it's really very little compared to the time spent on a compute node
spawning an image. The DB and message queue are, IME, not where scaling
problems occur. Instead, they occur in things like Nova pulling images
from Glance unnecessarily (an architectural problem, not a concurrency
problem) or the implementation of iptables saves when lots of security
groups on a single compute node would cause excessive rebuilds of the
routing tables.
Now, do I support the hummingbird Golang object server effort in Swift?
Absolutely, I do.
Because it 100% makes sense there. That part of the Swift code base is
where concurrency and performance matters big time. Would implementing
all of nova-compute in Golang result in huge performance gains? No, not
at all. It just doesn't make much sense, since much of nova-compute's
time is spent shell'd out in execution or waiting on locks (an
implementation thing that has little to nothing to do with the language
used).
So, bottom line for me: focus on the things that will have the biggest
impact to long-term cost reduction of our codebase.
Very little of that cost (in most of our OpenStack projects) has to do
with concurrency issues. A *lot* of that long-term cost has to do with
the unnecessary complexity of our architecture and subsystems, because
it's high-priced humans that need to get paid to maintain such complexity.
So, to me, the highest priority performance and scale fixes actually
have to do with the simplification of our subsystems and architecture,
not with whether we use mysql-python, PyMySQL, Python vs. Scala vs.
Rust, or any other distractions.
Best,
-jay
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
