Hey Chris, This e-mail chain might be of interest to you.
Talk to you later,
Mike
---------- Forwarded message ----------
From: John Burwell <jburw...@basho.com>
Date: Tue, Aug 20, 2013 at 3:43 PM
Subject: [DISCUSS/PROPOSAL] Upgrading Driver Model
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Cc: Daan Hoogland <daan.hoogl...@gmail.com>, Hugo Trippaers <
htrippa...@schubergphilis.com>, "La Motta, David" <david.lamo...@netapp.com>
All,
In capturing my thoughts on storage, my thinking backed into the driver
model. While we have the beginnings of such a model today, I see the
following deficiencies:
1. *Multiple Models*: The Storage, Hypervisor, and Security layers each
have a slightly different model for allowing system functionality to be
extended/substituted. These differences increase the barrier of entry for
vendors seeking to extend CloudStack and accrete code paths to be
maintained and verified.
2. *Leaky Abstraction*: Plugins are registered through a Spring
configuration file. In addition to being operator unfriendly (most
sysadmins are not Spring experts nor do they want to be), we expose the
core bootstrapping mechanism to operators. Therefore, a misconfiguration
could negatively impact the injection/configuration of internal management
server components. Essentially handing them a loaded shotgun pointed at
our right foot.
3. *Nondeterministic Load/Unload Model*: Because the core loading
mechanism is Spring, the management has little control over the timing and
order of component loading/unloading. Changes to the Management Server's
component dependency graph could break a driver by causing it to be started
at an unexpected time.
4. *Lack of Execution Isolation*: As a Spring component, plugins are
loaded into the same execution context as core management server
components. Therefore, an errant plugin can corrupt the entire management
server.
For next revision of the plugin/driver mechanism, I would like see us
migrate towards a standard pluggable driver model that supports all of the
management server's extension points (e.g. network devices, storage
devices, hypervisors, etc) with the following capabilities:
- *Consolidated Lifecycle and Startup Procedure*: Drivers share a
common state machine and categorization (e.g. network, storage, hypervisor,
etc) that permits the deterministic calculation of initialization and
destruction order (i.e. network layer drivers -> storage layer drivers ->
hypervisor drivers). Plugin inter-dependencies would be supported between
plugins sharing the same category.
- *In-process Installation and Upgrade*: Adding or upgrading a driver
does not require the management server to be restarted. This capability
implies a system that supports the simultaneous execution of multiple
driver versions and the ability to suspend continued execution work on a
resource while the underlying driver instance is replaced.
- *Execution Isolation*: The deployment packaging and execution
environment supports different (and potentially conflicting) versions of
dependencies to be simultaneously used. Additionally, plugins would be
sufficiently sandboxed to protect the management server against driver
instability.
- *Extension Data Model*: Drivers provide a property bag with a metadata
descriptor to validate and render vendor specific data. The contents of
this property bag will provided to every driver operation invocation at
runtime. The metadata descriptor would be a lightweight description that
provides a label resource key, a description resource key, data type
(string, date, number, boolean), required flag, and optional length limit.
- *Introspection: Administrative APIs/UIs allow operators to understand
the configuration of the drivers in the system, their configuration, and
their current state.*
- *Discoverability*: Optionally, drivers can be discovered via a project
repository definition (similar to Yum) allowing drivers to be remotely
acquired and operators to be notified regarding update availability. The
project would also provide, free of charge, certificates to sign plugins.
This mechanism would support local mirroring to support air gapped
management networks.
Fundamentally, I do not want to turn CloudStack into an erector set with
more screws than nuts which is a risk with highly pluggable architectures.
As such, I think we would need to tightly bound the scope of drivers and
their behaviors to prevent the loss system usability and stability. My
thinking is that drivers would be packaged into a custom JAR, CAR
(CloudStack ARchive), that would be structured as followed:
- META-INF
- MANIFEST.MF
- driver.yaml (driver metadata(e.g. version, name, description, etc)
serialized in YAML format)
- LICENSE (a text file containing the driver's license)
- lib (driver dependencies)
- classes (driver implementation)
- resources (driver message files and potentially JS resources)
The management server would acquire drivers through a simple scan of a URL
(e.g. file directory, S3 bucket, etc). For every CAR object found, the
management server would create an execution environment (likely a dedicated
ExecutorService and Classloader), and transition the state of the driver to
Running (the exact state model would need to be worked out). To be really
nice, we could develop a custom Ant task/Maven plugin/Gradle plugin to
create CARs. I can also imagine an opportunities to add hooks to this
model to register instrumentation information with JMX and authorization.
To keep the scope of this email confined, we would introduce the general
notion of a Resource, and (hand wave hand wave) eventually compartmentalize
the execution of work around a resource [1]. This (hand waved)
compartmentalization would allow us the controls necessary to safely and
reliably perform in-place driver upgrades. For an initial release, I would
recommend implementing the abstractions, loading mechanism, extension data
model, and discovery features. With these capabilities in place, we could
attack the in-place upgrade model.
If we were to adopt such a pluggable capability, we would have the
opportunity to decouple the vendor and CloudStack release schedules. For
example, if a vendor were introducing a new product that required a new or
updated driver, they would no longer need to wait for a CloudStack release
to support it. They would also gain the ability to fix high priority
defects in the same manner.
I have hand waved a number of issues that would need to be resolved before
such an approach could be implemented. However, I think we need to decide,
as a community, that it worth devoting energy and effort to enhancing the
plugin/driver model and the goals of that effort before driving head first
into the deep rabbit hole of design/implementation.
Thoughts? (/me ducks)
-John
[1]: My opinions on the matter from CloudStack Collab 2013 ->
http://www.slideshare.net/JohnBurwell1/how-to-run-from-a-zombie-cloud-stack-distributed-process-management
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*
signature.asc
Description: PGP signature
