Do all of these combinations need to be fully tested for each release?
What are the combinations that have been tested for the current release?
How many of these combinations are known to be running in production?
How many of these production organizations have test environments that
could be used? And operations staff that could run the tests.
They will test anyway so it is mostly a change to timing and the actual
scripts.
They may be able to augment the existing scripts with the test scripts
that they are using or work on the completion of the scripts already
planned.
I am unsure what Paul means by "we need hardware to run tests on".
Clearly hardware is required for testing but it would not seem to matter
where the hardware exists or who owns it as long as it is available.
Is there a list of tests that are missing?
Is the test suite documented so that end-users can actually use the
tests on their own test systems?
This is a bit of a switch in thinking about testing and about the role
of the users in the release management process but it has some benefits.
The testing function of the release team switches to a project
management role that involves tracking and coaching the testing ecosystem.
Ron
On 30/06/2017 4:57 PM, Paul Angus wrote:
Taken from a talk on CloudStack testing [1]...
There are Many, many, MANY permutations of a CloudStack deployment….
• Basic / Advanced
• Local / shared / mixed storage
• More than 8 common hypervisor types/versions
• 4 or 5 Management server OS possibilities
• That’s 144 combinations only looking the basics.
[1]
https://www.slideshare.net/ShapeBlue/cloudstack-eu-user-group-trillian?qid=74ff2be0-664c-4bca-a3dc-f30d880ca088&v=&b=&from_search=1
Trillian [2], can create any of those, and multiple at the same time, but the
amount of hardware required to do that means that we have to pick our battles.
Running the blueorangutan bot command '@blueorangutan matrix' in a PR will run
the smoke test suite against a PR using 3 environments, one each of KVM,
XenServer and vSphere and takes around 8 hours.
But that is only looking for major regressions. A full component test run
takes around 5 days to run and is riddled with bugs in the tests.
Ultimately these are still of limited scope, few people are as diligent as say
Mike T in creating practical marvin tests for their code / features.
[2] https://github.com/shapeblue/Trillian
Therefore we need hardware to run tests on, but more importantly we need the
tests to exist and work in the first place. Then we can really do something.
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Alex Hitchins [mailto:a...@alexhitchins.com]
Sent: 30 June 2017 21:34
To: dev@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: RE: [DISCUSS] - Releases, Project Management & Funding Thereof
Consultation with users is something that should definite be done. Canvas as
many as possible.
I agree that most people will be running test environments before full rollout
of any technology, I guess see it a little from a CTO eyes - why shortlist a
technology that doesn't even endorse its own releases?
Hopefully we will get some more replies to this thread from other CloudStack
enthusiasts to help shape this conversation.
I'm setting up a new development environment now to get my hands mildly soiled.
Going the Windows route again. Fancy a challenge for the weekend.
Alexander Hitchins
------------------------
E: a...@alexhitchins.com
W: alexhitchins.com
M: 07788 423 969
T: 01892 523 587
-----Original Message-----
From: Ron Wheeler [mailto:rwhee...@artifact-software.com]
Sent: 30 June 2017 21:08
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] - Releases, Project Management & Funding Thereof
On 30/06/2017 3:28 PM, Alex Hitchins wrote:
We can't validate all scenarios no, hence suggesting several common setups as a
reasonable baseline. I like the idea of users posting their hardware and
versions as a community endeavour.
I strongly feel there should be an established, physical setup that the release
team have access to in order to validate a release.
This is perhaps something that should be requested from the user community.
I would expect that anyone running Cloudstack in production on a major site
would have a test setup and might be very happy to have the dev team test on
their setup.
This would save them a lot of resources at the expense of a bit of time on
their test environment.
If this was some random cat meme generator on GitHub, I'd accept the risk in
running an untested version. For something I'd be running my business on
however I'd expect there being some weight behind the release.
Perhaps I have unrealistic expectations!
Not at all.
Your expectations might be the key to making a pitch to the user community for
some help from people and organizations that are not interested in writing code
but have a major interest in testing.
In addition to access to test equipment, this might actually get new people on
the team with the right skills required to extend the test scripts and test
procedure documentation.
Does anyone have a list of the configuration specifications that are required
to test a new release?
Would it help to approach major users of Cloudstack with a direct request for
use of their test equipment and QA staff in return for early access to new
releases and testing on their hardware?
Ron
Alexander Hitchins
------------------------
E: a...@alexhitchins.com
W: alexhitchins.com
M: 07788 423 969
T: 01892 523 587
-----Original Message-----
From: Ron Wheeler [mailto:rwhee...@artifact-software.com]
Sent: 30 June 2017 20:13
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] - Releases, Project Management & Funding
Thereof
On 30/06/2017 2:19 PM, Alex Hitchins wrote:
Releasing against a defined reference rig would be a very good idea, especially
if we could replicate several.
It concerns me slightly that we are building a platform we want to promote
people to deploy in enterprise environments with the caveat 'run at your own
risk'.
There is no choice as near as I can tell.
It seems that there are too many combinations of hardware, network
configurations and OSs to guarantee that a release will work on all of them and
still get a release delivered.
As Will pointed out, the Release Team does not have access to every combination
where previous releases are in production use, to test the new release
candidate.
Currently it may be not very explicit about what are the fully tested
configurations and from what Will said, I gather that there is no policy saying
what the minimum test set is to declare a release ready to go.
There is no reason preventing a release being tested after release by an end-user or a
developer and adding to the release documentation something to the effect that
"Users have reported that this release has been put into production on XYZ
configuration with no modifications."
This at least gets the release out the door for the 95% of the users that do
not have an XYZ rather than waiting for someone with an XYZ to find time to
test it.
It may also encourage companies using or selling XYZs to put up some resources
(hardware and people) dedicated to testing so that they get into the initial
release.
Ron
We need to up our game.
'We' he says, after two years MIA!
On 30 Jun 2017, at 18:41, Ron Wheeler <rwhee...@artifact-software.com> wrote:
How much time is there between a feature freeze and the RC being cut.?
Do people know far enough in advance that their feature is in or out and if in
must be ready to go to a RC release by such and such a date?
Is the use case testing well defined - hardware, configurations, etc.
Can you put out a release that says: "This release has been tested on these
configurations (A, B ,C) but the following configurations/use cases are not yet fully
tested and other configuration may be used at your own risk after your own internal tests
have been run successfully."
Is there any concept that "Cloudstack is verified to run on the following
configurations and should also run on these configurations but has not been tested fully.
It may run on these configurations but is not tested during the release cycle."
Ron
On 30/06/2017 1:14 PM, Will Stevens wrote:
Have not looked at Release Tsar, but worth checking out.
In general, the biggest problem we have with releasing on a
schedule is the lack of a CI setup which covers the entire
software. Or at least a 'supported' set of features. This means
that the release is always bound to a bunch of volunteers getting
around to testing their use case. Solidfire and Nuage are pretty good about
getting some CI run on some pieces.
Trillian is great for covering a portion of the tests, but it
currently does not cover the whole software use case. We also need
more trillian deployments in the wild to support the CI initiative.
We do need to be stricter about nothing going in after an RC is cut
but blockers. The limited CI coverage and the dependence on a few
people for testing exasperates this problem.
So there is multiple layers to this. I think someone dedicates to
the RM role would help this a lot because they would have a single
community focus mandate, so it is in their best interest to
implement a flow which does not inhibit their ability to deliver on their
mandate.
On Jun 30, 2017 12:53 PM, "Ron Wheeler"
<rwhee...@artifact-software.com>
wrote:
Perhaps a Release Tsar would be a better solution.
The RM needs to have absolute control over what is in or out.
Reasonable discussion allowed and then a decision once the RM
feels that the case has been fully explored and that a positive vote is
expected.
The importance on meeting deadlines needs to have a higher
priority. If a feature/fix can not meet the quality/testing
threshold on time then it gets dropped from the RC and scheduled for the next
release.
A few cycles of a bit of ruthlessness should get everyone`s
intention and shorten the release cycle.
Meeting release schedules would also reduce the pain of a feature
being deferred.
According to the schedule proposed last
year,(https://cwiki.apache.org
/confluence/display/CLOUDSTACK/%5BPROPOSAL%5D+2016-2017+
Release+Cycle+and+Calendar)
Cloudstack 4.9.10 (LTS) , 5.04.0 (LTS) as well as 5.1.3.0
(maintenance)
5.2.1.0 (Maintenance) were released June 2017.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Pro
c edure seems to be pretty reasonable. The RM probably needs to
moderate the vote and explain what -1 votes mean to product
credibility if they delay the release. Negative votes because
someone`s new feature did not make it should be ignored.
Ron
On 30/06/2017 12:09 PM, Paul Angus wrote:
We could probably split this topic down also....
I think I may have mentioned previously 😊 my view on how we have
somewhat shot ourselves in the foot with the release process this
time around. I think that for the most part, people have been
well intentioned, and have been trying to 'make this release as
good as possible' which is counter-productive, as it's been introducing new
blockers.
I'm not sure we have a problem in our 'loosely-agreed' process,
it's just that repeatedly people have ignored it.
WRT a full-time release manager, I suspect that they would find
that "you can lead a horse to water, but you can't make it drink".
They would not be able to compel anyone to 'hurry up and fix that
bug you created', although I guess maybe they could pull a feature if the
author(s) didn't sort it out.
Because ultimately a release manager, paid or otherwise should
only be doing what the 'community' decides the release manager's
role is. So we need to be clear about how we want releases to
work before worrying about who manages that.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Alex Hitchins [mailto:a...@alexhitchins.com]
Sent: 30 June 2017 15:05
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] - Releases, Project Management & Funding
Thereof
I am in complete agreement with you. Also on your other reply
regards to a FT release manager.
If 'we' don't go down this line, more and more people will follow
the Cosmic/Schuberg Philis path or even use Cosmic instead.
I'm encouraged by your response. Sounds like a few others hold
the same concerns.
Alexander Hitchins
------------------------
E: a...@alexhitchins.com
W: alexhitchins.com
M: 07788 423 969
T: 01892 523 587
-----Original Message-----
From: Will Stevens [mailto:williamstev...@gmail.com]
Sent: 30 June 2017 14:54
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] - Releases, Project Management & Funding
Thereof
Yes, Schuberg Philis, a very active community member forked
Cosmic off of CloudStack and has been developing their fork for their needs.
I do think we need to have a more consistent front on this
matter. I think it would make a big difference on the quality,
release cadence and perception of the project.
On Jun 30, 2017 9:48 AM, "Alex Hitchins" <a...@alexhitchins.com> wrote:
Thanks Will,
I understand it's something that comes with a big bag of
troublesome worries.
If this topic comes up again in any discussions, I'd be
interested to hear their thoughts on what I see as the
alternative; without a dedicated RM/PM/Captain, people will fork
off CS so they can achieve the same thing, and CS ultimately
looses out long term. I can't remember the name of the fork, but
I think I'm right that a previous large CS contributor/user
forked off as they wanted greater management in the areas we are discussing
here.
Alexander Hitchins
------------------------
E: a...@alexhitchins.com
W: alexhitchins.com
M: 07788 423 969
T: 01892 523 587
-----Original Message-----
From: Will Stevens [mailto:williamstev...@gmail.com]
Sent: 30 June 2017 14:31
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] - Releases, Project Management & Funding
Thereof
Apache has been historically against the idea of a cloudstack
foundation and there is a bit of a pandoras box there which we
will want to be careful about opening.
Apache added direct contribution, but it was unusable for us
historically because it required a minimum contribution of 50k,
which none of us can afford. However, there have been some
changes to the board recently which are in our favour if we want to put
pressure to lower that to say 5-10k.
Even if we do solve for smaller direct contributions, we will
have to jump through hoops to be able to use those funds for a
dedicated release manager. I do think this is a possibility if we
manage our needs and communications very well. I had some
preliminary discussions with some apache foundation folks to
express these specific concerns. I played off the fact that i
know they dont want to entertain a cloudstack foundation and
tried to see if i could get them to move on the direct
contribution mechanism to make it usable for us, specifically
with the goal of hiring a full time release manager. I definitely
had their ear and they acknowledged the problems we are facing
(and currently discussing). They expressed concerns about being
able to hire someone with the direct contributions, but
brainstormed a bit to potentially hire an agency who actually does the hire and
they pay the persons salary through the agency with the direct contribution
funds.
All to say, there are potential options here, but there be
dragons, so we have to handle this topic with care.
On Jun 30, 2017 9:12 AM, "Ron Wheeler"
<rwhee...@artifact-software.com>
wrote:
https://www.apache.org/foundation/contributing.html says:
"If you have a specific target or project that you wish to
directly support, pleasecontact us
<https://www.apache.org/founda
tion/contributing.html#Fundraising>and we will do our best to satisfy your
wishes."
1) Is Apache willing to allow projects to set up their own
foundations? I doubt but someone would need to check this out.
Does the PMC have the project charter or the agreement that was
signed when Cloudstack moved.
2) Has anyone tried to contact Apache about directing support to
Cloudstack.
I am not convinced that lack of paid staff is the issue.
This discussion reminded me of this.
Q: How many psychiatrists does it take to change a lightbulb ?
A: Only one, but the lightbulb must want to change
http://www.lightbulbjokes.com/directory/p.html
Ron
On 30/06/2017 6:48 AM, Alex Hitchins wrote:
As per Giles's comment to the previous thread, I thought I would
start a discussion on the subject to canvas peoples thoughts,
opinions
and fears.
My question for discussion, is there is any mileage in someone
creating a "CloudStack Foundation" as a non-profit entity,
funded largely by key CloudStack players with the sole function
of employing dedicated resource (part or full time) to handle
all releases and other essential 'back office' functions. The
idea being it's in everyone's interest to chip in a little each
to fund core project and
release management.
The idea might be utterly irrelevant, pointless and/or straight up daft.
I urge you all to let me know.
Something for you all to think over this weekend.
Alexander Hitchins
------------------------
E: a...@alexhitchins.com
W: alexhitchins.com
M: 07788 423 969
T: 01892 523 587
-----Original Message-----
From: Giles Sirett [mailto:giles.sir...@shapeblue.com]
Sent: 30 June 2017 09:51
To: dev@cloudstack.apache.org
Subject: RE: JIRA - PLEASE READ
All
This thread seems to have turned into 2 quite different discussions:
1. The use (or not) of Jira - which was the original discussion
2. Ways/means of encouraging (and paying for more structured
contributors)
I know that it could be argued that these are related. Could I
suggest opening up a thread on "release and project management
and funding it" and keeping this thread to the original
discussion
(I will weigh in on both of these at some stage)
Kind regards
Giles
giles.sir...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Alex Hitchins [mailto:a...@alexhitchins.com]
Sent: 29 June 2017 18:49
To: dev@cloudstack.apache.org
Subject: Re: JIRA - PLEASE READ
If it isn't being treated as a product it will be very
impossible to market it as enterprise ready.
I know we all know this.
Similar sized projects under the Apache banner must have the
same issue, what is the best way to gather experience of these projects?
See how they handle these growing pains.
A cloudstack foundation entity funded by companies earning from
cloudstack seems a good way forward.
Another tuppence, this is getting expensive.
On 29 Jun 2017, at 18:18, Ron Wheeler
<rwhee...@artifact-software.com>
wrote:
I understand that it is a volunteer organization.
I do not know how many (if any) of the committers and PMC
members are funded by their organizations (allowed or ordered
to work on Cloudstack during company time) which is often the
way that Apache projects get staffed.
Clearly it is hard to tell someone who is being funded by a
company to fix a problem or who is working on their own time,
to do or not do something.
On the other hand, the PMC has to build a community culture
that is good for the project.
That means describing a vision, planning and enforcing a
roadmap, and maintaining a focused project "marketing" effort.
There is a lot of extremely talented individuals working on
Cloudstack and it appears to have a very strong and valuable code-base.
To me the key question is about the PMC and the core committers'
ability to make Cloudstack a "product" that can compete for
market share and acceptance.
Is Cloudstack at a point in its development where it should be
treated like a product?
- sufficient functionality to compete
- sufficient user base to be a competitor in the market
- production reliability and stability
- business model for supporting companies to justify their
continued support
This may not require more effort but requires different
policies and different activities.
There has to be someone or a PMC that can say "No".
- This change can not be included in this release because it
will delay the release.
- This change adds an unacceptable level of complexity
- This bug fix will have to wait for the next release because
it is too late to test it and fix the docs.
- This fix breaks the docs
- The release can not be made until this doc is updated.
Does the core group want to make it a competitive product or
is it sufficient for the interested players to continue in its current form?
Ron
On 29/06/2017 9:42 AM, Will Stevens wrote:
I personally don't know how Jira solves any of this, but
assuming it does, fine...
The bigger problem which you have raised is that CloudStack
has zero funding. So we can't hire a project manager, or a
release manager or someone whose job it is to maintain
documentation. I have been trying to find a way to, at the
very least, fund a full time release manager who can focus
100% on the project. As the release manager for 4.9, I know
it is a full time job. I did my best, but it is a ton of work and is hard to
stay on top of.
Everyone contributing to CloudStack is donating their time.
They can't make a living off supporting ACS, so every one is
doing their best with the little time they can take away from
their day job or their family life.
Yes, having clear guidelines and sticking to them helps, but
without a solid CI infrastructure backing the project and
improved testing and automation, we will always struggles
with release schedules and such.
I have been involved in this project long enough to know that
all the problems you point out exist, but they are also not easily solved.
Obviously we have to work with the initiatives we have and
take small steps towards improvement, but we also have to be
realistic with our expectations because we are counting on
people's generosity to move them forward.
Simplifying moving parts and streamlining the process will
lead to more contribution because there is less barriers to
entry. This one reason why I struggle to see the value in Jira as it is used
today.
I personally don't understand what value it is giving us that
the github PRs and Issues don't solve.
I will remain open minded and will follow along with what
people think is best, but I think it is worth understanding
what we are trying to solve for and simplify our approach in
solving it so we can get better systems in place.
On Jun 29, 2017 9:17 AM, "Ron Wheeler"
<rwhee...@artifact-software.com>
wrote:
As a real outsider, IMHO Paul is right.
At times it seems that Cloudstack is a coding hobby rather
than a project or a production quality product.
Who decides what goes into a release? How does this affect
the release schedule?
Who is responsible for meeting the "published" roadmap (of
which there seem to be many) of releases?
How is a system admin that is not part of the project
supposed to plan for upgrade windows?
How does one know when a feature, bug fix or release will be
available?
How does the PMC manage function creep in a release, maintain
quality and consistency, reject changes that hurt the
overall vision or add too much complexity?
No one seems to care about documentation but if someone did,
how would they stop undocumented features or features that
contradict the documentation from being incorporated?
Who makes sure that the documentation is correct at the time
of the release?
Release notes are not much help for someone doing a new
install or evaluating Cloudstack.
Without a JIRA entry, how does an end-user who encounters a
problem know that it has been fixed already in the next release?
Without a JIRA entry, how does the community comment on a
proposed change before it gets coded?
If changes are going to be accepted without a JIRA, is there
a definition of a minor fix that does not require a JIRA?
- does not change functionality?
- only affects an "edge case" or cleans up an exception that
is not properly handled?
- only improves code readability or future extensibility?
- does not affect documentation?
Apache projects that are popular and enjoy wide support do
have strong management.
There are other examples where great Apache software is
failing to get recognized because the PMC is not paying
attention to the product management side of things.
I use Apache Jackrabbit which is a quality product with a
strong technical team supporting it.
It has very little following because the documentation and
marketing collateral is very poor.
It gets by because the audience for it is largely software
developers who can read code and can test features to work
out the functionality.
It would get a lot more attention if they paid attention to
the product management side of the project.
Cloudstack needs to avoid this situation and unfortunately
this takes effort and some discipline.
Ron
On 29/06/2017 8:03 AM, Will Stevens wrote:
Why are we still using jira instead of the PRs for that
communication? Can we not use issues in github now instead
of jira if someone needs to open an issue but does not yet
have code to contribute. If not, jira could still be used for that.
I think duplicating data between jira and the PR is kind of
pointless. I feel like the github PRs and the cide going in
should be the source of truth, not a random third party tool.
For the 4.9 release notes, i built a tool to generate the
release notes from the PRs merged in that release. I think
that is easier and more accurate than depending on jira
since it does not track the actual code tree.
Thats my 0.02$.
On Jun 29, 2017 5:25 AM, "Paul Angus"
<paul.an...@shapeblue.com>
wrote:
Such a view of CloudStack is what holds CloudStack back.
It stops users/operators from having any chance of
understanding what CloudStack does and how it does it.
Code for code's sake is no use to anyone.
Jira is about communication between developers and to everyone else.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
Sent: 29 June 2017 10:14
To: dev <dev@cloudstack.apache.org>
Subject: Re: JIRA - PLEASE READ
On Thu, Jun 29, 2017 at 11:06 AM, Paul Angus
<paul.an...@shapeblue.com>
wrote:
+ Release notes will be impossible to create without a
+ proper Jira
history.
And no one will know what has gone into CloudStack.
No they are not mr Grumpy. they should be base on the code
anyway,
hence on git, not jira. I do not appose to the use of Jira
but it is not required for good coding practices and as we
are not and will not function as a corporation, jira is an
extra for those that grave for it. not a requirement.
--
Daan
