I'm currently in holidays but I could not resist to take some time and
reply.
On 07/18/2015 06:32 PM, Dmitry Borodaenko wrote:
It has been 5 weeks since Emilien has asked Fuel developers to
contribute more
actively to Puppet OpenStack project [0]. We had a lively discussion on
openstack-dev, myself and other Fuel developers proposed some concrete
steps
that could be done to reconcile the two projects, the whole thread was
reported
and discussed on Linux Weekly News [1], and the things were looking up.
[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066544.html
[1] https://lwn.net/Articles/648331/
And now, 5 weeks later, Emilien has reported to the Technical Committee
that
there has been no progress on reconciliation between Fuel and Puppet
OpenStack,
and used his authority as a PTL to request that the Fuel's proposal to join
OpenStack [2] is rejected.
1) What you're writing here is not correct.
Please read again my first comment on the Governance patch:
"They are making progress and I'm sure they are doing their best."
I've been following all your actions since our *thread*, and so far all
of this seems good to me, I'm honestly satisfied to know that you guys
have plans.
2) The word 'authority' is also wrong. I'm not part of TC and my vote
does not count at all for the final decision. If Fuel has to be an
OpenStack project, TC is free to vote by themselves without me and it
seems they already negatively voted before my thoughts.
3) I -1 the governance patch because to me, it seems too early for
having Fuel part of OpenStack. We ran that discussion 5 weeks ago, a lot
of actions have been taken for I hope TC will wait for actual results
(mainly highlighted in the review itself).
Let me quote my first comment on the patch that clearly show I'm not
against the idea:
"I sincerely hope they'll realize what they plan to do, which is being
part of our community like other projects already succeed to do."
[2] https://review.openstack.org/199232
In further comments on the same review, Emilien has claimed that there's
"clearly less" contribution to Puppet OpenStack from Fuel developers than
before, and even brought up an example of a review in puppet-horizon
that was
proposed and then abandoned by Fuel team [3]. Jay went as far as calling
that
example "an obvious failure of working with the upstream Puppet-OpenStack
community".
Andrew took the best metric in your advantage... the review metric,
which is something given to anyone having signed the CLA.
I would rather focus on patchset, commits, bug fix, IRC, ML etc... which
really show collaboration in a group.
And I honestly think it's making progress, even though the numbers.
[3] https://review.openstack.org/198119
Needless to say, I found all these claims deeply disturbing, and had to
look
closely into what's going on.
The case of the puppet-horizon commit has turned out to be surprisingly
obvious.
Even before looking into the review comments, I could see a technical
reason
for abandoning the commit: if there is a bug in a component, fixing that
bug in
the package is preferrable to fixing it in puppet, because it allows
anybody to
benefit from the fix, not just the people deploying that package with
puppet.
You are not providing official Ubuntu packaging, but your own packages
mainly used by Fuel, while Puppet OpenStack modules are widely used by
OpenStack community.
Fixing that bug in Fuel packaging is the shortest & easiest way for you
to fix that, while we are really doing something wrong in puppet-horizon
about the 'compress' option.
So Fuel is now fixed and puppet-horizon broken.
And if you do look at the review in question, you'll find that
immediately (14
minutes, and that at 6pm on Friday night!) after Jay has asked in the
comments
to the review why it was abandoned, the commit author from the Fuel team
has
explained that this patch was a workaround for a packaging problem, and
that
this was pointed out in the review by a Horizon core reviewer more than
a week
ago, and later corroborated by a Puppet OpenStack core reviewer. Further
confirming that fixing this in the package instead of in puppet-horizon
was an
example of Fuel developers agreeing with other Puppet OpenStack
contributors
and doing the right thing.
This kind of bug has to be fixed between Horizon & Puppet team, and not
with some "workaround" in packaging tool.
We don't really want "workarounds" I guess, do we?
Emilien has clearly found this case important enough to report to the
TC, and
yet didn't find it important enough to simply ask Fuel developers why they
chose to abandon the commit. I guess you can call that an obvious
failure to
work together.
Looking further into Fuel team's reviews for puppet-horizon, I found
another
equally disturbing example [4].
[4] https://review.openstack.org/190548
Here's what I see in this review:
a) Fuel team has spent more than a month (since June 11) on trying to
land this
patch.
b) 2 out of 5 negative reviews are from Fuel team, proving that Fuel
developers
are not "rubberstamping" each other's commits as was implied by Emilien's
comments on the TC review above.
c) There was one patch set that was pushed 3 business days after a negative
review, all other patch sets (11 total) were pushed no later than next day
after a negative review.
All in all, I see great commitment and effort on behalf of Fuel team,
eventually awarded by a +2 from Michael Chapman.
On the same day (June 30), Emilien votes -2 for a correction in a
comment, and
walks away from the review for good. 18 days and 4 patch sets and 2
outstanding
+1's later, the review remains blocked by that -2. Reaching out on
#puppet-openstack [5] didn't help, either.
[5] http://irclog.perlgeek.de/puppet-openstack/2015-07-08#i_10867124
I -2 for this reason: this patch already had a +2 so it was closed to be
merged by someone else, while the patch was *breaking backward
compatiblity* in puppet-horizon, because Vasyl was changing the default
cache driver to force users to use Memcached by default.
We *never* break backward compatibility (this is a general rule in
OpenStack) and *never* change OpenStack default configuration (general
rule in Configuration management), but rather provide the interface
(called Puppet parameters) to change it.
At the same time, Emilien has commented on the TC review that the only
metric
he considers reflective of Fuel's contribution to Puppet OpenStack is
merged
commits. Isn't that a Catch-22 situation, requesting more merged commits
and
refusing to merge them?
When I compare the trends in the number of patch sets pushed by Fuel
developers
[6] against the number of merged commits [7], I see the same picture. It's
really obvious when comparing both graphs side by side.
[6]
http://stackalytics.com/?module=puppetopenstack-group&metric=patches&company=mirantis
[7]
http://stackalytics.com/?module=puppetopenstack-group&metric=commits&company=mirantis
Between May and July, the peak number of patchsets in Puppet OpenStack
from all
contributors increased by a factor of 1.78, while patchsets associated with
Mirantis jumped up by a factor of 6.75. At the same time, merged commits
from
all contributors increased by 2.44 (meaning 30% less patch sets per
commit than
in May), for Mirantis the factor is only 2.0 (meaning 3 times as many patch
sets per commit as in May).
Just look at the attached picture to see how obviously massive is the
increase
in the effort that Fuel team contributes to Puppet OpenStack. One can
argue for
different ways to explain why this effort does not convert into merged
commits,
but based on two examples above I find it very hard to believe that it
can be
explained by a lack of trying.
In fact, this is exactly the situation that I have predicted in the
original
thread when I said that commmitment to collaborate has to be mutual [8].
/me happy
I'll see the review backlog with Puppet group and see if we lack in reviews.
[8]
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066731.html
Fuel team can commit to reporting bugs and posting patches, Fuel team
can spend
reasonable amount of time and effort to respond to review comments, but all
that is worth nothing if there's no commitment from Puppet OpenStack
team to
review and merge these patches in a timely manner.
Please consider our group is part of a few people comparing to Fuel's
team, and we are not all working 100% on Puppet OpenStack (I'm lucky, I
do). Please take that in consideration.
Looking through the collaboration thread, the only commitment I see is
Emilien's promise that "if you submit a good patch now, it will land in
maximum
one week or so", and his comment on the TC review: "The end of the
thread was
kind of consensus to me and I was pretty happy to see Dmitry was really
open to
the discussion".
Clearly this kind of consensus is not working out, and I think at least one
reason is that my proposal for Puppet OpenStack developers to take over
stalled
commits was rejected. One way or another, cases like "16 days since the
first
patch set that had no negative reviews" that obviously violate the "land
within
a week" commitment have to be identified and prioritized. Being on the
hook to
take them over if they don't land on time is one way to introduce a
motivation
to make that happen. There could be other ways to address it, for example
having a review dashboard that highlights stuck reviews (as I proposed
in the
same email) would help add some visibility into the problem.
We have weekly review/bug triage, feel free to participate, it happens
during our meetings.
Please don't get me wrong. In the same reviews I've highlighted above I see
excellent example of collaboration from Puppet OpenStack team,
insightful and
timely comments, and I'm not denying the effort the team spends on
onboarding
new contributors flooding in from Fuel. But I don't see how this
situation can
be called "no progress" and "reduced contribution from Fuel" when making
further progress is limited by Puppet OpenStack team's ability to review
incoming patches much more than by Fuel team's commitment to proposing and
updating them.
Please don't get me wrong, but you're aggravating the situation.
We are having much more collaboration than before.
Fuel's team is much more involved on reviews as you highlighted, and
also on IRC, which is good.
Of course there are still technical things that will be improved with
more experience in working with Puppet OpenStack modules, but so far all
Puppet grouped noticed more collaboration I think.
I totally assume my -1 for adding Fuel to OpenStack because it's too
early now, even all your recent efforts.
Now, I think we just need you to let us more time, so your engineers and
our group can work together. I'm pretty sure the things will go
naturally like they happened for our other contributors.
Rather than doing yet another very long thread, maybe can we arrange a
public meeting on IRC and we can do some triage together. After my
holidays, I can even organize a 1 or 2 hour meeting so we can solve our
first blockers and make progress on both sides. I'll try to get
attention from our group if some help is needed.
What do you think?
Best regards,
Post-scriptum: I'm back online on July 23.
---
Emilien Macchi
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
