On Wed, 22 Aug 2018 09:49:13 -0400, Doug Hellmann wrote:
Excerpts from melanie witt's message of 2018-08-21 15:05:00 -0700:
On Tue, 21 Aug 2018 16:41:11 -0400, Doug Hellmann wrote:
Excerpts from melanie witt's message of 2018-08-21 12:53:43 -0700:
On Tue, 21 Aug 2018 06:50:56 -0500, Matt Riedemann wrote:
At this point, I think we're at:
1. Should placement be extracted into it's own git repo in Stein while
nova still has known major issues which will have dependencies on
placement changes, mainly modeling affinity?
2. If we extract, does it go under compute governance or a new project
with a new PTL.
As I've said, I personally believe that unless we have concrete plans
for the big items in #1, we shouldn't hold up the extraction. We said in
Dublin we wouldn't extract to a new git repo in Rocky but we'd work up
to that point so we could do it in Stein, so this shouldn't surprise
anyone. The actual code extraction and re-packaging and all that is
going to be the biggest technical issue with all of this, and will
likely take all of stein to complete it after all the bugs are shaken out.
For #2, I think for now, in the interim, while we deal with the
technical headache of the code extraction itself, it's best to leave the
new repo under compute governance so the existing team is intact and we
don't conflate the people issue with the technical issue at the same
time. Get the hard technical part done first, and then we can move it
out of compute governance. Once it's in its own git repo, we can change
the core team as needed but I think it should be initialized with
existing nova-core.
I'm in support of extracting placement into its own git repo because
Chris has done a lot of work to reduce dependencies in placement and
moving it into its own repo would help in not having to keep chasing
that. As has been said before, I think all of us agree that placement
should be separate as an end goal. The question is when to fully
separate it from governance.
It's true that we don't have concrete plans for affinity modeling and
shared storage modeling. But I think we do have concrete plans for vGPU
enhancements (being able to have different vGPU types on one compute
host and adding support for traits). vGPU support is an important and
highly sought after feature for operators and users, as we witnessed at
the last Summit in Vancouver. vGPU support is currently using a flat
resource provider structure that needs to be migrated to nested in order
to do the enhancement work, and that's how the reshaper work came about.
(Reshaper work will migrate a flat resource provider structure to a
nested one.)
We have the nested resource provider support in placement but we need to
integrate the Nova side, leveraging the reshaper code. The reshaper code
is still going through code review, then next we have the integration to
do. I think things are bound to break when we integrate it, just because
nothing is ever perfect, as much as we scrutinize it and the real test
is when we start using it for real. I think going through this
integration would be best done *before* extraction to a new repo. But
given that there is never a "good" time to extract something to a new
repo, I am OK with the idea of doing the extraction first, if that is
what most people want to do.
What I'm concerned about on the governance piece is how things look as
far as project priorities between the two projects if they are split.
Affinity modeling and shared storage support are compute features
OpenStack operators and users need. Operators need affinity modeling in
the placement is needed to achieve parity for affinity scheduling with
multiple cells. That means, affinity scheduling in Nova with multiple
cells is susceptible to races and does *not* work as well as the
previous single cell support. Shared storage support is something
operators have badly needed for years now and was envisioned to be
solved with placement.
Given all of that, I'm not seeing how *now* is a good time to separate
the placement project under separate governance with separate goals and
priorities. If operators need things for compute, that are well-known
and that placement was created to solve, how will placement have a
shared interest in solving compute problems, if it is not part of the
compute project?
Who are candidates to be members of a review team for the placement
repository after the code is moved out of openstack/nova?
How many of them are also members of the nova-core team?
I assume you pose this question in the proposed situation I described
where placement is a repo under compute. I expect the review team to be
No, not at all. I'm trying to understand how you think a completely
separate team is going to cause problems. Because it seems like at
least a large portion, if not all, of the contributors want it, and
I need to have a very good reason for denying their request, if we
do. Right now, I understand that there are concerns, but I don't
understand why.
I have been trying to explain why over several replies to this thread.
Fracturing a group is not something anyone does to foster cooperation
and shared priorities and goals. It is my job and responsibility to
ensure that features that operators and users need, get delivered to
them. We have a list of specific features that we know operators and
users need, right now. And fracturing the group is something that will
negatively impact our ability to implement them. Working farther apart
will negatively impact our ability to coordinate work that is closely
coupled. The last thing I want to do right now is build walls. I do not
want there to be separate goal-setting and priorities discussions -- I
want us to work together, coordinating as one group.
At the very least, separating the groups is something to be taken very
seriously and is not reversible. I want to be very sure it will be
better for operators and users relying on us, if the groups were
separated. I haven't heard any reasons why it would be, yet.
Thus, I would like to see an incremental approach, given the outstanding
work items we know we have. I would like to see us make changes and
*try* to create an environment where people can work together in one
group, before we go to the extreme option and separate everything
completely. I would like to extract the code into its own repo under
compute, with the expectation that it is to evolve *independently* of
Nova code, and with a subset of nova-core, plus a new placement-core
team with placement experts added to it. And we see how that goes. Then,
we reassess. If everyone gives that an earnest try, and it is not
working for people, then we will know for sure that we tried to make
changes and stay together as one group, and it didn't work, and
separating into two groups in the middle of the work items we have, is
what we have to do.
What do you think those folks are more interested in working on than the
things you listed as needing to be done to support the nova use cases?
I'm not thinking of anything specific here. At a high level, I don't see
how separating into two separate groups under separate leadership helps
us deliver the listed things for operators and users. I tend to think
that a unified group will be more successful at that.
OK. At the same time, I'm trying to understand why you have a hard
time believing a new team's priorities would not be aligned with
the nova team's priorities if, as it seems, a large percentage of
that new team would be made up of the same exact people.
I think it's about context. If two separate projects do their own
priority and goal setting, separately, I think they will naturally be
more different than they would be if they were one project. Currently,
we agree on goals and priorities together, in the compute context. If
placement has its own separate context, the priority setting and goal
planning will be done in the context of placement. In two separate
groups, someone who is a member of both the Nova and Placement teams
would have to persuade Placement-only members to agree to prioritize a
particular item. This may sound subtle, but it's a notable difference in
how things work when it's one team vs two separate teams. I think having
shared context and alignment, at this point in time, when we have
outstanding closely coupled nova/placement work to do, is critical in
delivering for operators and users who are depending on us.
What can they do to reassure you that they will work on the items
nova needs, regardless of the governance structure?
If they were separate groups, I don't see why the leadership of
placement would necessarily share goals and priorities with compute. I
think that is why it's much more difficult to get things done with two
separate groups, in general.
Most of the teams in the community seem to have a relatively easy time
coming to common agreement on priorities and goals, even when they are
not so closely related as the nova and placement teams would be. So I
guess I still don't see what the problem would be, and am looking for
more details, either about the concerns you all have or ways to
alleviate them.
The "relatively easy time" sounds a bit vague to me. My experience with
cross-project work (multi-attach) has been that someone has to serve as
a champion of a feature or effort, and persuade other team to care about
it and work together on it. If the team is sufficiently persuaded, they
agree to collaborate on a thing. Then, they go into their silos and do
some work on it. Then, the champion checks in on the teams after some
time passes. The champion gathers problems and blockers and sets up a
meeting between the two teams or takes the messages back and forth
themselves. The teams do some more work in their silos. The champion
checks in again after some time, setting up a meeting to re-synchronize
the teams. And this continues until the feature is complete, if the
champion has been around that long.
This can work, but it's a much different experience than being one team.
Being that we are still in the thick of it, and need to work on the
items I have mentioned before in my other replies, I don't think *now*
is a good time to separate groups. Separating now will hurt operators,
users, and customers. Separating now will make already challenging work
even more challenging, with more barriers to climb over and more effort
to stay connected. That is why I'm concerned.
I want to reiterate again that the only thing I care about here is
delivering functionality that operators and users need. vGPUs, in
particular, has been highly sought after at a community-wide level, not
just from the compute community. I want to deliver the features that
people are depending on and IMHO, being a unified group helps that. I
don't see how being two separate groups helps that.
I don't think doing those things is mutually exclusive with solving,
or at least addressing, the underlying trust and self-governance
issues here, though. In fact, I think dealing with *that* is going
to make us all more effective at delivering the software, in the
long term because we will have cleared up what the expectations
between the two teams is.
To be clear, there are not underlying trust issues with most of the
team. A trust issue was expressed by one member of the team and I think
it's unfair to apply it to everyone else.
Aside from that, it has always been difficult to add folks to nova-core
because of the large scope and expertise needed to approve code across
all of Nova. Extracting the placement code into its own repo alleviates
the problem of the massive scope and gives us the ability to have a
placement-core team. And I expect the placement repo to evolve
independently of Nova code, as it's been intended to be a separate
project, eventually.
My hope is that making these changes will improve the experience in the
placement subteam and compute as a whole. I hope that the idea of a
compromise to try it out will be amenable to everyone. I don't take the
idea of separating the groups completely, at this point in time, with
outstanding closely coupled nova/placement work, lightly. I think it
should be taken very seriously and done with care. And taking a step,
trying it out, and then reassessing, seems like the most prudent way to
go about it, IMO.
Best,
-melanie
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
