Re: [openstack-dev] [Neutron] Group Based Policy and the way forward

[Maru Newby]

On Aug 8, 2014, at 10:56 AM, Kevin Benton  wrote:

> There is an enforcement component to the group policy that allows you to use 
> the current APIs and it's the reason that group policy is integrated into the 
> neutron project. If someone uses the current APIs, the group policy plugin 
> will make sure they don't violate any policy constraints before passing the 
> request into the regular core/service plugins.


The enforcement requirement might be easier to implement through code-based 
integration, but a separate service could provide the same guarantee against 
constraint violation by proxying v2 API calls for an endpoint to which access 
was restricted.

Apologies if I've missed discussion of this (it's a big thread), but won't 
enforcement by group policy on the v2 API have the potential to violate 
stability requirements?  If new errors related to enforcement can result from 
API calls, that would seem to be a change in behavior.  Do we have a precedent 
for allowing extensions or new services to modify core behavior in this way?


m. 

> 
> 
> On Fri, Aug 8, 2014 at 11:02 AM, Salvatore Orlando  
> wrote:
> It might be because of the wording used, but it seems to me that you're 
> making it sound like the group policy effort could have been completely 
> orthogonal to neutron as we know it now.
> 
> What I understood is that the declarative abstraction offered by group policy 
> could do without any existing neutron entity leveraging "native" drivers, but 
> can actually be used also with existing neutron plugins through the mapping 
> driver - which will provide a sort of backward compatibility. And still in 
> that case I'm not sure one would be able to use "traditional" neutron API (or 
> "legacy" as it has been called), since I don't know if the mapping driver is 
> bidirectional.
> 
> I know this probably stems from my ignorance on the subject - I had 
> unfortunately very little time to catch-up with this effort in the past 
> months.
> 
> Salvatore
> 
> 
> On 8 August 2014 18:49, Ivar Lazzaro  wrote:
> Hi Jay,
> 
> You can choose. The whole purpose of this is about flexibility, if you want 
> to use GBP API 'only' with a specific driver you just can.
> Additionally, given the 'ML2 like' architecture, the reference mapping driver 
> can ideally run alongside by filling the core Neutron constructs without ever 
> 'disturbing' your own driver (I'm not entirely sure about this but it seems 
> feasible).
> 
> I hope this answers your question,
> Ivar.
> 
> 
> On Fri, Aug 8, 2014 at 6:28 PM, Jay Pipes  wrote:
> On 08/08/2014 08:55 AM, Kevin Benton wrote:
> The existing constructs will not change.
> 
> A followup question on the above...
> 
> If GPB API is merged into Neutron, the next logical steps (from what I can 
> tell) will be to add drivers that handle policy-based payloads/requests.
> 
> Some of these drivers, AFAICT, will *not* be deconstructing these policy 
> requests into the low-level port, network, and subnet 
> creation/attachment/detachment commands, but instead will be calling out 
> as-is to hardware that speaks the higher-level abstraction API [1], not the 
> lower-level port/subnet/network APIs. The low-level APIs would essentially be 
> consumed entirely within the policy-based driver, which would effectively 
> mean that the only way a system would be able to orchestrate networking in 
> systems using these drivers would be via the high-level policy API.
> 
> Is that correct? Very sorry if I haven't explained clearly my question... 
> this is a tough question to frame eloquently :(
> 
> Thanks,
> -jay
> 
> [1] 
> http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html
> 
> On Aug 8, 2014 9:49 AM, "CARVER, PAUL"  > wrote:
> 
> Wuhongning [mailto:wuhongn...@huawei.com
> ] wrote:
> 
>  >Does it make sense to move all advanced extension out of ML2, like
> security
>  >group, qos...? Then we can just talk about advanced service
> itself, without
>  >bothering basic neutron object (network/subnet/port)
> 
> A modular layer 3 (ML3) analogous to ML2 sounds like a good idea. I
> still
> think it's too late in the game to be shooting down all the work
> that the
> GBP team has put in unless there's a really clean and effective way of
> running AND iterating on GBP in conjunction with Neutron without being
> part of the Juno release. As far as I can tell they've worked really
> hard to follow the process and accommodate input. They shouldn't have
> to wait multiple more releases on a hypothetical refactoring of how
> L3+ vs
> L2 is structured.
> 
> But, just so I'm not making a horrible mistake, can someone reassure me
> that GBP isn't removing the constructs of network/subnet/port from
> Neutron?
> 
> I'm under the impression that GBP is adding a higher level abstraction
> but that it's 

[openstack-dev] Fwd: [nova][core] Expectations of core reviewers

[Maru Newby]

On Aug 13, 2014, at 2:57 AM, Daniel P. Berrange  wrote:

> On Wed, Aug 13, 2014 at 08:57:40AM +1000, Michael Still wrote:
>> Hi.
>> 
>> One of the action items from the nova midcycle was that I was asked to
>> make nova's expectations of core reviews more clear. This email is an
>> attempt at that.
>> 
>> Nova expects a minimum level of sustained code reviews from cores. In
>> the past this has been generally held to be in the order of two code
>> reviews a day, which is a pretty low bar compared to the review
>> workload of many cores. I feel that existing cores understand this
>> requirement well, and I am mostly stating it here for completeness.
>> 
>> Additionally, there is increasing levels of concern that cores need to
>> be on the same page about the criteria we hold code to, as well as the
>> overall direction of nova. While the weekly meetings help here, it was
>> agreed that summit attendance is really important to cores. Its the
>> way we decide where we're going for the next cycle, as well as a
>> chance to make sure that people are all pulling in the same direction
>> and trust each other.
>> 
>> There is also a strong preference for midcycle meetup attendance,
>> although I understand that can sometimes be hard to arrange. My stance
>> is that I'd like core's to try to attend, but understand that
>> sometimes people will miss one. In response to the increasing
>> importance of midcycles over time, I commit to trying to get the dates
>> for these events announced further in advance.
> 
> Personally I'm going to find it really hard to justify long distance
> travel 4 times a year for OpenStack for personal / family reasons,
> let alone company cost. I couldn't attend Icehouse mid-cycle because
> I just had too much travel in a short time to be able to do another
> week long trip away from family. I couldn't attend Juno mid-cycle
> because it clashed we personal holiday. There are other opensource
> related conferences that I also have to attend (LinuxCon, FOSDEM,
> KVM Forum, etc), etc so doubling the expected number of openstack
> conferences from 2 to 4 is really very undesirable from my POV.
> I might be able to attend the occassional mid-cycle meetup if the
> location was convenient, but in general I don't see myself being
> able to attend them regularly.
> 
> I tend to view the fact that we're emphasising the need of in-person
> meetups to be somewhat of an indication of failure of our community
> operation. The majority of open source projects work very effectively
> with far less face-to-face time. OpenStack is fortunate that companies
> are currently willing to spend 6/7-figure sums flying 1000's of
> developers around the world many times a year, but I don't see that
> lasting forever so I'm concerned about baking the idea of f2f midcycle
> meetups into our way of life even more strongly.

I was fortunate to attend both the Nova and Neutron mid-cycles last month, and 
I can attest to how productive these gatherings were.  Discussion moved quickly 
and misunderstandings were rapidly resolved.  Informal ('water-cooler') 
conversation led to many interactions that might not otherwise have occurred.  
Given your attendance of summit and other open source conferences, though, I'm 
assuming the value of f2f is not in question.

Nothing good is ever free.  The financial cost and exclusionary nature of an 
in-person meetup should definitely be weighed against the opportunity for 
focused and high-bandwidth communication.  It's clear to myself and other 
attendees just how valuable the recent mid-cycles were in terms of making 
technical decisions and building the relationships to support their 
implementation.  Maybe it isn't sustainable over the long-term to meet so 
often, but I don't think that should preclude us from deriving benefit in the 
short-term.  I also don't think we should ignore the opportunity for more 
effective decision-making on the grounds that not everyone can directly 
participate.  Not everyone is able to attend summit, but it is nonetheless a 
critical part of our community's decision-making process.  The topic lists for 
a mid-cycle are published beforehand, just like summit, to allow non-attendees 
the chance to present their views in advance and/or designate one or more 
attendees to advocate on
  their behalf.  It's not perfect, but the alternative - not holding mid-cycles 
- would seem to be a case of throwing out the baby with the bathwater.


Maru

> 
>> Given that we consider these physical events so important, I'd like
>> people to let me know if they have travel funding issues. I can then
>> approach the Foundation about funding travel if that is required.
> 
> Travel funding is certainly an issue, but I'm not sure that Foundation
> funding would be a solution, because the impact probably isn't directly
> on the core devs. Speaking with my Red Hat on, if the midcycle meetup
> is important enough, the core devs will likely get the funding to attend.
> The fall

Re: [openstack-dev] [nova][core] Expectations of core reviewers

[Maru Newby]

On Aug 13, 2014, at 2:57 AM, Daniel P. Berrange  wrote:

> On Wed, Aug 13, 2014 at 08:57:40AM +1000, Michael Still wrote:
>> Hi.
>> 
>> One of the action items from the nova midcycle was that I was asked to
>> make nova's expectations of core reviews more clear. This email is an
>> attempt at that.
>> 
>> Nova expects a minimum level of sustained code reviews from cores. In
>> the past this has been generally held to be in the order of two code
>> reviews a day, which is a pretty low bar compared to the review
>> workload of many cores. I feel that existing cores understand this
>> requirement well, and I am mostly stating it here for completeness.
>> 
>> Additionally, there is increasing levels of concern that cores need to
>> be on the same page about the criteria we hold code to, as well as the
>> overall direction of nova. While the weekly meetings help here, it was
>> agreed that summit attendance is really important to cores. Its the
>> way we decide where we're going for the next cycle, as well as a
>> chance to make sure that people are all pulling in the same direction
>> and trust each other.
>> 
>> There is also a strong preference for midcycle meetup attendance,
>> although I understand that can sometimes be hard to arrange. My stance
>> is that I'd like core's to try to attend, but understand that
>> sometimes people will miss one. In response to the increasing
>> importance of midcycles over time, I commit to trying to get the dates
>> for these events announced further in advance.
> 
> Personally I'm going to find it really hard to justify long distance
> travel 4 times a year for OpenStack for personal / family reasons,
> let alone company cost. I couldn't attend Icehouse mid-cycle because
> I just had too much travel in a short time to be able to do another
> week long trip away from family. I couldn't attend Juno mid-cycle
> because it clashed we personal holiday. There are other opensource
> related conferences that I also have to attend (LinuxCon, FOSDEM,
> KVM Forum, etc), etc so doubling the expected number of openstack
> conferences from 2 to 4 is really very undesirable from my POV.
> I might be able to attend the occassional mid-cycle meetup if the
> location was convenient, but in general I don't see myself being
> able to attend them regularly.
> 
> I tend to view the fact that we're emphasising the need of in-person
> meetups to be somewhat of an indication of failure of our community
> operation. The majority of open source projects work very effectively
> with far less face-to-face time. OpenStack is fortunate that companies
> are currently willing to spend 6/7-figure sums flying 1000's of
> developers around the world many times a year, but I don't see that
> lasting forever so I'm concerned about baking the idea of f2f midcycle
> meetups into our way of life even more strongly.

I was fortunate to attend both the Nova and Neutron mid-cycles last month, and 
I can attest to how productive these gatherings were.  Discussion moved quickly 
and misunderstandings were rapidly resolved.  Informal ('water-cooler') 
conversation led to many interactions that might not otherwise have occurred.  
Given your attendance of summit and other open source conferences, though, I'm 
assuming the value of f2f is not in question.

Nothing good is ever free.  The financial cost and exclusionary nature of an 
in-person meetup should definitely be weighed against the opportunity for 
focused and high-bandwidth communication.  It's clear to myself and other 
attendees just how valuable the recent mid-cycles were in terms of making 
technical decisions and building the relationships to support their 
implementation.  Maybe it isn't sustainable over the long-term to meet so 
often, but I don't think that should preclude us from deriving benefit in the 
short-term.  I also don't think we should ignore the opportunity for more 
effective decision-making on the grounds that not everyone can directly 
participate.  Not everyone is able to attend summit, but it is nonetheless a 
critical part of our community's decision-making process. The topic lists for a 
mid-cycle are published beforehand, just like summit, to allow non-attendees 
the chance to present their views in advance and/or designate one or more 
attendees to advocate on 
 their behalf.  It's not perfect, but the alternative - not holding mid-cycles 
- would seem to be a case of throwing out the baby with the bathwater.


Maru

> 
>> Given that we consider these physical events so important, I'd like
>> people to let me know if they have travel funding issues. I can then
>> approach the Foundation about funding travel if that is required.
> 
> Travel funding is certainly an issue, but I'm not sure that Foundation
> funding would be a solution, because the impact probably isn't directly
> on the core devs. Speaking with my Red Hat on, if the midcycle meetup
> is important enough, the core devs will likely get the funding to attend.
> The fallo

[openstack-dev] Fwd: [nova][core] Expectations of core reviewers

[Maru Newby]

My apologies, I managed to break the thread here.  Please respond to the thread 
with subject 'Re: [openstack-dev] [nova][core] Expectations of core reviewers' 
in preference to this one.


Maru

On Aug 13, 2014, at 9:01 AM, Maru Newby  wrote:

> 
> On Aug 13, 2014, at 2:57 AM, Daniel P. Berrange  wrote:
> 
>> On Wed, Aug 13, 2014 at 08:57:40AM +1000, Michael Still wrote:
>>> Hi.
>>> 
>>> One of the action items from the nova midcycle was that I was asked to
>>> make nova's expectations of core reviews more clear. This email is an
>>> attempt at that.
>>> 
>>> Nova expects a minimum level of sustained code reviews from cores. In
>>> the past this has been generally held to be in the order of two code
>>> reviews a day, which is a pretty low bar compared to the review
>>> workload of many cores. I feel that existing cores understand this
>>> requirement well, and I am mostly stating it here for completeness.
>>> 
>>> Additionally, there is increasing levels of concern that cores need to
>>> be on the same page about the criteria we hold code to, as well as the
>>> overall direction of nova. While the weekly meetings help here, it was
>>> agreed that summit attendance is really important to cores. Its the
>>> way we decide where we're going for the next cycle, as well as a
>>> chance to make sure that people are all pulling in the same direction
>>> and trust each other.
>>> 
>>> There is also a strong preference for midcycle meetup attendance,
>>> although I understand that can sometimes be hard to arrange. My stance
>>> is that I'd like core's to try to attend, but understand that
>>> sometimes people will miss one. In response to the increasing
>>> importance of midcycles over time, I commit to trying to get the dates
>>> for these events announced further in advance.
>> 
>> Personally I'm going to find it really hard to justify long distance
>> travel 4 times a year for OpenStack for personal / family reasons,
>> let alone company cost. I couldn't attend Icehouse mid-cycle because
>> I just had too much travel in a short time to be able to do another
>> week long trip away from family. I couldn't attend Juno mid-cycle
>> because it clashed we personal holiday. There are other opensource
>> related conferences that I also have to attend (LinuxCon, FOSDEM,
>> KVM Forum, etc), etc so doubling the expected number of openstack
>> conferences from 2 to 4 is really very undesirable from my POV.
>> I might be able to attend the occassional mid-cycle meetup if the
>> location was convenient, but in general I don't see myself being
>> able to attend them regularly.
>> 
>> I tend to view the fact that we're emphasising the need of in-person
>> meetups to be somewhat of an indication of failure of our community
>> operation. The majority of open source projects work very effectively
>> with far less face-to-face time. OpenStack is fortunate that companies
>> are currently willing to spend 6/7-figure sums flying 1000's of
>> developers around the world many times a year, but I don't see that
>> lasting forever so I'm concerned about baking the idea of f2f midcycle
>> meetups into our way of life even more strongly.
> 
> I was fortunate to attend both the Nova and Neutron mid-cycles last month, 
> and I can attest to how productive these gatherings were.  Discussion moved 
> quickly and misunderstandings were rapidly resolved.  Informal 
> ('water-cooler') conversation led to many interactions that might not 
> otherwise have occurred.  Given your attendance of summit and other open 
> source conferences, though, I'm assuming the value of f2f is not in question.
> 
> Nothing good is ever free.  The financial cost and exclusionary nature of an 
> in-person meetup should definitely be weighed against the opportunity for 
> focused and high-bandwidth communication.  It's clear to myself and other 
> attendees just how valuable the recent mid-cycles were in terms of making 
> technical decisions and building the relationships to support their 
> implementation.  Maybe it isn't sustainable over the long-term to meet so 
> often, but I don't think that should preclude us from deriving benefit in the 
> short-term.  I also don't think we should ignore the opportunity for more 
> effective decision-making on the grounds that not everyone can directly 
> participate.  Not everyone is able to attend summit, but it is nonetheless a 
> critical part of our community's decision-making pro

Re: [openstack-dev] [neutron] Rotating the weekly Neutron meeting

[Maru Newby]

+1


Maru


On Aug 13, 2014, at 7:05 AM, Kyle Mestery  wrote:

> Per this week's Neutron meeting [1], it was decided that offering a
> rotating meeting slot for the weekly Neutron meeting would be a good
> thing. This will allow for a much easier time for people in
> Asia/Pacific timezones, as well as for people in Europe.
> 
> So, I'd like to propose we rotate the weekly as follows:
> 
> Monday 2100UTC
> Tuesday 1400UTC
> 
> If people are ok with these time slots, I'll set this up and we'll
> likely start with this new schedule in September, after the FPF.
> 
> Thanks!
> Kyle
> 
> [1] 
> http://eavesdrop.openstack.org/meetings/networking/2014/networking.2014-08-11-21.00.html
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Which changes need accompanying bugs?

[Maru Newby]

On Aug 13, 2014, at 11:11 AM, Kevin Benton  wrote:

> Is the pylint static analysis that caught that error prone to false
> positives? If not, I agree that it would be really nice if that were made
> part of the tox check so these don't have to be fixed after the fact.
> 
> To me that particular patch seems like one that should be accompanied with
> a unit test because it's a failure case with cleanup code that isn't being
> touched by the unit tests.

+1

As a general rule I would like to see test addition included with any fix 
targeting a bug that was merged due to a lack of coverage.


m.


> On Aug 13, 2014 8:34 AM, "Armando M."  wrote:
> 
>> I am gonna add more color to this story by posting my replies on review
>> [1]:
>> 
>> Hi Angus,
>> 
>> You touched on a number of points. Let me try to give you an answer to all
>> of them.
>> 
 (I'll create a bug report too. I still haven't worked out which class
>> of changes need an accompanying bug report and which don't.)
>> 
>> The long story can be read below:
>> 
>> https://wiki.openstack.org/wiki/BugFilingRecommendations
>> 
>> https://wiki.openstack.org/wiki/GitCommitMessages
>> 
>> IMO, there's a grey area for some of the issues you found, but when I am
>> faced with a bug, I tend to answer myself? Would a bug report be useful to
>> someone else? The author of the code? A consumer of the code? Not everyone
>> follow the core review system all the time, whereas Launchpad is pretty
>> much the tool everyone uses to stay abreast with the OpenStack release
>> cycle. Obviously if you're fixing a grammar nit, or filing a cosmetic
>> change that has no functional impact then I warrant the lack of a test, but
>> in this case you're fixing a genuine error: let's say we want to backport
>> this to icehouse, how else would we make the release manager of that?
>> He/she is looking at Launchpad.
>> 
 I can add a unittest for this particular code path, but it would only
>> check this particular short segment of code, would need to be maintained as
>> the code changes, and wouldn't catch another occurrence somewhere else.
>> This seems an unsatisfying return on the additional work :(
>> 
>> You're looking at this from the wrong perspective. This is not about
>> ensuring that other code paths are valid, but that this code path stays
>> valid over time, ensuring that the code path is exercised and that no other
>> regression of any kind creep in. The reason why this error was introduced
>> in the first place is because the code wasn't tested when it should have.
>> If you don't get that this mechanical effort of fixing errors by static
>> analysis is kind of ineffective, which leads me to the last point
>> 
 I actually found this via static analysis using pylint - and my
>> question is: should I create some sort of pylint unittest that tries to
>> catch this class of problem across the entire codebase? [...]
>> 
>> I value what you're doing, however I would see even more value if we
>> prevented these types of errors from occurring in the first place via
>> automation. You run pylint today, but what about tomorrow, or a week from
>> now? Are you gonna be filing pylint fixes for ever? We might be better off
>> automating the check and catch these types of errors before they land in
>> the tree. This means that the work you are doing it two-pronged: a)
>> automate the detection of some failures by hooking this into tox.ini via
>> HACKING/pep8 or equivalent mechanism and b) file all the fixes that require
>> these validation tests to pass; c) everyone is happy, or at least they
>> should be.
>> 
>> I'd welcome to explore a better strategy to ensure a better quality of the
>> code base, without some degree of automation, nothing will stop these
>> conversation from happening again.
>> 
>> Cheers,
>> 
>> Armando
>> 
>> [1] https://review.openstack.org/#/c/113777/
>> 
>> 
>> On 13 August 2014 03:02, Ihar Hrachyshka  wrote:
>> 
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA512
>>> 
>>> On 13/08/14 09:28, Angus Lees wrote:
 I'm doing various small cleanup changes as I explore the neutron
 codebase. Some of these cleanups are to fix actual bugs discovered
 in the code.  Almost all of them are tiny and "obviously correct".
 
 A recurring reviewer comment is that the change should have had an
 accompanying bug report and that they would rather that change was
 not submitted without one (or at least, they've -1'ed my change).
 
 I often didn't discover these issues by encountering an actual
 production issue so I'm unsure what to include in the bug report
 other than basically a copy of the change description.  I also
 haven't worked out the pattern yet of which changes should have a
 bug and which don't need one.
 
 There's a section describing blueprints in NeutronDevelopment but
 nothing on bugs.  It would be great if someone who understands the
 nuances here could add some words on when

Re: [openstack-dev] [nova][core] Expectations of core reviewers

[Maru Newby]

On Aug 14, 2014, at 8:52 AM, Russell Bryant  wrote:

> On 08/14/2014 11:40 AM, David Kranz wrote:
>> On 08/14/2014 10:54 AM, Matt Riedemann wrote:
>>> 
>>> 
>>> On 8/14/2014 3:47 AM, Daniel P. Berrange wrote:
 On Thu, Aug 14, 2014 at 09:24:36AM +1000, Michael Still wrote:
> On Thu, Aug 14, 2014 at 3:09 AM, Dan Smith  wrote:
>>> I'm not questioning the value of f2f - I'm questioning the idea of
>>> doing f2f meetings sooo many times a year. OpenStack is very much
>>> the outlier here among open source projects - the vast majority of
>>> projects get along very well with much less f2f time and a far
>>> smaller % of their contributors attend those f2f meetings that do
>>> happen. So I really do question what is missing from OpenStack's
>>> community interaction that makes us believe that having 4 f2f
>>> meetings a year is critical to our success.
>> 
>> How many is too many? So far, I have found the midcycles to be
>> extremely
>> productive -- productive in a way that we don't see at the summits,
>> and
>> I think other attendees agree. Obviously if budgets start limiting
>> them,
>> then we'll have to deal with it, but I don't want to stop meeting
>> preemptively.
> 
> I agree they're very productive. Let's pick on the nova v3 API case as
> an example... We had failed as a community to reach a consensus using
> our existing discussion mechanisms (hundreds of emails, at least three
> specs, phone calls between the various parties, et cetera), yet at the
> summit and then a midcycle meetup we managed to nail down an agreement
> on a very contentious and complicated topic.
 
 We thought we had agreement on v3 API after Atlanta f2f summit and
 after Hong Kong f2f too. So I wouldn't neccessarily say that we
 needed another f2f meeting to resolve that, but rather than this is
 a very complex topic that takes a long time to resolve no matter
 how we discuss it and the discussions had just happened to reach
 a natural conclusion this time around. But lets see if this agreement
 actually sticks this time
 
> I can see the argument that travel cost is an issue, but I think its
> also not a very strong argument. We have companies spending millions
> of dollars on OpenStack -- surely spending a relatively small amount
> on travel to keep the development team as efficient as possible isn't
> a big deal? I wouldn't be at all surprised if the financial costs of
> the v3 API debate (staff time mainly) were much higher than the travel
> costs of those involved in the summit and midcycle discussions which
> sorted it out.
 
 I think the travel cost really is a big issue. Due to the number of
 people who had to travel to the many mid-cycle meetups, a good number
 of people I work with no longer have the ability to go to the Paris
 design summit. This is going to make it harder for them to feel a
 proper engaged part of our community. I can only see this situation
 get worse over time if greater emphasis is placed on attending the
 mid-cycle meetups.
 
> Travelling to places to talk to people isn't a great solution, but it
> is the most effective one we've found so far. We should continue to
> experiment with other options, but until we find something that works
> as well as meetups, I think we need to keep having them.
> 
>> IMHO, the reasons to cut back would be:
>> 
>> - People leaving with a "well, that was useless..." feeling
>> - Not enough people able to travel to make it worthwhile
>> 
>> So far, neither of those have been outcomes of the midcycles we've
>> had,
>> so I think we're doing okay.
>> 
>> The design summits are structured differently, where we see a lot more
>> diverse attendance because of the colocation with the user summit. It
>> doesn't lend itself well to long and in-depth discussions about
>> specific
>> things, but it's very useful for what it gives us in the way of
>> exposure. We could try to have less of that at the summit and more
>> midcycle-ish time, but I think it's unlikely to achieve the same level
>> of usefulness in that environment.
>> 
>> Specifically, the lack of colocation with too many other projects has
>> been a benefit. This time, Mark and Maru where there from Neutron.
>> Last
>> time, Mark from Neutron and the other Mark from Glance were there. If
>> they were having meetups in other rooms (like at summit) they wouldn't
>> have been there exposed to discussions that didn't seem like they'd
>> have
>> a component for their participation, but did after all (re: nova and
>> glance and who should own flavors).
> 
> I agree. The ability to focus on the issues that were blocking nova
> was very important. That's hard to do at a design su

Re: [openstack-dev] [nova][core] Expectations of core reviewers

[Maru Newby]

On Aug 13, 2014, at 10:32 PM, Michael Still  wrote:

> On Thu, Aug 14, 2014 at 2:48 PM, Joe Gordon  wrote:
>> On Wed, Aug 13, 2014 at 8:31 PM, Michael Still  wrote:
>>> On Thu, Aug 14, 2014 at 1:24 PM, Jay Pipes  wrote:
>>> 
 Just wanted to quickly weigh in with my thoughts on this important
 topic. I
 very much valued the face-to-face interaction that came from the
 mid-cycle
 meetup in Beaverton (it was the only one I've ever been to).
 
 That said, I do not believe it should be a requirement that cores make
 it to
 the face-to-face meetings in-person. A number of folks have brought up
 very
 valid concerns about personal/family time, travel costs and burnout.
>>> 
>>> I'm not proposing they be a requirement. I am proposing that they be
>>> strongly encouraged.
>>> 
 I believe that the issue raised about furthering the divide between core
 and
 non-core folks is actually the biggest reason I don't support a mandate
 to
 have cores at the face-to-face meetings, and I think we should make our
 best
 efforts to support quality virtual meetings that can be done on a more
 frequent basis than the face-to-face meetings that would be optional.
>>> 
>>> I am all for online meetings, but we don't have a practical way to do
>>> them at the moment apart from IRC. Until someone has a concrete
>>> proposal that's been shown to work, I feel its a straw man argument.
>> 
>> What about making it easier for remote people to participate at the
>> mid-cycle meetups? Set up some microphones and a Google hangout?  At least
>> that way attending the mid-cycle is not all or nothing.
>> 
>> We did something like this last cycle (IIRC we didn't have enough mics) and
>> it worked pretty well.
> 
> As I said, I'm open to experimenting, but I need someone other than me
> to own that. I'm simply too busy to get to it.
> 
> However, I don't think we should throw away the thing that works for
> best for us now, until we have a working replacement. I'm very much in
> favour of work being done on a replacement though.

+1

I agree that that mid-cycles may not be sustainable over the long-term due to 
issues of travel cost (financial and otherwise) and a lack of inclusiveness, 
but I don't think they should stop happening until a suitably productive 
alternative has been found.


Maru



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Runtime checks vs Sanity checks

[Maru Newby]

Kevin Benton has proposed adding a runtime check for netns permission problems:

https://review.openstack.org/#/c/109736/

There seems to be consensus on the patch that this is something that we want to 
do at runtime, but that would seem to run counter to the precedent that 
host-specific issues such as this one be considered a deployment-time 
responsibility.  The addition of the sanity check  framework would seem to 
support the latter case:

https://github.com/openstack/neutron/blob/master/neutron/cmd/sanity_check.py

Thoughts?


Maru


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective

[Maru Newby]

On Aug 23, 2014, at 4:06 AM, Sumit Naiksatam  wrote:

> On Thu, Aug 21, 2014 at 7:28 AM, Kyle Mestery  wrote:
>> On Thu, Aug 21, 2014 at 5:12 AM, Ihar Hrachyshka  wrote:
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA512
>>> 
>>> On 20/08/14 18:28, Salvatore Orlando wrote:
 Some comments inline.
 
 Salvatore
 
 On 20 August 2014 17:38, Ihar Hrachyshka >>> > wrote:
 
 Hi all,
 
 I've read the proposal for incubator as described at [1], and I
 have several comments/concerns/suggestions to this.
 
 Overall, the idea of giving some space for experimentation that
 does not alienate parts of community from Neutron is good. In that
 way, we may relax review rules and quicken turnaround for preview
 features without loosing control on those features too much.
 
 Though the way it's to be implemented leaves several concerns, as
 follows:
 
 1. From packaging perspective, having a separate repository and
 tarballs seems not optimal. As a packager, I would better deal with
 a single tarball instead of two. Meaning, it would be better to
 keep the code in the same tree.
 
 I know that we're afraid of shipping the code for which some users
 may expect the usual level of support and stability and
 compatibility. This can be solved by making it explicit that the
 incubated code is unsupported and used on your user's risk. 1) The
 experimental code wouldn't probably be installed unless explicitly
 requested, and 2) it would be put in a separate namespace (like
 'preview', 'experimental', or 'staging', as the call it in Linux
 kernel world [2]).
 
 This would facilitate keeping commit history instead of loosing it
 during graduation.
 
 Yes, I know that people don't like to be called experimental or
 preview or incubator... And maybe neutron-labs repo sounds more
 appealing than an 'experimental' subtree in the core project.
 Well, there are lots of EXPERIMENTAL features in Linux kernel that
 we actively use (for example, btrfs is still considered
 experimental by Linux kernel devs, while being exposed as a
 supported option to RHEL7 users), so I don't see how that naming
 concern is significant.
 
 
> I think this is the whole point of the discussion around the
> incubator and the reason for which, to the best of my knowledge,
> no proposal has been accepted yet.
 
>>> 
>>> I wonder where discussion around the proposal is running. Is it public?
>>> 
>> The discussion started out privately as the incubation proposal was
>> put together, but it's now on the mailing list, in person, and in IRC
>> meetings. Lets keep the discussion going on list now.
>> 
> 
> In the spirit of keeping the discussion going, I think we probably
> need to iterate in practice on this idea a little bit before we can
> crystallize on the policy and process for this new repo. Here are few
> ideas on how we can start this iteration:
> 
> * Namespace for the new repo:
> Should this be in the neutron namespace, or a completely different
> namespace like "neutron labs"? Perhaps creating a separate namespace
> will help the packagers to avoid issues of conflicting package owners
> of the namespace.

I don’t think there is a technical requirement to choose a new namespace.  
Python supports sharing a namespace, and packaging can support this feature 
(see: oslo.*).

> 
> * Dependency on Neutron (core) repository:
> We would need to sort this out so that we can get UTs to run and pass
> in the new repo. Can we set the dependency on Neutron milestone
> releases? We already publish tar balls for the milestone releases, but
> I am not sure we publish these as packages to pypi. If not could we
> start doing that? With this in place, the incubator would always lag
> the Neutron core by at the most one milestone release.

Given that it is possible to specify a dependency as a branch/hash/tag in a git 
repo [1], I’m not sure it’s worth figuring out how to target tarballs.  Master 
branch of the incubation repo could then target the master branch of the 
Neutron repo and always be assured of being current, and then released versions 
could target milestone tags or released versions.

1: http://pip.readthedocs.org/en/latest/reference/pip_install.html#git

> 
> * Modules overlapping with the Neutron (core) repository:
> We could initially start with the features that required very little
> or no changes to the Neutron core, to avoid getting into the issue of
> blocking on changes to the Neutron (core) repository before progress
> can be made in the incubator.

+1

I agree that it would be in an incubated effort’s best interest to put off 
doing invasive changes to the Neutron tree as long as possible to ensure 
sufficient time to hash out the best approach.

> 
> * Packaging of ancillary code (CLI, Horizon, Heat):
> We start by adding th

Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective

[Maru Newby]

On Aug 20, 2014, at 6:28 PM, Salvatore Orlando  wrote:

> Some comments inline.
> 
> Salvatore
> 
> On 20 August 2014 17:38, Ihar Hrachyshka  wrote:
> 
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA512
>> 
>> Hi all,
>> 
>> I've read the proposal for incubator as described at [1], and I have
>> several comments/concerns/suggestions to this.
>> 
>> Overall, the idea of giving some space for experimentation that does
>> not alienate parts of community from Neutron is good. In that way, we
>> may relax review rules and quicken turnaround for preview features
>> without loosing control on those features too much.
>> 
>> Though the way it's to be implemented leaves several concerns, as follows:
>> 
>> 1. From packaging perspective, having a separate repository and
>> tarballs seems not optimal. As a packager, I would better deal with a
>> single tarball instead of two. Meaning, it would be better to keep the
>> code in the same tree.
>> 
>> I know that we're afraid of shipping the code for which some users may
>> expect the usual level of support and stability and compatibility.
>> This can be solved by making it explicit that the incubated code is
>> unsupported and used on your user's risk. 1) The experimental code
>> wouldn't probably be installed unless explicitly requested, and 2) it
>> would be put in a separate namespace (like 'preview', 'experimental',
>> or 'staging', as the call it in Linux kernel world [2]).
>> 
>> This would facilitate keeping commit history instead of loosing it
>> during graduation.
>> 
>> Yes, I know that people don't like to be called experimental or
>> preview or incubator... And maybe neutron-labs repo sounds more
>> appealing than an 'experimental' subtree in the core project. Well,
>> there are lots of EXPERIMENTAL features in Linux kernel that we
>> actively use (for example, btrfs is still considered experimental by
>> Linux kernel devs, while being exposed as a supported option to RHEL7
>> users), so I don't see how that naming concern is significant.
>> 
> 
> I think this is the whole point of the discussion around the incubator and
> the reason for which, to the best of my knowledge, no proposal has been
> accepted yet.
> 
>> 
>> 2. If those 'extras' are really moved into a separate repository and
>> tarballs, this will raise questions on whether packagers even want to
>> cope with it before graduation. When it comes to supporting another
>> build manifest for a piece of code of unknown quality, this is not the
>> same as just cutting part of the code into a separate
>> experimental/labs package. So unless I'm explicitly asked to package
>> the incubator, I wouldn't probably touch it myself. This is just too
>> much effort (btw the same applies to moving plugins out of the tree -
>> once it's done, distros will probably need to reconsider which plugins
>> they really want to package; at the moment, those plugins do not
>> require lots of time to ship them, but having ~20 separate build
>> manifests for each of them is just too hard to handle without clear
>> incentive).
>> 
> 
> One reason instead for moving plugins out of the main tree is allowing
> their maintainers to have full control over them.
> If there was a way with gerrit or similars to give somebody rights to merge
> code only on a subtree I probably would not even consider the option of
> moving plugin and drivers away. From my perspective it's not that I don't
> want them in the main tree, it's that I don't think it's fair for core team
> reviewers to take responsibility of approving code that they can't fully
> tests (3rd partt CI helps, but is still far from having a decent level of
> coverage).
> 
> 
>> 
>> 3. The fact that neutron-incubator is not going to maintain any stable
>> branches for security fixes and major failures concerns me too. In
>> downstream, we don't generally ship the latest and greatest from PyPI.
>> Meaning, we'll need to maintain our own downstream stable branches for
>> major fixes. [BTW we already do that for python clients.]
>> 
>> 
> This is a valid point. We need to find an appropriate trade off. My
> thinking was that incubated projects could be treated just like client
> libraries from a branch perspective.
> 
> 
> 
>> 4. Another unclear part of the proposal is that notion of keeping
>> Horizon and client changes required for incubator features in
>> neutron-incubator. AFAIK the repo will be governed by Neutron Core
>> team, and I doubt the team is ready to review Horizon changes (?). I
>> think I don't understand how we're going to handle that. Can we just
>> postpone Horizon work till graduation?
>> 
>> 
> I too do not think it's a great idea, mostly because there will be horizon
> bits not shipped with horizon, and not verified by horizon core team.
> I think it would be ok to have horizon support for neutron incubator. It
> won't be the first time that support for experimental features is added in
> horizon.
> 
> 
> 5. The wiki page says that graduation will require

Re: [openstack-dev] [neutron][cisco] Cisco Nexus requires patched ncclient

[Maru Newby]

On Aug 14, 2014, at 1:55 PM, Ihar Hrachyshka  wrote:

> Signed PGP part
> FYI: I've uploaded a review for openstack/requirements to add the
> upstream module into the list of potential dependencies [1]. Once it's
> merged, I'm going to introduce this new requirement for Neutron.

The only reason someone would install ncclient would be because they had a 
brocade or cisco solution that they wanted to integrate with and I don't think 
that justifies making Neutron depend on the library.


Maru


> [1]: https://review.openstack.org/114213
> 
> /Ihar
> 
> On 12/08/14 16:27, Ihar Hrachyshka wrote:
> > Hey all,
> >
> > as per [1], Cisco Nexus ML2 plugin requires a patched version of
> > ncclient from github. I wonder:
> >
> > - whether this information is still current; - why don't we depend
> > on ncclient thru our requirements.txt file.
> >
> > [1]: https://wiki.openstack.org/wiki/Neutron/ML2/MechCiscoNexus
> >
> > Cheers, /Ihar
> >
> > ___ OpenStack-dev
> > mailing list OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][cisco] Cisco Nexus requires patched ncclient

[Maru Newby]

On Aug 24, 2014, at 5:14 PM, Henry Gessau  wrote:

> Ihar Hrachyshka  wrote:
>> Now, maybe putting the module into requirements.txt is an overkill
>> (though I doubt it). In that case, we could be interested in getting
>> the info in some other centralized way.
> 
> Maru is of the opinion that it is overkill. I feel the same way, but I am not
> involved much with deployment issues so my feelings should not sway anyone.
> 
> Note that ncclient is not the only package used by vendor solutions that is
> not listed in requirements.txt. The ones I am aware of are:
> 
>  ncclient (cisco and brocade)
>  heleosapi (embrane)
>  a10_neutron_lbaas (a10networks)
> 
> Maybe we should start exploring "some other centralized way" to list these
> type of dependencies?

I think each plugin should be able to have its own requirements.txt file to aid 
in manual deployment and in packaging of a given plugin.  This suggests the 
need to maintain a global plugin requirements file (in the tree) to ensure use 
of only a single version of any dependencies used across more than one plugin.

Given that 3rd party CI is likely having to install these dependencies anyway, 
I think it would be good to make this deployment reproducible while avoiding 
the need to add new dependencies to core Neutron.


Maru   




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][cisco] Cisco Nexus requires patched ncclient

[Maru Newby]

On Aug 25, 2014, at 11:38 AM, Robert Collins  wrote:

> Fwiw I would like to see such dependencies listed so that we can properly 
> install trunk via automation.

Do you mean that you want the requirements for all plugins listed somewhere in 
the tree?  Or specifically in Neutron’s requirements.txt?


> On 24 Aug 2014 10:43, "Maru Newby"  wrote:
> 
> On Aug 14, 2014, at 1:55 PM, Ihar Hrachyshka  wrote:
> 
> > Signed PGP part
> > FYI: I've uploaded a review for openstack/requirements to add the
> > upstream module into the list of potential dependencies [1]. Once it's
> > merged, I'm going to introduce this new requirement for Neutron.
> 
> The only reason someone would install ncclient would be because they had a 
> brocade or cisco solution that they wanted to integrate with and I don't 
> think that justifies making Neutron depend on the library.
> 
> 
> Maru
> 
> 
> > [1]: https://review.openstack.org/114213
> >
> > /Ihar
> >
> > On 12/08/14 16:27, Ihar Hrachyshka wrote:
> > > Hey all,
> > >
> > > as per [1], Cisco Nexus ML2 plugin requires a patched version of
> > > ncclient from github. I wonder:
> > >
> > > - whether this information is still current; - why don't we depend
> > > on ncclient thru our requirements.txt file.
> > >
> > > [1]: https://wiki.openstack.org/wiki/Neutron/ML2/MechCiscoNexus
> > >
> > > Cheers, /Ihar
> > >
> > > ___ OpenStack-dev
> > > mailing list OpenStack-dev@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective

[Maru Newby]

On Aug 25, 2014, at 4:39 PM, Pradeep Kilambi (pkilambi)  
wrote:

> 
> 
> On 8/23/14, 5:36 PM, "Maru Newby"  wrote:
> 
>> 
>> On Aug 23, 2014, at 4:06 AM, Sumit Naiksatam 
>> wrote:
>> 
>>> On Thu, Aug 21, 2014 at 7:28 AM, Kyle Mestery 
>>> wrote:
>>>> On Thu, Aug 21, 2014 at 5:12 AM, Ihar Hrachyshka 
>>>> wrote:
>>>>> -BEGIN PGP SIGNED MESSAGE-
>>>>> Hash: SHA512
>>>>> 
>>>>> On 20/08/14 18:28, Salvatore Orlando wrote:
>>>>>> Some comments inline.
>>>>>> 
>>>>>> Salvatore
>>>>>> 
>>>>>> On 20 August 2014 17:38, Ihar Hrachyshka >>>>> <mailto:ihrac...@redhat.com>> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> I've read the proposal for incubator as described at [1], and I
>>>>>> have several comments/concerns/suggestions to this.
>>>>>> 
>>>>>> Overall, the idea of giving some space for experimentation that
>>>>>> does not alienate parts of community from Neutron is good. In that
>>>>>> way, we may relax review rules and quicken turnaround for preview
>>>>>> features without loosing control on those features too much.
>>>>>> 
>>>>>> Though the way it's to be implemented leaves several concerns, as
>>>>>> follows:
>>>>>> 
>>>>>> 1. From packaging perspective, having a separate repository and
>>>>>> tarballs seems not optimal. As a packager, I would better deal with
>>>>>> a single tarball instead of two. Meaning, it would be better to
>>>>>> keep the code in the same tree.
>>>>>> 
>>>>>> I know that we're afraid of shipping the code for which some users
>>>>>> may expect the usual level of support and stability and
>>>>>> compatibility. This can be solved by making it explicit that the
>>>>>> incubated code is unsupported and used on your user's risk. 1) The
>>>>>> experimental code wouldn't probably be installed unless explicitly
>>>>>> requested, and 2) it would be put in a separate namespace (like
>>>>>> 'preview', 'experimental', or 'staging', as the call it in Linux
>>>>>> kernel world [2]).
>>>>>> 
>>>>>> This would facilitate keeping commit history instead of loosing it
>>>>>> during graduation.
>>>>>> 
>>>>>> Yes, I know that people don't like to be called experimental or
>>>>>> preview or incubator... And maybe neutron-labs repo sounds more
>>>>>> appealing than an 'experimental' subtree in the core project.
>>>>>> Well, there are lots of EXPERIMENTAL features in Linux kernel that
>>>>>> we actively use (for example, btrfs is still considered
>>>>>> experimental by Linux kernel devs, while being exposed as a
>>>>>> supported option to RHEL7 users), so I don't see how that naming
>>>>>> concern is significant.
>>>>>> 
>>>>>> 
>>>>>>> I think this is the whole point of the discussion around the
>>>>>>> incubator and the reason for which, to the best of my knowledge,
>>>>>>> no proposal has been accepted yet.
>>>>>> 
>>>>> 
>>>>> I wonder where discussion around the proposal is running. Is it
>>>>> public?
>>>>> 
>>>> The discussion started out privately as the incubation proposal was
>>>> put together, but it's now on the mailing list, in person, and in IRC
>>>> meetings. Lets keep the discussion going on list now.
>>>> 
>>> 
>>> In the spirit of keeping the discussion going, I think we probably
>>> need to iterate in practice on this idea a little bit before we can
>>> crystallize on the policy and process for this new repo. Here are few
>>> ideas on how we can start this iteration:
>>> 
>>> * Namespace for the new repo:
>>> Should this be in the neutron namespace, or a completely different
>>> namespace like "neutron labs"? Perhaps creating a separate namespace
>>> will help the packagers to avoid issues of conflicting package owners
>>>

Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective

[Maru Newby]

On Aug 26, 2014, at 5:06 PM, Pradeep Kilambi (pkilambi)  
wrote:

> 
> 
> On 8/26/14, 4:49 AM, "Maru Newby"  wrote:
> 
>> 
>> On Aug 25, 2014, at 4:39 PM, Pradeep Kilambi (pkilambi)
>>  wrote:
>> 
>>> 
>>> 
>>> On 8/23/14, 5:36 PM, "Maru Newby"  wrote:
>>> 
>>>> 
>>>> On Aug 23, 2014, at 4:06 AM, Sumit Naiksatam 
>>>> wrote:
>>>> 
>>>>> On Thu, Aug 21, 2014 at 7:28 AM, Kyle Mestery 
>>>>> wrote:
>>>>>> On Thu, Aug 21, 2014 at 5:12 AM, Ihar Hrachyshka
>>>>>> 
>>>>>> wrote:
>>>>>>> -BEGIN PGP SIGNED MESSAGE-
>>>>>>> Hash: SHA512
>>>>>>> 
>>>>>>> On 20/08/14 18:28, Salvatore Orlando wrote:
>>>>>>>> Some comments inline.
>>>>>>>> 
>>>>>>>> Salvatore
>>>>>>>> 
>>>>>>>> On 20 August 2014 17:38, Ihar Hrachyshka >>>>>>> <mailto:ihrac...@redhat.com>> wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> I've read the proposal for incubator as described at [1], and I
>>>>>>>> have several comments/concerns/suggestions to this.
>>>>>>>> 
>>>>>>>> Overall, the idea of giving some space for experimentation that
>>>>>>>> does not alienate parts of community from Neutron is good. In that
>>>>>>>> way, we may relax review rules and quicken turnaround for preview
>>>>>>>> features without loosing control on those features too much.
>>>>>>>> 
>>>>>>>> Though the way it's to be implemented leaves several concerns, as
>>>>>>>> follows:
>>>>>>>> 
>>>>>>>> 1. From packaging perspective, having a separate repository and
>>>>>>>> tarballs seems not optimal. As a packager, I would better deal with
>>>>>>>> a single tarball instead of two. Meaning, it would be better to
>>>>>>>> keep the code in the same tree.
>>>>>>>> 
>>>>>>>> I know that we're afraid of shipping the code for which some users
>>>>>>>> may expect the usual level of support and stability and
>>>>>>>> compatibility. This can be solved by making it explicit that the
>>>>>>>> incubated code is unsupported and used on your user's risk. 1) The
>>>>>>>> experimental code wouldn't probably be installed unless explicitly
>>>>>>>> requested, and 2) it would be put in a separate namespace (like
>>>>>>>> 'preview', 'experimental', or 'staging', as the call it in Linux
>>>>>>>> kernel world [2]).
>>>>>>>> 
>>>>>>>> This would facilitate keeping commit history instead of loosing it
>>>>>>>> during graduation.
>>>>>>>> 
>>>>>>>> Yes, I know that people don't like to be called experimental or
>>>>>>>> preview or incubator... And maybe neutron-labs repo sounds more
>>>>>>>> appealing than an 'experimental' subtree in the core project.
>>>>>>>> Well, there are lots of EXPERIMENTAL features in Linux kernel that
>>>>>>>> we actively use (for example, btrfs is still considered
>>>>>>>> experimental by Linux kernel devs, while being exposed as a
>>>>>>>> supported option to RHEL7 users), so I don't see how that naming
>>>>>>>> concern is significant.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> I think this is the whole point of the discussion around the
>>>>>>>>> incubator and the reason for which, to the best of my knowledge,
>>>>>>>>> no proposal has been accepted yet.
>>>>>>>> 
>>>>>>> 
>>>>>>> I wonder where discussion around the proposal is running. Is it
>>>>>>> public?
>>>>>>> 
>>>>>> The discussion started out privately as the incubation proposal was
>>>>>> put togeth

Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate

[Maru Newby]

On Aug 27, 2014, at 1:47 AM, Salvatore Orlando  wrote:

> TL; DR
> A few folks are proposing to stop running tests for neutron advanced services 
> [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron 
> gate.
> 
> Reason: projects like nova are 100% orthogonal to neutron advanced services. 
> Also, there have been episodes in the past of unreliability of tests for 
> these services, and it would be good to limit affected projects considering 
> that more api tests and scenarios are being added.

Given how many rechecks I’ve had to do to merge what are effectively no-op 
patches to infra/config, most often due to the full neutron job exhibiting 
sporadic failures, I fully support this change.  I think we need time to 
stabilize the tests for advanced services against just neutron before we 
consider slowing down merges for other projects.


> 
> -
> 
> So far the neutron full job runs tests (api and scenarios) for neutron "core" 
> functionality as well as neutron "advanced services", which run as neutron 
> service plugin.
> 
> It's highly unlikely, if not impossible, that changes in projects such as 
> nova, glance or ceilometer can have an impact on the stability of these 
> services.
> On the other hand, instability in these services can trigger gate failures in 
> unrelated projects as long as tests for these services are run in the neutron 
> full job in the integrated gate. There have already been several 
> gate-breaking bugs in lbaas scenario tests are firewall api tests.
> Admittedly, advanced services do not have the same level of coverage as core 
> neutron functionality. Therefore as more tests are being added, there is an 
> increased possibility of unearthing dormant bugs.
> 
> For this reason we are proposing to not run anymore tests for neutron 
> advanced services in the integrated gate, but keep them running on the 
> neutron gate.
> This means we will have two neutron jobs:
> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron 
> functionality
> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full 
> job is today.
> 
> The former will be part of the integrated gate, the latter will be part of 
> the neutron gate.
> Considering that other integrating services should not have an impact on 
> neutron advanced services, this should not make gate testing asymmetric.
> 
> However, there might be exceptions for:
> - "orchestration" project like heat which in the future might leverage 
> capabilities like load balancing
> - oslo-* libraries, as changes in them might have an impact on neutron 
> advanced services, since they consume those libraries
> 
> Another good question is whether "extended" tests should be performed as part 
> of functional or tempest checks. My take on this is that scenario tests 
> should always be part of tempest. On the other hand I reckon API tests should 
> exclusively be part of functional tests, but as so far tempest is running a 
> gazillion of API tests, this is probably a discussion for the medium/long 
> term. 

As you say, tempest should retain responsibility for ‘golden-path’ integration 
tests involving other OpenStack services (’scenario tests’).  Everything else 
should eventually be in-tree, though the transition period to achieve this is 
likely to be multi-cycle.


m.

> 
> In order to add this new job there are a few patches under review:
> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
> [3] Are the patches implementing a blueprint which will enable us to specify 
> for which extensions test should be executed.
> 
> Finally, one more note about smoketests. Although we're planning to get rid 
> of them soon, we still have failures in the pg job because of [4]. For this 
> reasons smoketests are still running for postgres in the integrated gate. As 
> load balancing and firewall API tests are part of it, they should be removed 
> from the smoke test executed on the integrated gate ([5], [6]). This is a 
> temporary measure until the postgres issue is fixed.
> 
> Regards,
> Salvatore
> 
> [1] https://review.openstack.org/#/c/114933/
> [2] https://review.openstack.org/#/c/114932/
> [3] 
> https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
> [4] https://bugs.launchpad.net/nova/+bug/1305892
> [5] https://review.openstack.org/#/c/115022/
> [6] https://review.openstack.org/#/c/115023/
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][db] Ensuring db isolation between tests

[Maru Newby]

For legacy reasons, the Neutron test suite creates and destroys a db for each 
test.  There is a patch proposed to create the tables once and then ensure the 
tables are wiped at the end of each test [1], providing a performance 
improvement of ~10%.  I was wondering if this is the best way to provide 
isolation, since I’ve heard that isolation via per-test transactions should 
also work.  The patch author reported problems with this approach - apparently 
nested commits were not being rolled back.  Is there some trick to isolating 
with transactions that wouldn’t be immediately obvious?

Thanks,


Maru

1: https://review.openstack.org/#/c/122028/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Rename 'tenant' to 'project' as per Keystone?

[Maru Newby]

Keystone is almost finished replacing the term 'tenant' with 'project' (see 
recent thread 
https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg09709.html)  
and we might want to think about how and when Neutron makes a similar 
transition.  It's an unlikely priority in the near term given the focus on 
stability, but I wanted to broach the topic for discussion in case people think 
it might be worth attempting later in the cycle.  I've filed a preliminary 
blueprint in any case: 
https://blueprints.launchpad.net/neutron/+spec/rename-tenant-to-project 


Maru


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR

[Maru Newby]

Just a heads up, the console output for neutron gate jobs is about to get a lot 
noisier.  Any log output that contains 'ERROR' is going to be dumped into the 
console output so that we can identify and eliminate unnecessary error logging. 
 Once we've cleaned things up, the presence of unexpected (non-whitelisted) 
error output can be used to fail jobs, as per the following Tempest blueprint:

https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors

I've filed a related Neutron blueprint for eliminating the unnecessary error 
logging:

https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error

I'm looking for volunteers to help with this effort, please reply in this 
thread if you're willing to assist.

Thanks,


Maru
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR

[Maru Newby]

On Nov 28, 2013, at 1:08 AM, Salvatore Orlando  wrote:

> Thanks Maru,
> 
> This is something my team had on the backlog for a while.
> I will push some patches to contribute towards this effort in the next few 
> days.
> 
> Let me know if you're already thinking of targeting the completion of this 
> job for a specific deadline.

I'm thinking this could be a task for those not involved in fixing race 
conditions, and be done in parallel.  I guess that would be for icehouse-1 
then?  My hope would be that the early signs of race conditions would then be 
caught earlier.


m.

> 
> Salvatore
> 
> 
> On 27 November 2013 17:50, Maru Newby  wrote:
> Just a heads up, the console output for neutron gate jobs is about to get a 
> lot noisier.  Any log output that contains 'ERROR' is going to be dumped into 
> the console output so that we can identify and eliminate unnecessary error 
> logging.  Once we've cleaned things up, the presence of unexpected 
> (non-whitelisted) error output can be used to fail jobs, as per the following 
> Tempest blueprint:
> 
> https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors
> 
> I've filed a related Neutron blueprint for eliminating the unnecessary error 
> logging:
> 
> https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error
> 
> I'm looking for volunteers to help with this effort, please reply in this 
> thread if you're willing to assist.
> 
> Thanks,
> 
> 
> Maru
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR

[Maru Newby]

On Dec 2, 2013, at 2:07 AM, Anita Kuno  wrote:

> Great initiative putting this plan together, Maru. Thanks for doing
> this. Thanks for volunteering to help, Salvatore (I'm thinking of asking
> for you to be cloned - once that becomes available.) if you add your
> patch urls (as you create them) to the blueprint Maru started [0] that
> would help to track the work.
> 
> Armando, thanks for doing this work as well. Could you add the urls of
> the patches you reference to the exceptional-conditions blueprint?
> 
> For icehouse-1 to be a realistic goal for this assessment and clean-up,
> patches for this would need to be up by Tuesday Dec. 3 at the latest
> (does 13:00 UTC sound like a reasonable target?) so that they can make
> it through review and check testing, gate testing and merging prior to
> the Thursday Dec. 5 deadline for icehouse-1. I would really like to see
> this, I just want the timeline to be conscious.

My mistake, getting this done by Tuesday does not seem realistic.  icehouse-2, 
then.


m.

> 
> I would like to say talk to me tomorrow in -neutron to ensure you are
> getting the support you need to achieve this but I will be flying (wifi
> uncertain). I do hope that some additional individuals come forward to
> help with this.
> 
> Thanks Maru, Salvatore and Armando,
> Anita.
> 
> [0]
> https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error
> 
> On 11/30/2013 08:24 PM, Maru Newby wrote:
>> 
>> On Nov 28, 2013, at 1:08 AM, Salvatore Orlando  wrote:
>> 
>>> Thanks Maru,
>>> 
>>> This is something my team had on the backlog for a while.
>>> I will push some patches to contribute towards this effort in the next few 
>>> days.
>>> 
>>> Let me know if you're already thinking of targeting the completion of this 
>>> job for a specific deadline.
>> 
>> I'm thinking this could be a task for those not involved in fixing race 
>> conditions, and be done in parallel.  I guess that would be for icehouse-1 
>> then?  My hope would be that the early signs of race conditions would then 
>> be caught earlier.
>> 
>> 
>> m.
>> 
>>> 
>>> Salvatore
>>> 
>>> 
>>> On 27 November 2013 17:50, Maru Newby  wrote:
>>> Just a heads up, the console output for neutron gate jobs is about to get a 
>>> lot noisier.  Any log output that contains 'ERROR' is going to be dumped 
>>> into the console output so that we can identify and eliminate unnecessary 
>>> error logging.  Once we've cleaned things up, the presence of unexpected 
>>> (non-whitelisted) error output can be used to fail jobs, as per the 
>>> following Tempest blueprint:
>>> 
>>> https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors
>>> 
>>> I've filed a related Neutron blueprint for eliminating the unnecessary 
>>> error logging:
>>> 
>>> https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error
>>> 
>>> I'm looking for volunteers to help with this effort, please reply in this 
>>> thread if you're willing to assist.
>>> 
>>> Thanks,
>>> 
>>> 
>>> Maru
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] request-id in API response

[Maru Newby]

On Nov 30, 2013, at 1:00 AM, Sean Dague  wrote:

> On 11/29/2013 10:33 AM, Jay Pipes wrote:
>> On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
>>> Hi,
>>> 
>>> I am working on adding request-id to API response in Neutron.
>>> After I checked what header is used in other projects
>>> header name varies project by project.
>>> It seems there is no consensus what header is recommended
>>> and it is better to have some consensus.
>>> 
>>>   nova: x-compute-request-id
>>>   cinder:   x-compute-request-id
>>>   glance:   x-openstack-request-id
>>>   neutron:  x-network-request-id  (under review)
>>> 
>>> request-id is assigned and used inside of each project now,
>>> so x--request-id looks good. On the other hand,
>>> if we have a plan to enhance request-id across projects,
>>> x-openstack-request-id looks better.
>> 
>> My vote is for:
>> 
>> x-openstack-request-id
>> 
>> With an implementation of "create a request UUID if none exists yet" in
>> some standardized WSGI middleware...
> 
> Agreed. I don't think I see any value in having these have different
> service names, having just x-openstack-request-id across all the
> services seems a far better idea, and come back through and fix nova and
> cinder to be that as well.

+1 

An openstack request id should be service agnostic to allow tracking of a 
request across many services (e.g. a call to nova to boot a VM should generate 
a request id that is provided to other services in requests to provision said 
VM).  All services would ideally share a facility for generating new request 
ids and for securely accepting request ids from other services.


m.

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR

[Maru Newby]

On Dec 2, 2013, at 10:19 PM, Joe Gordon  wrote:

> 
> On Dec 2, 2013 3:39 AM, "Maru Newby"  wrote:
> >
> >
> > On Dec 2, 2013, at 2:07 AM, Anita Kuno  wrote:
> >
> > > Great initiative putting this plan together, Maru. Thanks for doing
> > > this. Thanks for volunteering to help, Salvatore (I'm thinking of asking
> > > for you to be cloned - once that becomes available.) if you add your
> > > patch urls (as you create them) to the blueprint Maru started [0] that
> > > would help to track the work.
> > >
> > > Armando, thanks for doing this work as well. Could you add the urls of
> > > the patches you reference to the exceptional-conditions blueprint?
> > >
> > > For icehouse-1 to be a realistic goal for this assessment and clean-up,
> > > patches for this would need to be up by Tuesday Dec. 3 at the latest
> > > (does 13:00 UTC sound like a reasonable target?) so that they can make
> > > it through review and check testing, gate testing and merging prior to
> > > the Thursday Dec. 5 deadline for icehouse-1. I would really like to see
> > > this, I just want the timeline to be conscious.
> >
> > My mistake, getting this done by Tuesday does not seem realistic.  
> > icehouse-2, then.
> >
> 
> With icehouse-2 being the nova-network feature freeze reevaluation point 
> (possibly lifting it) I think gating on new stacktraces by icehouse-2 is too 
> late.  Even a huge whitelist of errors is better then letting new errors in. 

No question that it needs to happen asap.  If we're talking about milestones, 
though, and icehouse-1 patches need to be in by Tuesday, I don't think 
icehouse-1 is realistic.  It will have to be early in icehouse-2.


m.

> >
> > m.
> >
> > >
> > > I would like to say talk to me tomorrow in -neutron to ensure you are
> > > getting the support you need to achieve this but I will be flying (wifi
> > > uncertain). I do hope that some additional individuals come forward to
> > > help with this.
> > >
> > > Thanks Maru, Salvatore and Armando,
> > > Anita.
> > >
> > > [0]
> > > https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error
> > >
> > > On 11/30/2013 08:24 PM, Maru Newby wrote:
> > >>
> > >> On Nov 28, 2013, at 1:08 AM, Salvatore Orlando  
> > >> wrote:
> > >>
> > >>> Thanks Maru,
> > >>>
> > >>> This is something my team had on the backlog for a while.
> > >>> I will push some patches to contribute towards this effort in the next 
> > >>> few days.
> > >>>
> > >>> Let me know if you're already thinking of targeting the completion of 
> > >>> this job for a specific deadline.
> > >>
> > >> I'm thinking this could be a task for those not involved in fixing race 
> > >> conditions, and be done in parallel.  I guess that would be for 
> > >> icehouse-1 then?  My hope would be that the early signs of race 
> > >> conditions would then be caught earlier.
> > >>
> > >>
> > >> m.
> > >>
> > >>>
> > >>> Salvatore
> > >>>
> > >>>
> > >>> On 27 November 2013 17:50, Maru Newby  wrote:
> > >>> Just a heads up, the console output for neutron gate jobs is about to 
> > >>> get a lot noisier.  Any log output that contains 'ERROR' is going to be 
> > >>> dumped into the console output so that we can identify and eliminate 
> > >>> unnecessary error logging.  Once we've cleaned things up, the presence 
> > >>> of unexpected (non-whitelisted) error output can be used to fail jobs, 
> > >>> as per the following Tempest blueprint:
> > >>>
> > >>> https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors
> > >>>
> > >>> I've filed a related Neutron blueprint for eliminating the unnecessary 
> > >>> error logging:
> > >>>
> > >>> https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error
> > >>>
> > >>> I'm looking for volunteers to help with this effort, please reply in 
> > >>> this thread if you're willing to assist.
> > >>>
> > >>> Thanks,
> > >>>
> > >>>
> > >&g

[openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

I've been investigating a bug that is preventing VM's from receiving IP 
addresses when a Neutron service is under high load:

https://bugs.launchpad.net/neutron/+bug/1192381

High load causes the DHCP agent's status updates to be delayed, causing the 
Neutron service to assume that the agent is down.  This results in the Neutron 
service not sending notifications of port addition to the DHCP agent.  At 
present, the notifications are simply dropped.  A simple fix is to send 
notifications regardless of agent status.  Does anybody have any objections to 
this stop-gap approach?  I'm not clear on the implications of sending 
notifications to agents that are down, but I'm hoping for a simple fix that can 
be backported to both havana and grizzly (yes, this bug has been with us that 
long).

Fixing this problem for real, though, will likely be more involved.  The 
proposal to replace the current wsgi framework with Pecan may increase the 
Neutron service's scalability, but should we continue to use a 'fire and 
forget' approach to notification?  Being able to track the success or failure 
of a given action outside of the logs would seem pretty important, and allow 
for more effective coordination with Nova than is currently possible.


Maru
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Tuning QueuePool parameters?

[Maru Newby]

I recently ran into this bug while trying to concurrently boot a large number 
(75) of VMs: 

https://bugs.launchpad.net/neutron/+bug/1160442

I see that the fix for the bug added configuration of SQLAlchemy QueuePool 
parameters that should prevent the boot failures I was seeing.  However, I 
don't see a good explanation on the bug as to what values to set the 
configuration to or why the defaults weren't updated to something sane.  If 
that information is available somewhere, please share!  I'm not sure why this 
bug is considered fixed if it's still possible to trigger it with no clear path 
to resolution.


Maru


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 4, 2013, at 1:47 AM, Stephen Gran  wrote:

> On 03/12/13 16:08, Maru Newby wrote:
>> I've been investigating a bug that is preventing VM's from receiving IP 
>> addresses when a Neutron service is under high load:
>> 
>> https://bugs.launchpad.net/neutron/+bug/1192381
>> 
>> High load causes the DHCP agent's status updates to be delayed, causing the 
>> Neutron service to assume that the agent is down.  This results in the 
>> Neutron service not sending notifications of port addition to the DHCP 
>> agent.  At present, the notifications are simply dropped.  A simple fix is 
>> to send notifications regardless of agent status.  Does anybody have any 
>> objections to this stop-gap approach?  I'm not clear on the implications of 
>> sending notifications to agents that are down, but I'm hoping for a simple 
>> fix that can be backported to both havana and grizzly (yes, this bug has 
>> been with us that long).
>> 
>> Fixing this problem for real, though, will likely be more involved.  The 
>> proposal to replace the current wsgi framework with Pecan may increase the 
>> Neutron service's scalability, but should we continue to use a 'fire and 
>> forget' approach to notification?  Being able to track the success or 
>> failure of a given action outside of the logs would seem pretty important, 
>> and allow for more effective coordination with Nova than is currently 
>> possible.
> 
> It strikes me that we ask an awful lot of a single neutron-server instance - 
> it has to take state updates from all the agents, it has to do scheduling, it 
> has to respond to API requests, and it has to communicate about actual 
> changes with the agents.
> 
> Maybe breaking some of these out the way nova has a scheduler and a conductor 
> and so on might be a good model (I know there are things people are unhappy 
> about with nova-scheduler, but imagine how much worse it would be if it was 
> built into the API).
> 
> Doing all of those tasks, and doing it largely single threaded, is just 
> asking for overload.

I'm sorry if it wasn't clear in my original message, but my primary concern 
lies with the reliability rather than the scalability of the Neutron service.  
Carl's addition of multiple workers is a good stop-gap to minimize the impact 
of blocking IO calls in the current architecture, and we already have consensus 
on the need to separate RPC and WSGI functions as part of the Pecan rewrite.  I 
am worried, though, that we are not being sufficiently diligent in how we 
manage state transitions through notifications.  Managing transitions and their 
associate error states is needlessly complicated by the current ad-hoc 
approach, and I'd appreciate input on the part of distributed systems experts 
as to how we could do better.


m. 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 4, 2013, at 11:02 AM, Yongsheng Gong  wrote:

> another way is to have a large agent_down_time, by default it is 9 secs.

I don't believe that increasing the timeout by itself is a good solution.  
Relying on the agent state to know whether to send a notification has simply 
proven unreliable with the current architecture of a poorly-performing single 
process server handling both RPC and WSGI.


m.

> 
> 
> On Wed, Dec 4, 2013 at 7:55 AM, Carl Baldwin  wrote:
> Stephen, all,
> 
> I agree that there may be some opportunity to split things out a bit.
> However, I'm not sure what the best way will be.  I recall that Mark
> mentioned breaking out the processes that handle API requests and RPC
> from each other at the summit.  Anyway, it is something that has been
> discussed.
> 
> I actually wanted to point out that the neutron server now has the
> ability to run a configurable number of sub-processes to handle a
> heavier load.  Introduced with this commit:
> 
> https://review.openstack.org/#/c/37131/
> 
> Set api_workers to something > 1 and restart the server.
> 
> The server can also be run on more than one physical host in
> combination with multiple child processes.
> 
> Carl
> 
> On Tue, Dec 3, 2013 at 9:47 AM, Stephen Gran
>  wrote:
> > On 03/12/13 16:08, Maru Newby wrote:
> >>
> >> I've been investigating a bug that is preventing VM's from receiving IP
> >> addresses when a Neutron service is under high load:
> >>
> >> https://bugs.launchpad.net/neutron/+bug/1192381
> >>
> >> High load causes the DHCP agent's status updates to be delayed, causing
> >> the Neutron service to assume that the agent is down.  This results in the
> >> Neutron service not sending notifications of port addition to the DHCP
> >> agent.  At present, the notifications are simply dropped.  A simple fix is
> >> to send notifications regardless of agent status.  Does anybody have any
> >> objections to this stop-gap approach?  I'm not clear on the implications of
> >> sending notifications to agents that are down, but I'm hoping for a simple
> >> fix that can be backported to both havana and grizzly (yes, this bug has
> >> been with us that long).
> >>
> >> Fixing this problem for real, though, will likely be more involved.  The
> >> proposal to replace the current wsgi framework with Pecan may increase the
> >> Neutron service's scalability, but should we continue to use a 'fire and
> >> forget' approach to notification?  Being able to track the success or
> >> failure of a given action outside of the logs would seem pretty important,
> >> and allow for more effective coordination with Nova than is currently
> >> possible.
> >
> >
> > It strikes me that we ask an awful lot of a single neutron-server instance -
> > it has to take state updates from all the agents, it has to do scheduling,
> > it has to respond to API requests, and it has to communicate about actual
> > changes with the agents.
> >
> > Maybe breaking some of these out the way nova has a scheduler and a
> > conductor and so on might be a good model (I know there are things people
> > are unhappy about with nova-scheduler, but imagine how much worse it would
> > be if it was built into the API).
> >
> > Doing all of those tasks, and doing it largely single threaded, is just
> > asking for overload.
> >
> > Cheers,
> > --
> > Stephen Gran
> > Senior Systems Integrator - theguardian.com
> > Please consider the environment before printing this email.
> > --
> > Visit theguardian.com
> > On your mobile, download the Guardian iPhone app theguardian.com/iphone and
> > our iPad edition theguardian.com/iPad   Save up to 33% by subscribing to the
> > Guardian and Observer - choose the papers you want and get full digital
> > access.
> > Visit subscribe.theguardian.com
> >
> > This e-mail and all attachments are confidential and may also
> > be privileged. If you are not the named recipient, please notify
> > the sender and delete the e-mail and all attachments immediately.
> > Do not disclose the contents to another person. You may not use
> > the information for any purpose, or store, or copy, it in any way.
> >
> > Guardian News & Media Limited is not liable for any computer
> > viruses or other material transmitted with or as part of this
> > e-mail. You should employ virus checking software.
> >
> > Guardian Ne

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 4, 2013, at 11:57 AM, Clint Byrum  wrote:

> Excerpts from Maru Newby's message of 2013-12-03 08:08:09 -0800:
>> I've been investigating a bug that is preventing VM's from receiving IP 
>> addresses when a Neutron service is under high load:
>> 
>> https://bugs.launchpad.net/neutron/+bug/1192381
>> 
>> High load causes the DHCP agent's status updates to be delayed, causing the 
>> Neutron service to assume that the agent is down.  This results in the 
>> Neutron service not sending notifications of port addition to the DHCP 
>> agent.  At present, the notifications are simply dropped.  A simple fix is 
>> to send notifications regardless of agent status.  Does anybody have any 
>> objections to this stop-gap approach?  I'm not clear on the implications of 
>> sending notifications to agents that are down, but I'm hoping for a simple 
>> fix that can be backported to both havana and grizzly (yes, this bug has 
>> been with us that long).
>> 
>> Fixing this problem for real, though, will likely be more involved.  The 
>> proposal to replace the current wsgi framework with Pecan may increase the 
>> Neutron service's scalability, but should we continue to use a 'fire and 
>> forget' approach to notification?  Being able to track the success or 
>> failure of a given action outside of the logs would seem pretty important, 
>> and allow for more effective coordination with Nova than is currently 
>> possible.
>> 
> 
> Dropping requests without triggering a user-visible error is a pretty
> serious problem. You didn't mention if you have filed a bug about that.
> If not, please do or let us know here so we can investigate and file
> a bug.

There is a bug linked to in the original message that I am already working on.  
The fact that that bug title is 'dhcp agent doesn't configure ports' rather 
than 'dhcp notifications are silently dropped' is incidental.

> 
> It seems to me that they should be put into a queue to be retried.
> Sending the notifications blindly is almost as bad as dropping them,
> as you have no idea if the agent is alive or not.

This is more the kind of discussion I was looking for.  

In the current architecture, the Neutron service handles RPC and WSGI with a 
single process and is prone to being overloaded such that agent heartbeats can 
be delayed beyond the limit for the agent being declared 'down'.  Even if we 
increased the agent timeout as Yongsheg suggests, there is no guarantee that we 
can accurately detect whether an agent is 'live' with the current architecture. 
 Given that amqp can ensure eventual delivery - it is a queue - is sending a 
notification blind such a bad idea?  In the best case the agent isn't really 
down and can process the notification.  In the worst case, the agent really is 
down but will be brought up eventually by a deployment's monitoring solution 
and process the notification when it returns.  What am I missing? 

Please consider that while a good solution will track notification delivery and 
success, we may need 2 solutions:

1. A 'good-enough', minimally-invasive stop-gap that can be back-ported to 
grizzly and havana.

2. A 'best-effort' refactor that maximizes the reliability of the DHCP agent.

I'm hoping that coming up with a solution to #1 will allow us the breathing 
room to work on #2 in this cycle.


m.



> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 4, 2013, at 8:55 AM, Carl Baldwin  wrote:

> Stephen, all,
> 
> I agree that there may be some opportunity to split things out a bit.
> However, I'm not sure what the best way will be.  I recall that Mark
> mentioned breaking out the processes that handle API requests and RPC
> from each other at the summit.  Anyway, it is something that has been
> discussed.
> 
> I actually wanted to point out that the neutron server now has the
> ability to run a configurable number of sub-processes to handle a
> heavier load.  Introduced with this commit:
> 
> https://review.openstack.org/#/c/37131/
> 
> Set api_workers to something > 1 and restart the server.
> 
> The server can also be run on more than one physical host in
> combination with multiple child processes.

I completely misunderstood the import of the commit in question.  Being able to 
run the wsgi server(s) out of process is a nice improvement, thank you for 
making it happen.  Has there been any discussion around making the default for 
api_workers > 0 (at least 1) to ensure that the default configuration separates 
wsgi and rpc load?  This also seems like a great candidate for backporting to 
havana and maybe even grizzly, although api_workers should probably be 
defaulted to 0 in those cases.

FYI, I re-ran the test that attempted to boot 75 micro VM's simultaneously with 
api_workers = 2, with mixed results.  The increased wsgi throughput resulted in 
almost half of the boot requests failing with 500 errors due to QueuePool 
errors (https://bugs.launchpad.net/neutron/+bug/1160442) in Neutron.  It also 
appears that maximizing the number of wsgi requests has the side-effect of 
increasing the RPC load on the main process, and this means that the problem of 
dhcp notifications being dropped is little improved.  I intend to submit a fix 
that ensures that notifications are sent regardless of agent status, in any 
case.


m.

> 
> Carl
> 
> On Tue, Dec 3, 2013 at 9:47 AM, Stephen Gran
>  wrote:
>> On 03/12/13 16:08, Maru Newby wrote:
>>> 
>>> I've been investigating a bug that is preventing VM's from receiving IP
>>> addresses when a Neutron service is under high load:
>>> 
>>> https://bugs.launchpad.net/neutron/+bug/1192381
>>> 
>>> High load causes the DHCP agent's status updates to be delayed, causing
>>> the Neutron service to assume that the agent is down.  This results in the
>>> Neutron service not sending notifications of port addition to the DHCP
>>> agent.  At present, the notifications are simply dropped.  A simple fix is
>>> to send notifications regardless of agent status.  Does anybody have any
>>> objections to this stop-gap approach?  I'm not clear on the implications of
>>> sending notifications to agents that are down, but I'm hoping for a simple
>>> fix that can be backported to both havana and grizzly (yes, this bug has
>>> been with us that long).
>>> 
>>> Fixing this problem for real, though, will likely be more involved.  The
>>> proposal to replace the current wsgi framework with Pecan may increase the
>>> Neutron service's scalability, but should we continue to use a 'fire and
>>> forget' approach to notification?  Being able to track the success or
>>> failure of a given action outside of the logs would seem pretty important,
>>> and allow for more effective coordination with Nova than is currently
>>> possible.
>> 
>> 
>> It strikes me that we ask an awful lot of a single neutron-server instance -
>> it has to take state updates from all the agents, it has to do scheduling,
>> it has to respond to API requests, and it has to communicate about actual
>> changes with the agents.
>> 
>> Maybe breaking some of these out the way nova has a scheduler and a
>> conductor and so on might be a good model (I know there are things people
>> are unhappy about with nova-scheduler, but imagine how much worse it would
>> be if it was built into the API).
>> 
>> Doing all of those tasks, and doing it largely single threaded, is just
>> asking for overload.
>> 
>> Cheers,
>> --
>> Stephen Gran
>> Senior Systems Integrator - theguardian.com
>> Please consider the environment before printing this email.
>> --
>> Visit theguardian.com
>> On your mobile, download the Guardian iPhone app theguardian.com/iphone and
>> our iPad edition theguardian.com/iPad   Save up to 33% by subscribing to the
>> Guardian and Observer - choose the papers you want and get full digital
>> access.
>> Visit s

Re: [openstack-dev] [Openstack][qa][Tempest][Network][Neutron] tenant_networks_reachable

[Maru Newby]

On Dec 4, 2013, at 6:34 PM, Yair Fried  wrote:

> Hi
> In tempest.conf, Is the parameter "tenant_networks_reachable" affected by 
> anything other than neutron using ip name-spaces (which is the default 
> setting) or not, and if not using it - if the tempest host is the same as the 
> neutron server host?

It is entirely possible for a tenant network to be configured to be 
reachable/routable.  The common case, though, is for a tenant network to be 
non-routable and isolated by network namespaces.  Even if tempest were to run 
on the network controller, networks isolated by network namespaces would not be 
reachable.


m. 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] request-id in API response

[Maru Newby]

On Dec 3, 2013, at 12:18 AM, Joe Gordon  wrote:

> 
> 
> 
> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson  wrote:
> Just to add to the story, Swift uses "X-Trans-Id" and generates it in the 
> outer-most "catch_errors" middleware.
> 
> Swift's catch errors middleware is responsible for ensuring that the 
> transaction id exists on each request, and that all errors previously 
> uncaught, anywhere in the pipeline, are caught and logged. If there is not a 
> common way to do this, yet, I submit it as a great template for solving this 
> problem. It's simple, scalable, and well-tested (ie tests and running in prod 
> for years).
> 
> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
> 
> Leaving aside error handling and only focusing on the transaction id (or 
> request id) generation, since OpenStack services are exposed to untrusted 
> clients, how would you propose communicating the appropriate transaction id 
> to a different service? I can see great benefit to having a glance 
> transaction ID carry through to Swift requests (and so on), but how should 
> the transaction id be communicated? It's not sensitive info, but I can 
> imagine a pretty big problem when trying to track down errors if a client 
> application decides to set eg the X-Set-Transaction-Id header on every 
> request to the same thing.
> 
> -1 to cross service request IDs, for the reasons John mentions above.
> 
> 
> Thanks for bringing this up, and I'd welcome a patch in Swift that would use 
> a common library to generate the transaction id, if it were installed. I can 
> see that there would be huge advantage to operators to trace requests through 
> multiple systems.
> 
> Another option would be for each system that calls an another OpenStack 
> system to expect and log the transaction ID for the request that was given. 
> This would be looser coupling and be more forgiving for a heterogeneous 
> cluster. Eg when Glance makes a call to Swift, Glance cloud log the 
> transaction id that Swift used (from the Swift response). Likewise, when 
> Swift makes a call to Keystone, Swift could log the Keystone transaction id. 
> This wouldn't result in a single transaction id across all systems, but it 
> would provide markers so an admin could trace the request.
> 
> There was a session on this at the summit, and although the notes are a 
> little scarce this was the conclusion we came up with.  Every time a cross 
> service call is made, we will log and send a notification for ceilometer to 
> consume, with the request-ids of both request ids.  One of the benefits of 
> this approach is that we can easily generate a tree of all the API calls that 
> are made (and clearly show when multiple calls are made to the same service), 
> something that just a cross service request id would have trouble with.

Is wise to trust anything a client provides to ensure traceability?  If a user 
receives a request id back from Nova, then submits that request id in an 
unrelated request to Neutron, the traceability would be effectively corrupted.  
If the consensus is that we don't want to securely deliver request ids for 
inter-service calls, how about requiring a service to log its request id along 
with the request id returned from a call to another service to achieve the a 
similar result?  The catch is that every call point (or client instantiation?) 
would have to be modified to pass the request id instead of just logging at one 
place in each service.  Is that a cost worth paying?


m.


> 
> https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability 
> 
> 
> With that in mind I think having a standard x-openstack-request-id makes 
> things a little more uniform, and means that adding new services doesn't 
> require new logic to handle new request ids.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 5, 2013, at 6:43 AM, Carl Baldwin  wrote:

> I have offered up https://review.openstack.org/#/c/60082/ as a
> backport to Havana.  Interest was expressed in the blueprint for doing
> this even before this thread.  If there is consensus for this as the
> stop-gap then it is there for the merging.  However, I do not want to
> discourage discussion of other stop-gap solutions like what Maru
> proposed in the original post.
> 
> Carl

Awesome.  No worries, I'm still planning on submitting a patch to improve 
notification reliability.

We seem to be cpu bound now in processing RPC messages.  Do you think it would 
be reasonable to run multiple processes for RPC?


m.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 5, 2013, at 5:21 PM, Isaku Yamahata  wrote:

> On Wed, Dec 04, 2013 at 12:37:19PM +0900,
> Maru Newby  wrote:
> 
>> In the current architecture, the Neutron service handles RPC and WSGI with a 
>> single process and is prone to being overloaded such that agent heartbeats 
>> can be delayed beyond the limit for the agent being declared 'down'.  Even 
>> if we increased the agent timeout as Yongsheg suggests, there is no 
>> guarantee that we can accurately detect whether an agent is 'live' with the 
>> current architecture.  Given that amqp can ensure eventual delivery - it is 
>> a queue - is sending a notification blind such a bad idea?  In the best case 
>> the agent isn't really down and can process the notification.  In the worst 
>> case, the agent really is down but will be brought up eventually by a 
>> deployment's monitoring solution and process the notification when it 
>> returns.  What am I missing? 
>> 
> 
> Do you mean overload of neutron server? Not neutron agent.
> So event agent sends periodic 'live' report, the reports are piled up
> unprocessed by server.
> When server sends notification, it considers agent dead wrongly.
> Not because agent didn't send live reports due to overload of agent.
> Is this understanding correct?

Your interpretation is likely correct.  The demands on the service are going to 
be much higher by virtue of having to field RPC requests from all the agents to 
interact with the database on their behalf.


>> Please consider that while a good solution will track notification delivery 
>> and success, we may need 2 solutions:
>> 
>> 1. A 'good-enough', minimally-invasive stop-gap that can be back-ported to 
>> grizzly and havana.
> 
> How about twisting DhcpAgent._periodic_resync_helper?
> If no notification is received form server from last sleep,
> it calls self.sync_state() even if self.needs_resync = False. Thus the
> inconsistency between agent and server due to losing notification
> will be fixed.

Unless I'm missing something, wouldn't forcing more and potentially unnecessary 
resyncs increase the load on the Neutron service and negatively impact 
reliability?


>> 2. A 'best-effort' refactor that maximizes the reliability of the DHCP agent.
>> 
>> I'm hoping that coming up with a solution to #1 will allow us the breathing 
>> room to work on #2 in this cycle.
> 
> Loss of notifications is somewhat inevitable, I think.
> (Or logging tasks to stable storage shared between server and agent)
> And Unconditionally sending notifications would cause problem.

Regarding sending notifications unconditionally, what specifically are you 
worried about?  I can imagine 2 scenarios:

Case 1: Send notification to an agent that is incorrectly reported as down. 
Result:  Agent receives notification and acts on it.

Case 2: Send notification to an agent that is actually down.
Result: Agent comes up eventually (in a production environment this should be a 
given) and calls sync_state().  We definitely need to make sync_state more 
reliable, though (I discuss the specifics later in this message).

Notifications could of course be dropped if AMQP queues are not persistent and 
are lost, but I don't think there needs to be a code-based remedy for this.  An 
operator is likely to deploy the AMQP service in HA to prevent the queues from 
being lost, and know to restart everything in the event of catastrophic failure.

That's not to say we don't have work to do, though.  An agent is responsible 
for communicating resource state changes to the service, but the service 
neither detects nor reacts when the state of a resource is scheduled to change 
and fails to do so in a reasonable timeframe.  Thus, as in the bug that 
prompted this discussion, it is up to the user to detect the failure (a VM 
without connectivity).  Ideally, Neutron should be tracking resource state 
changes with sufficient detail and reviewing them periodically to allow timely 
failure detection and remediation.  However, such a change is unlikely to be a 
candidate for backport so it will have to wait.


> 
> You mentioned agent crash. Server crash should also be taken care of
> for reliability. Admin also sometimes wants to restart neutron
> server/agents for some reasons.
> Agent can crash after receiving notifications before start processing
> actual tasks. Server can crash after commiting changes to DB before sending
> notifications. In such cases, notification will be lost.
> Polling to resync would be necessary somewhere.

Agreed, we need to consider the cases of both agent and service failure.  

In the case of service failure, thanks to recently merged patches, the dhcp 
agent will at least force a

Re: [openstack-dev] request-id in API response

[Maru Newby]

On Dec 6, 2013, at 1:09 AM, John Dickinson  wrote:

> 
> On Dec 5, 2013, at 1:36 AM, Maru Newby  wrote:
> 
>> 
>> On Dec 3, 2013, at 12:18 AM, Joe Gordon  wrote:
>> 
>>> 
>>> 
>>> 
>>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson  wrote:
>>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in the 
>>> outer-most "catch_errors" middleware.
>>> 
>>> Swift's catch errors middleware is responsible for ensuring that the 
>>> transaction id exists on each request, and that all errors previously 
>>> uncaught, anywhere in the pipeline, are caught and logged. If there is not 
>>> a common way to do this, yet, I submit it as a great template for solving 
>>> this problem. It's simple, scalable, and well-tested (ie tests and running 
>>> in prod for years).
>>> 
>>> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
>>> 
>>> Leaving aside error handling and only focusing on the transaction id (or 
>>> request id) generation, since OpenStack services are exposed to untrusted 
>>> clients, how would you propose communicating the appropriate transaction id 
>>> to a different service? I can see great benefit to having a glance 
>>> transaction ID carry through to Swift requests (and so on), but how should 
>>> the transaction id be communicated? It's not sensitive info, but I can 
>>> imagine a pretty big problem when trying to track down errors if a client 
>>> application decides to set eg the X-Set-Transaction-Id header on every 
>>> request to the same thing.
>>> 
>>> -1 to cross service request IDs, for the reasons John mentions above.
>>> 
>>> 
>>> Thanks for bringing this up, and I'd welcome a patch in Swift that would 
>>> use a common library to generate the transaction id, if it were installed. 
>>> I can see that there would be huge advantage to operators to trace requests 
>>> through multiple systems.
>>> 
>>> Another option would be for each system that calls an another OpenStack 
>>> system to expect and log the transaction ID for the request that was given. 
>>> This would be looser coupling and be more forgiving for a heterogeneous 
>>> cluster. Eg when Glance makes a call to Swift, Glance cloud log the 
>>> transaction id that Swift used (from the Swift response). Likewise, when 
>>> Swift makes a call to Keystone, Swift could log the Keystone transaction 
>>> id. This wouldn't result in a single transaction id across all systems, but 
>>> it would provide markers so an admin could trace the request.
>>> 
>>> There was a session on this at the summit, and although the notes are a 
>>> little scarce this was the conclusion we came up with.  Every time a cross 
>>> service call is made, we will log and send a notification for ceilometer to 
>>> consume, with the request-ids of both request ids.  One of the benefits of 
>>> this approach is that we can easily generate a tree of all the API calls 
>>> that are made (and clearly show when multiple calls are made to the same 
>>> service), something that just a cross service request id would have trouble 
>>> with.
>> 
>> Is wise to trust anything a client provides to ensure traceability?  If a 
>> user receives a request id back from Nova, then submits that request id in 
>> an unrelated request to Neutron, the traceability would be effectively 
>> corrupted.  If the consensus is that we don't want to securely deliver 
>> request ids for inter-service calls, how about requiring a service to log 
>> its request id along with the request id returned from a call to another 
>> service to achieve the a similar result?
> 
> Yes, this is what I was proposing. I think this is the best path forward.

Ok, great.  And as per your suggestion, a middleware-based error handler will 
soon be proposed for oslo that will secondarily ensure that a request id is 
added to the request.  

> 
> 
>> The catch is that every call point (or client instantiation?) would have to 
>> be modified to pass the request id instead of just logging at one place in 
>> each service.  Is that a cost worth paying?
> 
> Perhaps this is my ignorance of how other projects work today, but does this 
> not already happen? Is it possible to get a response from an API call to an 
> OpenStack project that doesn't include a request id?

We'll have it in Neutron real-soon-now, and then I think the answer will be 
'yes'.

On reflection, it should be easy enough for a given service to ensure that 
calls to other services are automatically instrumented to log request id pairs. 
 Again, probably something for oslo.


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 7, 2013, at 6:21 PM, Robert Collins  wrote:

> On 7 December 2013 21:53, Isaku Yamahata  wrote:
>> 
>> Case 3: Hardware failure. So an agent on the node is gone.
>>Another agent will run on another node.
>> 
>> If AMQP service is set up not to lose notification, notifications will be 
>> piled up
>> and stress AMQP service. I would say single node failure isn't catastrophic.
> 
> So we should have AMQP set to discard notifications if there is noone

What are the semantics of AMQP discarding notifications when a consumer is no 
longer present?  Can this be relied upon to ensure that potentially stale 
notifications do not remain in the queue when an agent restarts?


> listening: when an agent connects after an outage, it first starts
> listening, then does a poll for updates it missed.

Are you suggesting that processing of notifications and full state 
synchronization are able to cooperate safely?  Or hoping that it will be so in 
the future?


m.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][neutron] Post-mortem debugging for tests?

[Maru Newby]

Are any other projects interested in adding back the post-mortem debugging 
support we lost in the move away from nose?  I have a patch in review for 
neutron and salv-orlando asked whether oslo might be the better place for it.

https://review.openstack.org/#/c/43776/


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 10, 2013, at 4:47 PM, Isaku Yamahata  wrote:

> On Tue, Dec 10, 2013 at 07:28:10PM +1300,
> Robert Collins  wrote:
> 
>> On 10 December 2013 19:16, Isaku Yamahata  wrote:
>> 
>>> Answering myself. If connection is closed, it will reconnects automatically
>>> at rpc layer. See neutron.openstack.common.rpc.impl_{kombu, qpid}.py.
>>> So notifications during reconnects can be lost if AMQP service is set
>>> to discard notifications during no subscriber.
>> 
>> Which is fine: the agent repulls the full set it's running on that
>> machine, and life goes on.
> 
> On what event?
> Polling in agent seems effectively disabled by self.needs_resync with
> the current code.

If the agent is not connected, it is either down (needs_resync will be set to 
True on launch) or experiencing a loss of connectivity to the amqp service 
(needs_resync will have been set to True on error).  The loss of notifications 
is not a problem in either case.


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 5, 2013, at 4:43 PM, Édouard Thuleau  wrote:

> There also another bug you can link/duplicate with #1192381 is
> https://bugs.launchpad.net/neutron/+bug/1185916.
> I proposed a fix but it's not the good way. I abandoned it.
> 
> Édouard.

Thank you for pointing this out!


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 10, 2013, at 6:36 PM, Isaku Yamahata  wrote:

> On Mon, Dec 09, 2013 at 08:43:59AM +1300,
> Robert Collins  wrote:
> 
 listening: when an agent connects after an outage, it first starts
 listening, then does a poll for updates it missed.
>>> 
>>> Are you suggesting that processing of notifications and full state 
>>> synchronization are able to cooperate safely?  Or hoping that it will be so 
>>> in the future?
>> 
>> I'm saying that you can avoid race conditions by a combination of
>> 'subscribe to changes' + 'give me the full state'.
> 
> Like this?
> https://review.openstack.org/#/c/61057/
> This patch is just to confirm the idea.

I'm afraid the proposed patch is no more reliable than the current approach of 
using file-based locking.   I am working on an alternate patch that puts the 
rpc event loop in the dhcp agent so that better coordination between full 
synchronization and notification handling is possible.  This approach has 
already been taken with the L3 agent and work on the L2 agent is in process.  


m.

> -- 
> Isaku Yamahata 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 11, 2013, at 8:39 AM, Isaku Yamahata  wrote:

> On Wed, Dec 11, 2013 at 01:23:36AM +0900,
> Maru Newby  wrote:
> 
>> 
>> On Dec 10, 2013, at 6:36 PM, Isaku Yamahata  wrote:
>> 
>>> On Mon, Dec 09, 2013 at 08:43:59AM +1300,
>>> Robert Collins  wrote:
>>> 
>>>>>> listening: when an agent connects after an outage, it first starts
>>>>>> listening, then does a poll for updates it missed.
>>>>> 
>>>>> Are you suggesting that processing of notifications and full state 
>>>>> synchronization are able to cooperate safely?  Or hoping that it will be 
>>>>> so in the future?
>>>> 
>>>> I'm saying that you can avoid race conditions by a combination of
>>>> 'subscribe to changes' + 'give me the full state'.
>>> 
>>> Like this?
>>> https://review.openstack.org/#/c/61057/
>>> This patch is just to confirm the idea.
>> 
>> I'm afraid the proposed patch is no more reliable than the current approach 
>> of using file-based locking.   I am working on an alternate patch that puts 
>> the rpc event loop in the dhcp agent so that better coordination between 
>> full synchronization and notification handling is possible.  This approach 
>> has already been taken with the L3 agent and work on the L2 agent is in 
>> process.  
> 
> You objected against agent polling in the discussion.
> But you're now proposing polling now. Did you change your mind?

Uh, no.  I'm proposing better coordination between notification processing and 
full state synchronization beyond simple exclusionary primitives  
(utils.synchronize etc).  I apologize if my language was unclear.  


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Cores - Prioritize merging migration fixes after tox change merges

[Maru Newby]

As per Anita's email, we're not to approve anything until the following tox fix 
merges:  https://review.openstack.org/#/c/60825

Please keep an eye on the change, and once it merges, make sure that the 
following patches merge before regular approval rules resume:

https://review.openstack.org/#/c/61677 
https://review.openstack.org/#/c/61663

Without these migration patches, devstack is broken for neutron.

Thanks!


Maru


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] DHCP Agent Reliability

[Maru Newby]

On Dec 13, 2013, at 8:06 PM, Isaku Yamahata  wrote:

> On Fri, Dec 06, 2013 at 04:30:17PM +0900,
> Maru Newby  wrote:
> 
>> 
>> On Dec 5, 2013, at 5:21 PM, Isaku Yamahata  wrote:
>> 
>>> On Wed, Dec 04, 2013 at 12:37:19PM +0900,
>>> Maru Newby  wrote:
>>> 
>>>> In the current architecture, the Neutron service handles RPC and WSGI with 
>>>> a single process and is prone to being overloaded such that agent 
>>>> heartbeats can be delayed beyond the limit for the agent being declared 
>>>> 'down'.  Even if we increased the agent timeout as Yongsheg suggests, 
>>>> there is no guarantee that we can accurately detect whether an agent is 
>>>> 'live' with the current architecture.  Given that amqp can ensure eventual 
>>>> delivery - it is a queue - is sending a notification blind such a bad 
>>>> idea?  In the best case the agent isn't really down and can process the 
>>>> notification.  In the worst case, the agent really is down but will be 
>>>> brought up eventually by a deployment's monitoring solution and process 
>>>> the notification when it returns.  What am I missing? 
>>>> 
>>> 
>>> Do you mean overload of neutron server? Not neutron agent.
>>> So event agent sends periodic 'live' report, the reports are piled up
>>> unprocessed by server.
>>> When server sends notification, it considers agent dead wrongly.
>>> Not because agent didn't send live reports due to overload of agent.
>>> Is this understanding correct?
>> 
>> Your interpretation is likely correct.  The demands on the service are going 
>> to be much higher by virtue of having to field RPC requests from all the 
>> agents to interact with the database on their behalf.
> 
> Is this strongly indicating thread-starvation. i.e. too much unfair
> thread scheduling.
> Given that eventlet is cooperative threading, should sleep(0) to 
> hogging thread?

I'm afraid that's a question for a profiler: 
https://github.com/colinhowe/eventlet_profiler


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][qa] The 'spec' parameter of mock.patch()

[Maru Newby]

I recently saw a case [1] where a misspelled assertion method 
(asoptt_called_once_with vs assert_called_once_with) did not result in a test 
failure because the object it was called on was created by mock.patch() without 
any of the spec/spec_set/autospec parameters being set.  Might it make sense to 
require that calls to mock.patch() set autospec=True [2]?


m.

1: 
https://review.openstack.org/#/c/61105/7/neutron/tests/unit/openvswitch/test_ovs_lib.py
 (line 162)
2: http://www.voidspace.org.uk/python/mock/patch.html#mock.patch


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tempest][Production] Tempest / the gate / real world load

[Maru Newby]

I'm afraid I missed this topic the first time around, and I think it bears 
revisiting.

tl;dr: I think we should consider ensuring gate stability in the face of 
resource-starved services by some combination of more intelligent test design 
and better handling of resource starvation (for example, rate-limiting).  
Stress-testing would be more effective if it were explicitly focused on 
real-world usage scenarios and run separately from the gate.  I think 
stress-testing is about the 'when' of failure, whereas the gate is about 'if'.

I don't think it can be argued that OpenStack services (especially Neutron) can 
do better to ensure reliability under load.  Running things in parallel in the 
gate shone a bright light on many problem areas and that was inarguably a good 
thing.  Now that we have a better sense of the problem, though, it may be time 
to think about evolving our approach.

>From the perspective of gating commits, I think it makes sense to (a) minimize 
>gate execution time and (b) provide some guarantees of reliability under 
>reasonable load.  I don't think either of these requires continuing to 
>evaluate unrealistic usage scenarios against services running in a severely 
>resource-starved environment.  Every service eventually falls over when too 
>much is asked of it.  These kinds of failure are not likely to be particularly 
>deterministic, so wouldn't it make sense to avoid triggering them in the gate 
>as much as possible?

In the specific case of Neutron, the current approach to testing isolation 
involves creating and tearing down networks at a tremendous rate.  I'm not sure 
anyone can argue that this constitutes a usage scenario that is likely to 
appear in production, but because it causes problems in the gate, we've had to 
prioritize working on it over initiatives that might prove more useful to 
operators.  While this may have been a necessary stop on the road to Neutron 
stability, I think it may be worth considering whether we want the gate to 
continue having an outsized role in defining optimization priorities.  

Thoughts?


m.

On Dec 12, 2013, at 11:23 AM, Robert Collins  wrote:

> A few times now we've run into patches for devstack-gate / devstack
> that change default configuration to handle 'tempest load'.
> 
> For instance - https://review.openstack.org/61137 (Sorry Salvatore I'm
> not picking on you really!)
> 
> So there appears to be a meme that the gate is particularly stressful
> - a bad environment - and that real world situations have less load.
> 
> This could happen a few ways: (a) deployers might separate out
> components more; (b) they might have faster machines; (c) they might
> have less concurrent activity.
> 
> (a) - unlikely! Deployers will cram stuff together as much as they can
> to save overheads. Big clouds will have components split out - yes,
> but they will also have correspondingly more load to drive that split
> out.
> 
> (b) Perhaps, but not orders of magnitude faster, the clouds we run on
> are running on fairly recent hardware, and by using big instances we
> don't get crammed it with that many other tenants.
> 
> (c) Almost certainly not. Tempest currently does a maximum of four
> concurrent requests. A small business cloud could easily have 5 or 6
> people making concurrent requests from time to time, and bigger but
> not huge clouds will certainly have that. Their /average/ rate of API
> requests may be much lower, but when they point service orchestration
> tools at it -- particularly tools that walk their dependencies in
> parallel - load is going to be much much higher than what we generate
> with Tempest.
> 
> tl;dr : if we need to change a config file setting in devstack-gate or
> devstack *other than* setting up the specific scenario, think thrice -
> should it be a production default and set in the relevant projects
> default config setting.
> 
> Cheers,
> Rob
> -- 
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][neutron] API tests in the Neutron tree

[Maru Newby]

At the last 2 summits, I've suggested that API tests could be maintained in the 
Neutron tree and reused by Tempest.  I've finally submitted some patches that 
demonstrate this concept:

https://review.openstack.org/#/c/72585/  (implements a unit test for the 
lifecycle of the network resource)
https://review.openstack.org/#/c/72588/  (runs the test with tempest rest 
clients)

My hope is to make API test maintenance a responsibility of the Neutron team.  
The API compatibility of each Neutron plugin has to be validated by Neutron 
tests anyway, and if the tests are structured as I am proposing, Tempest can 
reuse those efforts rather than duplicating them.

I've added this topic to this week's agenda, and I would really appreciate it 
interested parties would take a look at the patches in question to prepare 
themselves to participate in the discussion.


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree

[Maru Newby]

On Feb 12, 2014, at 12:36 PM, Sean Dague  wrote:

> On 02/12/2014 01:48 PM, Maru Newby wrote:
>> At the last 2 summits, I've suggested that API tests could be maintained in 
>> the Neutron tree and reused by Tempest.  I've finally submitted some patches 
>> that demonstrate this concept:
>> 
>> https://review.openstack.org/#/c/72585/  (implements a unit test for the 
>> lifecycle of the network resource)
>> https://review.openstack.org/#/c/72588/  (runs the test with tempest rest 
>> clients)
>> 
>> My hope is to make API test maintenance a responsibility of the Neutron 
>> team.  The API compatibility of each Neutron plugin has to be validated by 
>> Neutron tests anyway, and if the tests are structured as I am proposing, 
>> Tempest can reuse those efforts rather than duplicating them.
>> 
>> I've added this topic to this week's agenda, and I would really appreciate 
>> it interested parties would take a look at the patches in question to 
>> prepare themselves to participate in the discussion.
>> 
>> 
>> m.
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> Realistically, having API tests duplicated in the Tempest tree is a
> feature, not a bug.
> 
> tempest/api is there for double book keep accounting, and it has been
> really effective at preventing accidental breakage of our APIs (which
> used to happen all the time), so I don't think putting API testing in
> neutron obviates that.

Given how limited our testing resources are, might it be worth considering 
whether 'double-entry accounting' is actually the best way to preventing 
accidental breakage going forward?  Might reasonable alternatives exist, such 
as clearly separating api tests from other tests in the neutron tree and giving 
review oversight only to qualified individuals?


> 
> Today most projects (excepting swift... which I'll get to in a second)
> think about testing in 2 ways. Unit tests driven by tox in a venv, and
> tempest tests in a devstack environment.
> 
> Because of this dualism, people have put increasingly more awkward live
> environments in the tox unit tests jobs. For instance, IIRC, neutron
> actually starts a full wsgi stack to tests every single in tree plugin,
> instead of just testing the driver call down path.

Yeah, and this full-stack approach means that neutron tests are incredibly hard 
to maintain and debug, to say nothing of the time penalty (approaching the 
duration of a tempest smoke run).  Part of the benefit of the proposal in 
question is to allow targeting the plugins directly with the same tests that 
will run against the full stack.


m.

> 
> Swift did something a little different. They have 3 classes of things.
> Unit tests, Tempest Tests, and Swift Functional tests. The Swift
> functional tests run in a devstack, but not with Tempest. Instead they
> run their own suite.
> 
> This test suite only runs on the swift project, not on other projects,
> and it's something they can use to test functional scenarios that
> wouldn't fit in tempest, and extend to their heart's content, not having
> to worry about the interaction with other components, because it's only
> pushing on swift.
> 
> Going down this third path with neutron testing is I think very
> valuable. Honestly, I'd like to encourage more projects to do this as
> well. It would give them a greater component assuredness before entering
> the integrated gate.
> 
>   -Sean
> 
> -- 
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree

[Maru Newby]

On Feb 12, 2014, at 1:59 PM, Sean Dague  wrote:

> On 02/12/2014 04:25 PM, Maru Newby wrote:
>> 
>> On Feb 12, 2014, at 12:36 PM, Sean Dague  wrote:
>> 
>>> On 02/12/2014 01:48 PM, Maru Newby wrote:
>>>> At the last 2 summits, I've suggested that API tests could be maintained 
>>>> in the Neutron tree and reused by Tempest.  I've finally submitted some 
>>>> patches that demonstrate this concept:
>>>> 
>>>> https://review.openstack.org/#/c/72585/  (implements a unit test for the 
>>>> lifecycle of the network resource)
>>>> https://review.openstack.org/#/c/72588/  (runs the test with tempest rest 
>>>> clients)
>>>> 
>>>> My hope is to make API test maintenance a responsibility of the Neutron 
>>>> team.  The API compatibility of each Neutron plugin has to be validated by 
>>>> Neutron tests anyway, and if the tests are structured as I am proposing, 
>>>> Tempest can reuse those efforts rather than duplicating them.
>>>> 
>>>> I've added this topic to this week's agenda, and I would really appreciate 
>>>> it interested parties would take a look at the patches in question to 
>>>> prepare themselves to participate in the discussion.
>>>> 
>>>> 
>>>> m.
>>>> ___
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev@lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> 
>>> 
>>> Realistically, having API tests duplicated in the Tempest tree is a
>>> feature, not a bug.
>>> 
>>> tempest/api is there for double book keep accounting, and it has been
>>> really effective at preventing accidental breakage of our APIs (which
>>> used to happen all the time), so I don't think putting API testing in
>>> neutron obviates that.
>> 
>> Given how limited our testing resources are, might it be worth considering 
>> whether 'double-entry accounting' is actually the best way to preventing 
>> accidental breakage going forward?  Might reasonable alternatives exist, 
>> such as clearly separating api tests from other tests in the neutron tree 
>> and giving review oversight only to qualified individuals?
> 
> Our direct experience is that if we don't do this, within 2 weeks some
> project will have landed API breaking changes. This approach actually
> takes a lot of review load off the core reviewers, so reverting to a
> model which puts more work back on the review team (given the current
> review load), isn't something I think we want.

Just so I'm clear, is there anything I could say that would change your mind?


m.


> 
> I get that there is a cost with this. But there is a cost of all of
> this. And because API tests should be write once for the API (and
> specifically not evolving), the upfront cost vs. the continually paid
> cost of review time in tree has been the better trade off.
> 
>   -Sean
> 
> -- 
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][nova] Neutron plugin authors: Does port status indicate liveness?

[Maru Newby]

Booting a Nova instance when Neutron is enabled is often unreliable due to the 
lack of coordination between Nova and Neutron apart from port allocation.  
Aaron Rosen and I have been talking about fixing this by having Nova perform a 
check for port 'liveness' after vif plug and before vm boot.  The idea is to 
have Nova fail the instance if its ports are not seen to be 'live' within a 
reasonable timeframe after plug.  Our initial thought is that the compute node 
would call Nova's networking subsystem which could query Neutron for the status 
of the instance's ports.

The open question is whether the port 'status' field can be relied upon to 
become ACTIVE for all the plugins currently in the tree.  If this is not the 
case, please reply to this thread with an indication of how one would be able 
to tell the 'liveness' of a port managed by the plugin you maintain.

In the event that one or more plugins cannot reliably indicate port liveness, 
we'll need to ensure that the port liveness check can be optionally disabled so 
that the existing behavior of racing vm boot is maintained for plugins that 
need it.

Thanks in advance,


Maru


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Core/Vendor code decomposition

[Maru Newby]

On Dec 7, 2014, at 10:51 AM, Gary Kotton  wrote:

> Hi Kyle,
> I am not missing the point. I understand the proposal. I just think that it 
> has some shortcomings (unless I misunderstand, which will certainly not be 
> the first time and most definitely not the last). The thinning out is to have 
> a shim in place. I understand this and this will be the entry point for the 
> plugin. I do not have a concern for this. My concern is that we are not doing 
> this with the ML2 off the bat. That should lead by example as it is our 
> reference architecture. Lets not kid anyone, but we are going  to hit some 
> problems with the decomposition. I would prefer that it be done with the 
> default implementation. Why?

The proposal is to move vendor-specific logic out of the tree to increase 
vendor control over such code while decreasing load on reviewers.  ML2 doesn’t 
contain vendor-specific logic - that’s the province of ML2 drivers - so it is 
not a good target for the proposed decomposition by itself.


>   • Cause we will fix them quicker as it is something that prevent 
> Neutron from moving forwards
>   • We will just need to fix in one place first and not in N (where N is 
> the vendor plugins)
>   • This is a community effort – so we will have a lot more eyes on it
>   • It will provide a reference architecture for all new plugins that 
> want to be added to the tree
>   • It will provide a working example for plugin that are already in tree 
> and are to be replaced by the shim
> If we really want to do this, we can say freeze all development (which is 
> just approvals for patches) for a few days so that we will can just focus on 
> this. I stated what I think should be the process on the review. For those 
> who do not feel like finding the link:
>   • Create a stack forge project for ML2
>   • Create the shim in Neutron
>   • Update devstack for the to use the two repos and the shim
> When #3 is up and running we switch for that to be the gate. Then we start a 
> stopwatch on all other plugins.

As was pointed out on the spec (see Miguel’s comment on r15), the ML2 plugin 
and the OVS mechanism driver need to remain in the main Neutron repo for now.  
Neutron gates on ML2+OVS and landing a breaking change in the Neutron repo 
along with its corresponding fix to a separate ML2 repo would be all but 
impossible under the current integrated gating scheme.  Plugins/drivers that do 
not gate Neutron have no such constraint.


Maru


> Sure, I’ll catch you on IRC tomorrow. I guess that you guys will bash out the 
> details at the meetup. Sadly I will not be able to attend – so you will have 
> to delay on the tar and feathers.
> Thanks
> Gary
> 
> 
> From: "mest...@mestery.com" 
> Reply-To: OpenStack List 
> Date: Sunday, December 7, 2014 at 7:19 PM
> To: OpenStack List 
> Cc: "openst...@lists.openstack.org" 
> Subject: Re: [openstack-dev] [Neutron] Core/Vendor code decomposition
> 
> Gary, you are still miss the point of this proposal. Please see my comments 
> in review. We are not forcing things out of tree, we are thinning them. The 
> text you quoted in the review makes that clear. We will look at further 
> decomposing ML2 post Kilo, but we have to be realistic with what we can 
> accomplish during Kilo.
> 
> Find me on IRC Monday morning and we can discuss further if you still have 
> questions and concerns.
> 
> Thanks!
> Kyle
> 
> On Sun, Dec 7, 2014 at 2:08 AM, Gary Kotton  wrote:
>> Hi,
>> I have raised my concerns on the proposal. I think that all plugins should 
>> be treated on an equal footing. My main concern is having the ML2 plugin in 
>> tree whilst the others will be moved out of tree will be problematic. I 
>> think that the model will be complete if the ML2 was also out of tree. This 
>> will help crystalize the idea and make sure that the model works correctly.
>> Thanks
>> Gary
>> 
>> From: "Armando M." 
>> Reply-To: OpenStack List 
>> Date: Saturday, December 6, 2014 at 1:04 AM
>> To: OpenStack List , 
>> "openst...@lists.openstack.org" 
>> Subject: [openstack-dev] [Neutron] Core/Vendor code decomposition
>> 
>> Hi folks,
>> 
>> For a few weeks now the Neutron team has worked tirelessly on [1].
>> 
>> This initiative stems from the fact that as the project matures, evolution 
>> of processes and contribution guidelines need to evolve with it. This is to 
>> ensure that the project can keep on thriving in order to meet the needs of 
>> an ever growing community.
>> 
>> The effort of documenting intentions, and fleshing out the various details 
>> of the proposal is about to reach an end, and we'll soon kick the tires to 
>> put the proposal into practice. Since the spec has grown pretty big, I'll 
>> try to capture the tl;dr below.
>> 
>> If you have any comment please do not hesitate to raise them here and/or 
>> reach out to us.
>> 
>> tl;dr >>>
>> 
>> From the Kilo release, we'll initiate a set of steps to change the following 
>> areas:
>>  • 

Re: [openstack-dev] [Nova][Neutron] out-of-tree plugin for Mech driver/L2 and vif_driver

[Maru Newby]

On Dec 9, 2014, at 7:04 AM, Daniel P. Berrange  wrote:

> On Tue, Dec 09, 2014 at 10:53:19AM +0100, Maxime Leroy wrote:
>> I have also proposed a blueprint to have a new plugin mechanism in
>> nova to load external vif driver. (nova-specs:
>> https://review.openstack.org/#/c/136827/ and nova (rfc patch):
>> https://review.openstack.org/#/c/136857/)
>> 
>> From my point-of-view of a developer having a plugin framework for
>> internal/external vif driver seems to be a good thing.
>> It makes the code more modular and introduce a clear api for vif driver 
>> classes.
>> 
>> So far, it raises legitimate questions concerning API stability and
>> public API that request a wider discussion on the ML (as asking by
>> John Garbut).
>> 
>> I think having a plugin mechanism and a clear api for vif driver is
>> not going against this policy:
>> http://docs.openstack.org/developer/nova/devref/policies.html#out-of-tree-support.
>> 
>> There is no needs to have a stable API. It is up to the owner of the
>> external VIF driver to ensure that its driver is supported by the
>> latest API. And not the nova community to manage a stable API for this
>> external VIF driver. Does it make senses ?
> 
> Experiance has shown that even if it is documented as unsupported, once
> the extension point exists, vendors & users will ignore the small print
> about support status. There will be complaints raised every time it gets
> broken until we end up being forced to maintain it as stable API whether
> we want to or not. That's not a route we want to go down.

Is the support contract for a given API have to be binary - ‘supported’ vs 
‘unsupported’?  The stability requirements for REST API’s that end users and 
all kinds of tooling consume are arguably different from those of an internal 
API, and recognizing this difference could be useful.


> 
>> Considering the network V2 API, L2/ML2 mechanism driver and VIF driver
>> need to exchange information such as: binding:vif_type and
>> binding:vif_details.
>> 
>> From my understanding, 'binding:vif_type' and 'binding:vif_details' as
>> a field part of the public network api. There is no validation
>> constraints for these fields (see
>> http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html),
>> it means that any value is accepted by the API. So, the values set in
>> 'binding:vif_type' and 'binding:vif_details' are not part of the
>> public API. Is my understanding correct ?
> 
> The VIF parameters are mapped into the nova.network.model.VIF class
> which is doing some crude validation. I would anticipate that this
> validation will be increasing over time, because any functional data
> flowing over the API and so needs to be carefully managed for upgrade
> reasons.
> 
> Even if the Neutron impl is out of tree, I would still expect both
> Nova and Neutron core to sign off on any new VIF type name and its
> associated details (if any).
> 
>> What other reasons am I missing to not have VIF driver classes as a
>> public extension point ?
> 
> Having to find & install VIF driver classes from countless different
> vendors, each hiding their code away in their own obsecure website,
> will lead to awful end user experiance when deploying Nova. Users are
> better served by having it all provided when they deploy Nova IMHO

Shipping drivers in-tree makes sense for a purely open source solution for the 
reasons you mention.  The logic doesn’t necessarily extend to deployment of a 
proprietary solution, though.  If a given OpenStack deployment is intended to 
integrate with such a solution, it is likely that the distro/operator/deployer 
will have a direct relationship with the solution provider and the required 
software (including VIF driver(s), if necessary) is likely to have a 
well-defined distribution channel.


> 
> If every vendor goes off & works in their own isolated world we also
> loose the scope to align the implementations, so that common concepts
> work the same way in all cases and allow us to minimize the number of
> new VIF types required. The proposed vhostuser VIF type is a good
> example of this - it allows a single Nova VIF driver to be capable of
> potentially supporting multiple different impls on the Neutron side.
> If every vendor worked in their own world, we would have ended up with
> multiple VIF drivers doing the same thing in Nova, each with their own
> set of bugs & quirks.

I’m not sure the suggestion is that every vendor go off and do their own thing. 
 Rather, the option for out-of-tree drivers could be made available to those 
that are pursuing initiatives that aren’t found to be in-keeping with Nova’s 
current priorities.  I believe that allowing out-of-tree extensions is 
essential to ensuring the long-term viability of OpenStack.  There is only so 
much experimental work that is going to be acceptable in core OpenStack 
projects, if only to ensure stability.  Yes, there is the potential for 
duplicative effort with results of varying qu

Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group

[Maru Newby]

On Dec 11, 2014, at 2:27 PM, Sean Dague  wrote:

> On 12/11/2014 04:16 PM, Jay Pipes wrote:
>> On 12/11/2014 04:07 PM, Vishvananda Ishaya wrote:
>>> On Dec 11, 2014, at 1:04 PM, Jay Pipes  wrote:
 On 12/11/2014 04:01 PM, Vishvananda Ishaya wrote:
> 
> On Dec 11, 2014, at 8:00 AM, Henry Gessau  wrote:
> 
>> On Thu, Dec 11, 2014, Mark McClain  wrote:
>>> 
 On Dec 11, 2014, at 8:43 AM, Jay Pipes >>> > wrote:
 
 I'm generally in favor of making name attributes opaque, utf-8
 strings that
 are entirely user-defined and have no constraints on them. I
 consider the
 name to be just a tag that the user places on some resource. It
 is the
 resource's ID that is unique.
 
 I do realize that Nova takes a different approach to *some*
 resources,
 including the security group name.
 
 End of the day, it's probably just a personal preference whether
 names
 should be unique to a tenant/user or not.
 
 Maru had asked me my opinion on whether names should be unique and I
 answered my personal opinion that no, they should not be, and if
 Neutron
 needed to ensure that there was one and only one default security
 group for
 a tenant, that a way to accomplish such a thing in a race-free
 way, without
 use of SELECT FOR UPDATE, was to use the approach I put into the
 pastebin on
 the review above.
 
>>> 
>>> I agree with Jay.  We should not care about how a user names the
>>> resource.
>>> There other ways to prevent this race and Jay’s suggestion is a
>>> good one.
>> 
>> However we should open a bug against Horizon because the user
>> experience there
>> is terrible with duplicate security group names.
> 
> The reason security group names are unique is that the ec2 api
> supports source
> rule specifications by tenant_id (user_id in amazon) and name, so
> not enforcing
> uniqueness means that invocation in the ec2 api will either fail or be
> non-deterministic in some way.
 
 So we should couple our API evolution to EC2 API then?
 
 -jay
>>> 
>>> No I was just pointing out the historical reason for uniqueness, and
>>> hopefully
>>> encouraging someone to find the best behavior for the ec2 api if we
>>> are going
>>> to keep the incompatibility there. Also I personally feel the ux is
>>> better
>>> with unique names, but it is only a slight preference.
>> 
>> Sorry for snapping, you made a fair point.
> 
> Yeh, honestly, I agree with Vish. I do feel that the UX of that
> constraint is useful. Otherwise you get into having to show people UUIDs
> in a lot more places. While those are good for consistency, they are
> kind of terrible to show to people.

While there is a good case for the UX of unique names - it also makes 
orchestration via tools like puppet a heck of a lot simpler - the fact is that 
most OpenStack resources do not require unique names.  That being the case, why 
would we want security groups to deviate from this convention?


Maru



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group

[Maru Newby]

On Dec 12, 2014, at 1:40 PM, Sean Dague  wrote:

> On 12/12/2014 01:05 PM, Maru Newby wrote:
>> 
>> On Dec 11, 2014, at 2:27 PM, Sean Dague  wrote:
>> 
>>> On 12/11/2014 04:16 PM, Jay Pipes wrote:
>>>> On 12/11/2014 04:07 PM, Vishvananda Ishaya wrote:
>>>>> On Dec 11, 2014, at 1:04 PM, Jay Pipes  wrote:
>>>>>> On 12/11/2014 04:01 PM, Vishvananda Ishaya wrote:
>>>>>>> 
>>>>>>> On Dec 11, 2014, at 8:00 AM, Henry Gessau  wrote:
>>>>>>> 
>>>>>>>> On Thu, Dec 11, 2014, Mark McClain  wrote:
>>>>>>>>> 
>>>>>>>>>> On Dec 11, 2014, at 8:43 AM, Jay Pipes >>>>>>>>> <mailto:jaypi...@gmail.com>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I'm generally in favor of making name attributes opaque, utf-8
>>>>>>>>>> strings that
>>>>>>>>>> are entirely user-defined and have no constraints on them. I
>>>>>>>>>> consider the
>>>>>>>>>> name to be just a tag that the user places on some resource. It
>>>>>>>>>> is the
>>>>>>>>>> resource's ID that is unique.
>>>>>>>>>> 
>>>>>>>>>> I do realize that Nova takes a different approach to *some*
>>>>>>>>>> resources,
>>>>>>>>>> including the security group name.
>>>>>>>>>> 
>>>>>>>>>> End of the day, it's probably just a personal preference whether
>>>>>>>>>> names
>>>>>>>>>> should be unique to a tenant/user or not.
>>>>>>>>>> 
>>>>>>>>>> Maru had asked me my opinion on whether names should be unique and I
>>>>>>>>>> answered my personal opinion that no, they should not be, and if
>>>>>>>>>> Neutron
>>>>>>>>>> needed to ensure that there was one and only one default security
>>>>>>>>>> group for
>>>>>>>>>> a tenant, that a way to accomplish such a thing in a race-free
>>>>>>>>>> way, without
>>>>>>>>>> use of SELECT FOR UPDATE, was to use the approach I put into the
>>>>>>>>>> pastebin on
>>>>>>>>>> the review above.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I agree with Jay.  We should not care about how a user names the
>>>>>>>>> resource.
>>>>>>>>> There other ways to prevent this race and Jay’s suggestion is a
>>>>>>>>> good one.
>>>>>>>> 
>>>>>>>> However we should open a bug against Horizon because the user
>>>>>>>> experience there
>>>>>>>> is terrible with duplicate security group names.
>>>>>>> 
>>>>>>> The reason security group names are unique is that the ec2 api
>>>>>>> supports source
>>>>>>> rule specifications by tenant_id (user_id in amazon) and name, so
>>>>>>> not enforcing
>>>>>>> uniqueness means that invocation in the ec2 api will either fail or be
>>>>>>> non-deterministic in some way.
>>>>>> 
>>>>>> So we should couple our API evolution to EC2 API then?
>>>>>> 
>>>>>> -jay
>>>>> 
>>>>> No I was just pointing out the historical reason for uniqueness, and
>>>>> hopefully
>>>>> encouraging someone to find the best behavior for the ec2 api if we
>>>>> are going
>>>>> to keep the incompatibility there. Also I personally feel the ux is
>>>>> better
>>>>> with unique names, but it is only a slight preference.
>>>> 
>>>> Sorry for snapping, you made a fair point.
>>> 
>>> Yeh, honestly, I agree with Vish. I do feel that the UX of that
>>> constraint is useful. Otherwise you get into having to show people UUIDs
>>> in a lot more places. While those are good for consistency, they are
>>> kind of terrible to show to people.
>> 
>> While there is a good case for the UX of unique names - it also makes 
>> orchestration via tools like puppet a heck of a lot simpler - the fact is 
>> that most OpenStack resources do not require unique names.  That being the 
>> case, why would we want security groups to deviate from this convention?
> 
> Maybe the other ones are the broken ones?
> 
> Honestly, any sanely usable system makes names unique inside a
> container. Like files in a directory. In this case the tenant is the
> container, which makes sense.
> 
> It is one of many places that OpenStack is not consistent. But I'd
> rather make things consistent and more usable than consistent and less.

You might prefer less consistency for the sake of usability, but for me, 
consistency is a large enough factor in usability that allowing seemingly 
arbitrary deviation doesn’t seem like a huge win.  Regardless, I’d like to see 
us came to decisions on API usability on an OpenStack-wide basis, so the API 
working group is probably where this discussion should continue.


Maru

  


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group

[Maru Newby]

On Dec 15, 2014, at 9:13 AM, Assaf Muller  wrote:

> 
> 
> - Original Message -
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA512
>> 
>> I was (rightfully) asked to share my comments on the matter that I
>> left in gerrit here. See below.
>> 
>> On 12/12/14 22:40, Sean Dague wrote:
>>> On 12/12/2014 01:05 PM, Maru Newby wrote:
>>>> 
>>>> On Dec 11, 2014, at 2:27 PM, Sean Dague  wrote:
>>>> 
>>>>> On 12/11/2014 04:16 PM, Jay Pipes wrote:
>>>>>> On 12/11/2014 04:07 PM, Vishvananda Ishaya wrote:
>>>>>>> On Dec 11, 2014, at 1:04 PM, Jay Pipes 
>>>>>>> wrote:
>>>>>>>> On 12/11/2014 04:01 PM, Vishvananda Ishaya wrote:
>>>>>>>>> 
>>>>>>>>> On Dec 11, 2014, at 8:00 AM, Henry Gessau
>>>>>>>>>  wrote:
>>>>>>>>> 
>>>>>>>>>> On Thu, Dec 11, 2014, Mark McClain 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Dec 11, 2014, at 8:43 AM, Jay Pipes
>>>>>>>>>>>> mailto:jaypi...@gmail.com>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm generally in favor of making name attributes
>>>>>>>>>>>> opaque, utf-8 strings that are entirely
>>>>>>>>>>>> user-defined and have no constraints on them. I
>>>>>>>>>>>> consider the name to be just a tag that the user
>>>>>>>>>>>> places on some resource. It is the resource's ID
>>>>>>>>>>>> that is unique.
>>>>>>>>>>>> 
>>>>>>>>>>>> I do realize that Nova takes a different approach
>>>>>>>>>>>> to *some* resources, including the security group
>>>>>>>>>>>> name.
>>>>>>>>>>>> 
>>>>>>>>>>>> End of the day, it's probably just a personal
>>>>>>>>>>>> preference whether names should be unique to a
>>>>>>>>>>>> tenant/user or not.
>>>>>>>>>>>> 
>>>>>>>>>>>> Maru had asked me my opinion on whether names
>>>>>>>>>>>> should be unique and I answered my personal
>>>>>>>>>>>> opinion that no, they should not be, and if
>>>>>>>>>>>> Neutron needed to ensure that there was one and
>>>>>>>>>>>> only one default security group for a tenant,
>>>>>>>>>>>> that a way to accomplish such a thing in a
>>>>>>>>>>>> race-free way, without use of SELECT FOR UPDATE,
>>>>>>>>>>>> was to use the approach I put into the pastebin
>>>>>>>>>>>> on the review above.
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I agree with Jay.  We should not care about how a
>>>>>>>>>>> user names the resource. There other ways to
>>>>>>>>>>> prevent this race and Jay’s suggestion is a good
>>>>>>>>>>> one.
>>>>>>>>>> 
>>>>>>>>>> However we should open a bug against Horizon because
>>>>>>>>>> the user experience there is terrible with duplicate
>>>>>>>>>> security group names.
>>>>>>>>> 
>>>>>>>>> The reason security group names are unique is that the
>>>>>>>>> ec2 api supports source rule specifications by
>>>>>>>>> tenant_id (user_id in amazon) and name, so not
>>>>>>>>> enforcing uniqueness means that invocation in the ec2
>>>>>>>>> api will either fail or be non-deterministic in some
>>>>>>>>> way.
>>>>>>>> 
>>>>>>>> So we should couple our API evolution to EC2 API then?
>>>>>>>> 
>>>>>>>> -jay
>>>>>>> 
>&

Re: [openstack-dev] The state of nova-network to neutron migration

[Maru Newby]

As per a recent exchange on #openstack-neutron, I’ve been asked to present my 
views on this effort.  What follows is in no way intended to detract from the 
hard work and dedication of those undertaking it, but I think that their energy 
could be better spent.

At nova’s juno mid-cycle in July, there was a discussion about deprecating 
nova-network.  Most of the work-items on the TC’s gap analysis [1] had been 
covered off, with one notable exception: Gap 6, the requirement to provide a 
migration plan between nova-network and neutron, had stalled over questions of 
implementation strategy.

In my recollection of the conversation that followed, broad consensus was 
reached that the costs of automating a reliable and fault-tolerant migration 
strategy would be  considerable.  The technical complexity of targeting a fixed 
deployment scenario would be challenging enough, and targeting heterogenous 
scenarios would magnify that complexity many-fold.  Given the cost and high 
risks associated with implementing an automated solution, everyone seemed to 
agree that it was not worth pursuing.  Our understanding was that not pursuing 
an automated solution could still be in keeping with the TC’s requirements for 
deprecation, which required that a migration plan be formulated but not that it 
be automated.  Documentation was deemed sufficient, and that was to be the path 
forward in covering Gap 6.  The documentation would allow deployers and 
operators to devise migration strategies to suit their individual requirements.

Then, when the Kilo summit schedule was announced, there was a session 
scheduled in the nova track for discussing how to implement an automated 
migration.  I only managed to catch the tail end of the session, but the 
etherpad [2] makes no mention of the rationale for requiring an automated 
migration in the first place.  It was like the discussion at the mid-cycle, and 
all the talk of the risks outweighing the potential benefits of such an effort, 
had simply not occurred.

So, in the interests of a full and open discussion on this matter, can someone 
please explain to me why the risks discussed at the mid-cycle were suddenly 
deemed justifiable, seemingly against all technical rationale?  Criticism has 
been leveled at the neutron project for our supposed inaction in implementing 
an automated solution, and I don’t think I’m the only one who is concerned that 
this is an unreasonable requirement imposed without due consideration to the 
risks involved.  Yes, most of us want to see nova-network deprecated, but why 
is the lack of migration automation blocking that?  An automated migration was 
not a requirement in the TC’s original assessment of the preconditions for 
deprecation.  I think that if neutron is deemed to be of sufficiently high 
quality that it can serve as an effective replacement for nova-network, and we 
can document a migration plan between them, then deprecation should proceed.


Maru


1: 
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee/Neutron_Gap_Coverage
2: https://etherpad.openstack.org/p/kilo-nova-nova-network-to-neutron

> On Dec 19, 2014, at 8:59 AM, Anita Kuno  wrote:
> 
> Rather than waste your time making excuses let me state where we are and
> where I would like to get to, also sharing my thoughts about how you can
> get involved if you want to see this happen as badly as I have been told
> you do.
> 
> Where we are:
>* a great deal of foundation work has been accomplished to achieve
> parity with nova-network and neutron to the extent that those involved
> are ready for migration plans to be formulated and be put in place
>* a summit session happened with notes and intentions[0]
>* people took responsibility and promptly got swamped with other
> responsibilities
>* spec deadlines arose and in neutron's case have passed
>* currently a neutron spec [1] is a work in progress (and it needs
> significant work still) and a nova spec is required and doesn't have a
> first draft or a champion
> 
> Where I would like to get to:
>* I need people in addition to Oleg Bondarev to be available to help
> come up with ideas and words to describe them to create the specs in a
> very short amount of time (Oleg is doing great work and is a fabulous
> person, yay Oleg, he just can't do this alone)
>* specifically I need a contact on the nova side of this complex
> problem, similar to Oleg on the neutron side
>* we need to have a way for people involved with this effort to find
> each other, talk to each other and track progress
>* we need to have representation at both nova and neutron weekly
> meetings to communicate status and needs
> 
> We are at K-2 and our current status is insufficient to expect this work
> will be accomplished by the end of K-3. I will be championing this work,
> in whatever state, so at least it doesn't fall off the map. If you would
> like to help this effort please get in contact. I will be thinking of
>

Re: [openstack-dev] The state of nova-network to neutron migration

[Maru Newby]

> On Jan 8, 2015, at 3:54 PM, Sean Dague  wrote:
> 
> On 01/08/2015 06:41 PM, Maru Newby wrote:
>> As per a recent exchange on #openstack-neutron, I’ve been asked to present 
>> my views on this effort.  What follows is in no way intended to detract from 
>> the hard work and dedication of those undertaking it, but I think that their 
>> energy could be better spent.
>> 
>> At nova’s juno mid-cycle in July, there was a discussion about deprecating 
>> nova-network.  Most of the work-items on the TC’s gap analysis [1] had been 
>> covered off, with one notable exception: Gap 6, the requirement to provide a 
>> migration plan between nova-network and neutron, had stalled over questions 
>> of implementation strategy.
>> 
>> In my recollection of the conversation that followed, broad consensus was 
>> reached that the costs of automating a reliable and fault-tolerant migration 
>> strategy would be  considerable.  The technical complexity of targeting a 
>> fixed deployment scenario would be challenging enough, and targeting 
>> heterogenous scenarios would magnify that complexity many-fold.  Given the 
>> cost and high risks associated with implementing an automated solution, 
>> everyone seemed to agree that it was not worth pursuing.  Our understanding 
>> was that not pursuing an automated solution could still be in keeping with 
>> the TC’s requirements for deprecation, which required that a migration plan 
>> be formulated but not that it be automated.  Documentation was deemed 
>> sufficient, and that was to be the path forward in covering Gap 6.  The 
>> documentation would allow deployers and operators to devise migration 
>> strategies to suit their individual requirements.
>> 
>> Then, when the Kilo summit schedule was announced, there was a session 
>> scheduled in the nova track for discussing how to implement an automated 
>> migration.  I only managed to catch the tail end of the session, but the 
>> etherpad [2] makes no mention of the rationale for requiring an automated 
>> migration in the first place.  It was like the discussion at the mid-cycle, 
>> and all the talk of the risks outweighing the potential benefits of such an 
>> effort, had simply not occurred.
>> 
>> So, in the interests of a full and open discussion on this matter, can 
>> someone please explain to me why the risks discussed at the mid-cycle were 
>> suddenly deemed justifiable, seemingly against all technical rationale?  
>> Criticism has been leveled at the neutron project for our supposed inaction 
>> in implementing an automated solution, and I don’t think I’m the only one 
>> who is concerned that this is an unreasonable requirement imposed without 
>> due consideration to the risks involved.  Yes, most of us want to see 
>> nova-network deprecated, but why is the lack of migration automation 
>> blocking that?  An automated migration was not a requirement in the TC’s 
>> original assessment of the preconditions for deprecation.  I think that if 
>> neutron is deemed to be of sufficiently high quality that it can serve as an 
>> effective replacement for nova-network, and we can document a migration plan 
>> between them, then deprecation should proceed.
>> 
>> 
>> Maru
> 
> The crux of it comes from the fact that the operator voice (especially
> those folks with large nova-network deploys) wasn't represented there.
> Once we got back from the mid-cycle and brought it to the list, there
> was some very understandable push back on deprecating without a
> migration plan.

I think it’s clear that a migration plan is required.  An automated migration, 
not so much.

> 
> I believe we landed at the need for the common case, flat multi host
> networking, to be migrated to something equivalent in neutron land
> (dvr?). And it needs to be something that Metacloud and CERN can get
> behind, as they represent 2 very large nova-network deploys (and have
> reasonably well defined down time allowances for this).
> 
> This doesn't have to be automation for all cases, but we need to support
> a happy path from one to the other that's repeatable, reasonably
> automatic (as much as possible), and provides minimum down time for
> guests running on the environment.

The fact that operators running nova-network would like the upstream community 
to pay for implementing an automated migration solution for them is hardly 
surprising.  It is less clear to me that implementing such a solution, with all 
the attendant cost and risks, should take priority over efforts that benefit a 
broader swath of the community.  Are the operators in question so strapped for 
resources that they are not able to automate their migrations themselves, 
provided a sufficiently detailed plan to do so?


Maru  





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Changes to the core team

[Maru Newby]

+1 to Doug’s addition.  He’s been a workhorse in moving lbaas and the services 
split forward and his being able to finally merge code in the main repo should 
be a boon to our merge velocity.

Many thanks to Sumit for his long-serving dedication as a core reviewer.  I 
hope to see him return to the fold soon!


Maru

> On Jan 15, 2015, at 2:31 PM, Kyle Mestery  wrote:
> 
> The last time we looked at core reviewer stats was in December [1]. In 
> looking at the current stats, I'm going to propose some changes to the core 
> team. Reviews are the most important part of being a core reviewer, so we 
> need to ensure cores are doing reviews. The stats for the 90 day period [2] 
> indicate some changes are needed for core reviewers who are no longer 
> reviewing on pace with the other core reviewers.
> 
> First of all, I'm removing Sumit Naiksatam from neutron-core. Sumit has been 
> a core reviewer for a long time, and his past contributions are very much 
> thanked by the entire OpenStack Neutron team. If Sumit jumps back in with 
> thoughtful reviews in the future, we can look at getting him back as a 
> Neutron core reviewer. But for now, his stats indicate he's not reviewing at 
> a level consistent with the rest of the Neutron core reviewers.
> 
> As part of the change, I'd like to propose Doug Wiegley as a new Neutron core 
> reviewer. Doug has been actively reviewing code across not only all the 
> Neutron projects, but also other projects such as infra. His help and work in 
> the services split in December were the reason we were so successful in 
> making that happen. Doug has also been instrumental in the Neutron LBaaS V2 
> rollout, as well as helping to merge code in the other neutron service 
> repositories.
> 
> I'd also like to take this time to remind everyone that reviewing code is a 
> responsibility, in Neutron the same as other projects. And core reviewers are 
> especially beholden to this responsibility. I'd also like to point out that 
> +1/-1 reviews are very useful, and I encourage everyone to continue reviewing 
> code even if you are not a core reviewer.
> 
> Existing neutron cores, please vote +1/-1 for the addition of Doug to the 
> core team.
> 
> Thanks!
> Kyle
> 
> [1] 
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/051986.html
> [2] http://russellbryant.net/openstack-stats/neutron-reviewers-90.txt
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][QA] Partial fix for unittest memory consumption

[Maru Newby]

Memory usage due to plugin+mock leakage is addressed by the following patch:

https://review.openstack.org/#/c/92793/

I'm seeing residual (post-test) memory usage decrease from ~4.5gb to ~1.3gb 
with 12 concurrent test runners.  Of the 1.3gb, sqlalchemy is taking the lion 
share at ~500mb, so that will be my next target.


Maru
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Request to consider merging OVS ARP responder

[Maru Newby]

The patch in question is doing an invasive runtime check [1] to determine 
whether arp header modification is supported by ovs.  I seem to remember a lack 
of consensus on whether invasive runtime checks were acceptable around vxlan 
detection, and I would appreciate confirmation from other cores that this 
approach is considered acceptable for this patch.


m.

1: https://review.openstack.org/#/c/49227/41/neutron/agent/linux/ovs_lib.py

On May 19, 2014, at 1:07 PM, Assaf Muller  wrote:

> Hello Mark, Bob and Kyle,
> 
> You've all been involved with the OVS ARP responder patch:
> https://review.openstack.org/#/c/49227/
> 
> I kindly request that you revisit this patch - It's ready from
> my point of view, if that means anything. The last few patchsets
> have been nitpicks. Looking at the history, the patch has already
> had a large amount of +1's and +2's through it's various stages.
> 
> I think it'd be technically superior if this patch was merged
> before further progress is made in the DVR patch series.
> 
> 
> Thanks for your time,
> Assaf Muller, Cloud Networking Engineer
> Red Hat


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Proposed changes to core team

[Maru Newby]

On May 21, 2014, at 1:59 PM, Kyle Mestery  wrote:

> Neutron cores, please vote +1/-1 for the proposed addition of Carl
> Baldwin to Neutron core.

+1 from me

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?

[Maru Newby]

At the summit session last week for group-based policy, there were many 
concerns voiced about the approach being undertaken.  I think those concerns 
deserve a wider audience, and I'm going to highlight some of them here.

The primary concern seemed to be related to the complexity of the approach 
implemented for the POC.  A number of session participants voiced concern that 
the simpler approach documented in the original proposal [1] (described in the 
section titled 'Policies applied between groups') had not been implemented in 
addition to or instead of what appeared in the POC (described in the section 
titled 'Policies applied as a group API').  The simpler approach was considered 
by those participants as having the advantage of clarity and immediate 
usefulness, whereas the complex approach was deemed hard to understand and 
without immediate utility.

A secondary but no less important concern is related to the impact on Neutron 
of the approach implemented in the POC.  The POC was developed monolithically, 
without oversight through gerrit, and the resulting patches were excessive in 
size (~4700 [2] and ~1500 [3] lines).  Such large patches are effectively 
impossible to review.  Even broken down into reviewable chunks, though, it does 
not seem realistic to target juno-1 for merging this kind of complexity.  The 
impact on stability could be considerable, and it is questionable whether the 
necessary review effort should be devoted to fast-tracking group-based policy 
at all, let alone an approach that is considered by many to be unnecessarily 
complicated.  

The blueprint for group policy [4] is currently listed as a 'High' priority.  
With the above concerns in mind, does it make sense to continue prioritizing an 
effort that at present would seem to require considerably more resources than 
the benefit it appears to promise?


Maru

1: https://etherpad.openstack.org/p/group-based-policy
2: https://review.openstack.org/93853
3: https://review.openstack.org/93935
4: https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?

[Maru Newby]

On May 22, 2014, at 11:03 AM, Maru Newby  wrote:

> At the summit session last week for group-based policy, there were many 
> concerns voiced about the approach being undertaken.  I think those concerns 
> deserve a wider audience, and I'm going to highlight some of them here.
> 
> The primary concern seemed to be related to the complexity of the approach 
> implemented for the POC.  A number of session participants voiced concern 
> that the simpler approach documented in the original proposal [1] (described 
> in the section titled 'Policies applied between groups') had not been 
> implemented in addition to or instead of what appeared in the POC (described 
> in the section titled 'Policies applied as a group API').  The simpler 
> approach was considered by those participants as having the advantage of 
> clarity and immediate usefulness, whereas the complex approach was deemed 
> hard to understand and without immediate utility.
> 
> A secondary but no less important concern is related to the impact on Neutron 
> of the approach implemented in the POC.  The POC was developed 
> monolithically, without oversight through gerrit, and the resulting patches 
> were excessive in size (~4700 [2] and ~1500 [3] lines).  Such large patches 
> are effectively impossible to review.  Even broken down into reviewable 
> chunks, though, it does not seem realistic to target juno-1 for merging this 
> kind of complexity.  The impact on stability could be considerable, and it is 
> questionable whether the necessary review effort should be devoted to 
> fast-tracking group-based policy at all, let alone an approach that is 
> considered by many to be unnecessarily complicated.  
> 
> The blueprint for group policy [4] is currently listed as a 'High' priority.  
> With the above concerns in mind, does it make sense to continue prioritizing 
> an effort that at present would seem to require considerably more resources 
> than the benefit it appears to promise?
> 
> 
> Maru
> 
> 1: https://etherpad.openstack.org/p/group-based-policy

Apologies, this link is to the summit session etherpad.  The link to the 
original proposal is:

https://docs.google.com/document/d/1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit

> 2: https://review.openstack.org/93853
> 3: https://review.openstack.org/93935
> 4: 
> https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?

[Maru Newby]

priorities of resources, then 
I have a responsibility to voice that concern and push back.  You're free to 
implement whatever you want outside of the tree, but participating in the 
Neutron community means accepting the norms that we have adopted.  If you don't 
like them, you are free to work to effect change, but it won't happen by simply 
wishing it so.

And as per the summit session, there appears to be significant user demand for 
a subset of the POC that represents the 'link-based' approach.  If we are to 
talk about taking user interests into account, I think the starting point would 
be implementing what they are asking for first and and evolving more complex 
solutions from there.


m.


> 3. Re: Could dev/review cycles be better spent on refactoring
> I think that most people agree that policy control is an important feature 
> that fundamentally improves neutron (by solving the automation and scale 
> issues). In a large project, multiple sub-projects can, and for a healthy 
> project should, work in parallel. I understand that the neutron core team is 
> stretched. But we still need to be able to balance the needs of today (paying 
> off the technical debt/existing-issues by doing refactoring) with needs of 
> tomorrow (new features like GP and LBaaS). GP effort was started in Havana, 
> and now we are trying to get this in Juno. I think that is reasonable and a 
> long enough cycle for a "high priority" project to be able to get some core 
> attention. Again I refer to LBaaS experience, as they struggled with very 
> similar issues.
> 
> 4. Re: If refactored neutron was available, would a simpler option become 
> more viable
> We would love to be able to answer that question. We have been trying to 
> understand the refactoring work to understand this (see another ML thread) 
> and we are open to understanding your position on that. We will call the 
> ad-hoc meeting that you suggested and we would like to understand the 
> refactoring work that might be reused for simpler policy implementation. At 
> the same time, we would like to build on what is available today, and when 
> the required refactored neutron becomes available (say Juno or K-release), we 
> are more than happy to adapt to it at that time. Serializing all development 
> around an effort that is still in inception phase is not a good solution. We 
> are looking forward to participating in the core refactoring work, and based 
> on the final spec that come up with, we would love to be able to eventually 
> make the policy implementation simpler.
> 
> Regards,
> Mandeep
> 
> 
> 
> 
> On Thu, May 22, 2014 at 11:44 AM, Armando M.  wrote:
> I would second Maru's concerns, and I would also like to add the following:
> 
> We need to acknowledge the fact that there are certain architectural
> aspects of Neutron as a project that need to be addressed; at the
> summit we talked about the core refactoring, a task oriented API, etc.
> To me these items have been neglected far too much over the past and
> would need a higher priority and a lot more attention during the Juno
> cycle. Being stretched as we are I wonder if dev/review cycles
> wouldn't be better spent devoting more time to these efforts rather
> than GP.
> 
> That said, I appreciate that GP is important and needs to move
> forward, but at the same time I am thinking that there must be a
> better way for addressing it and yet relieve some of the pressure that
> GP complexity imposes to the Neutron team. One aspect it was discussed
> at the summit was that the type of approach shown in [2] and [3]
> below, was chosen because of lack of proper integration hooks...so I
> am advocating: let's talk about those first before ruling them out in
> favor of a monolithic approach that seems to violate some engineering
> principles, like modularity and loose decoupling of system components.
> 
> I think we didn't have enough time during the summit to iron out some
> of the concerns voiced here, and it seems like the IRC meeting for
> Group Policy would not be the right venue to try and establish a
> common ground among the people driving this effort and the rest of the
> core team.
> 
> Shall we try and have an ad-hoc meeting and an ad-hoc agenda to find a
> consensus?
> 
> Many thanks,
> Armando
> 
> On 22 May 2014 11:38, Maru Newby  wrote:
>> 
>> On May 22, 2014, at 11:03 AM, Maru Newby  wrote:
>> 
>>> At the summit session last week for group-based policy, there were many 
>>> concerns voiced about the approach being undertaken.  I think those 
>>> concerns deserve a wider audience, and I'm going to highlight some of them 
>>> here.
>>> 
>>> The primary

Re: [openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?

[Maru Newby]

On May 22, 2014, at 4:35 PM, Mandeep Dhami  wrote:

> > each patch needs to receive core reviewer attention and that subsequent 
> > patches incorporate their feedback.
> 
> At least two core neutron members were involved in creating the PoC, and at 
> least two more cores were involved in reviews at various times. In addition 
> to them, senior developers from at least seven networking companies were 
> involved in developing this code. I concede that this code was on github for 
> a few weeks, as that made the prototyping faster and allowed us to "fail 
> faster", but it was open and reviewed with the team above (and with the cores 
> in that team). Based on our learning from that prototype activity, and 
> feedback of those cores, we are upstreaming the improved "production code" to 
> gerrit. All that involvement from the neutron core reviewers was critical in 
> keeping the larger PoC team above focused on neutron norms and expectations 
> from design and code.

The feedback from reviewers needs to be provided on openstack infrastructure 
rather than outside it so that it is both visible to all reviewers (not just 
those directly involved) and that an enduring history of the process is 
retained.  These requirements were not met in working in github on the POC, 
regardless of your protestations of how 'open' that work was and of who was 
involved.  This isn't to suggest that out-of-tree prototyping isn't useful - of 
course it is.  But I think it important to recognize that out-of-tree 
development is unlikely to be an effective way to develop code that can be 
easily merged to Neutron, and that the project can ill-afford the additional 
review cost it is likely to impose.

As such, and as was agreed to in the irc meeting this morning, the way forward 
is to recognize that the POC is best considered a prototype useful in informing 
efforts to iterate in the open.


m.


> 
> 
> 
> On Thu, May 22, 2014 at 4:03 PM, Maru Newby  wrote:
> On May 22, 2014, at 1:59 PM, Mandeep Dhami  wrote:
> 
> >
> > Maru's concerns are that:
> > 1. It is large
> > 2. It is complex
> 
> As per the discussion in the irc meeting today, I hope it is clear now that 
> eventual size and complexity are not real issue.  Rather, I am concerned at 
> how we get there.
> 
> I keep talking about 'iterating in the open', and want to make it clear what 
> I mean by this.  It involves proposing a reviewable patch to openstack 
> gerrit, working with reviewers to get the patch merged, and then 
> incorporating their feedback into the overall design to drive the 
> implementation of future patches.
> 
> 'Iterating in the open' does not imply working outside of gerrit to create a 
> monolithic codebase that needs to be manually decomposed into reviewable 
> chunks at the end.  I understand that this may be an effective way to create 
> a POC, but it is not an effective way to produce code that can be merged into 
> Neutron.  Core reviewers have a mandate to ensure the quality of every patch, 
> and their feedback is likely to have an impact on subsequent implementation.
> 
> 
> >
> > And Armando's related concerns are:
> > 3. Could dev/review cycles be better spent on refactoring
> > 4. If refactored neutron was available, would a simpler option become more 
> > viable
> >
> > Let me address them in that order.
> >
> > 1. Re: It is large
> > Group policy has an ambitious goal  - provide devop teams with policy based 
> > controls that are usable at scale and with automation (say a higher 
> > governance layer like Congress). The fact that meeting a large challenge 
> > requires more code is natural. We understand that challenge, and that is 
> > why we did a prototype (as PoC that was demonstrated on the summit). And 
> > based on that learning we are incrementally creating patches for building 
> > the group based policy. Just because a task is large, we as neutron can not 
> > shy away from building it. That will only drive people who need it out side 
> > neutron (as we are seeing with the frustration that the LBaaS team had 
> > because they have a requirement that is "large" as well).
> 
> Again, the amount of code is not the problem.  How code is introduced into 
> the tree, and how the design is socialized (both with developers and users), 
> _is_ of critical importance.  Neutron is not alone in requiring an 'iterate 
> in the open' approach - it is a characteristic common to many open source 
> projects.
> 
> 
> >
> > 2. Re: It is complex
> > Complexity depends on the context. Our goal was to make the end-user's life 
> > simpler (an

Re: [openstack-dev] [Neutron][FYI] Bookmarklet for neutron gerrit review

[Maru Newby]

+1

I think there should be naming standard for all reviews (e.g. [test] Jenkins, 
[test-external] VMware) so that gerrit css can colorize automatic review 
comments no matter the project.


m.

On Mar 7, 2014, at 2:08 AM, Chmouel Boudjnah  wrote:

> if peoples like this why don't we have it directly on the reviews?
> 
> Chmouel.
> 
> 
> On Tue, Mar 4, 2014 at 10:00 PM, Carl Baldwin  wrote:
> Nachi,
> 
> Great!  I'd been meaning to do something like this.  I took yours and
> tweaked it a bit to highlight failed Jenkins builds in red and grey
> other Jenkins messages.  Human reviews are left in blue.
> 
> javascript:(function(){
> list = document.querySelectorAll('td.GJEA35ODGC');
> for(i in list) {
> title = list[i];
> if(! title.innerHTML) { continue; }
> text = title.nextSibling;
> if (text.innerHTML.search('Build failed') > 0) {
> title.style.color='red'
> } else if(title.innerHTML.search('Jenkins|CI|Ryu|Testing|Mine') >= 0) 
> {
> title.style.color='#66'
> } else {
> title.style.color='blue'
> }
> }
> })()
> 
> Carl
> 
> On Wed, Feb 26, 2014 at 12:31 PM, Nachi Ueno  wrote:
> > Hi folks
> >
> > I wrote an bookmarklet for neutron gerrit review.
> > This bookmarklet make the comment title for 3rd party ci as gray.
> >
> > javascript:(function(){list =
> > document.querySelectorAll('td.GJEA35ODGC'); for(i in
> > list){if(!list[i].innerHTML){continue;};if(list[i].innerHTML &&
> > list[i].innerHTML.search('CI|Ryu|Testing|Mine') >
> > 0){list[i].style.color='#66'}else{list[i].style.color='red'}};})()
> >
> > enjoy :)
> > Nachi
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

[Maru Newby]

On Mar 21, 2014, at 9:01 AM, David Kranz  wrote:

> On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
>> 
>>> -Original Message-
>>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com]
>>> Sent: Thursday, March 20, 2014 12:13 PM
>>> 
>>> 'project specific functional testing' in the Marconi context is
>>> treating
>>> Marconi as a complete system, making Marconi API calls & verifying the
>>> response - just like an end user would, but without keystone. If one of
>>> these tests fail, it is because there is a bug in the Marconi code ,
>>> and
>>> not because its interaction with Keystone caused it to fail.
>>> 
>>> "That being said there are certain cases where having a project
>>> specific
>>> functional test makes sense. For example swift has a functional test
>>> job
>>> that
>>> starts swift in devstack. But, those things are normally handled on a
>>> per
>>> case
>>> basis. In general if the project is meant to be part of the larger
>>> OpenStack
>>> ecosystem then Tempest is the place to put functional testing. That way
>>> you know
>>> it works with all of the other components. The thing is in openstack
>>> what
>>> seems
>>> like a project isolated functional test almost always involves another
>>> project
>>> in real use cases. (for example keystone auth with api requests)
>>> 
>>> "



>>> 
>>> One of the concerns we heard in the review was 'having the functional
>>> tests elsewhere (I.e within the project itself) does not count and they
>>> have to be in Tempest'.
>>> This has made us as a team wonder if we should migrate all our
>>> functional
>>> tests to Tempest.
>>> But from Matt's response, I think it is reasonable to continue in our
>>> current path & have the functional tests in Marconi coexist  along with
>>> the tests in Tempest.
>>> 
>> I think that what is being asked, really is that the functional tests could 
>> be a single set of tests that would become a part of the tempest repository 
>> and that these tests would have an ENV variable as part of the configuration 
>> that would allow either "no Keystone" or "Keystone" or some such, if that is 
>> the only configuration issue that separates running the tests isolated vs. 
>> integrated.  The functional tests need to be as much as possible a single 
>> set of tests to reduce duplication and remove the likelihood of two sets 
>> getting out of sync with each other/development.  If they only run in the 
>> integrated environment, that's ok, but if you want to run them isolated to 
>> make debugging easier, then it should be a configuration option and a 
>> separate test job.
>> 
>> So, if my assumptions are correct, QA only requires functional tests for 
>> integrated runs, but if the project QAs/Devs want to run isolated for dev 
>> and devtest purposes, more power to them.  Just keep it a single set of 
>> functional tests and put them in the Tempest repository so that if a failure 
>> happens, anyone can find the test and do the debug work without digging into 
>> a separate project repository.
>> 
>> Hopefully, the tests as designed could easily take a new configuration 
>> directive and a short bit of work with OS QA will get the integrated FTs 
>> working as well as the isolated ones.
>> 
>> --Rocky
> This issue has been much debated. There are some active members of our 
> community who believe that all the functional tests should live outside of 
> tempest in the projects, albeit with the same idea that such tests could be 
> run either as part of today's "real" tempest runs or mocked in various ways 
> to allow component isolation or better performance. Maru Newby posted a patch 
> with an example of one way to do this but I think it expired and I don't have 
> a pointer.

I think the best place for functional api tests to be maintained is in the 
projects themselves.  The domain expertise required to write api tests is 
likely to be greater among project resources, and they should be tasked with 
writing api tests pre-merge.  The current 'merge-first, test-later' procedure 
of maintaining api tests in the Tempest repo makes that impossible.  Worse, the 
cost of developing functional api tests is higher in the integration 
environment that is the Tempest default.

The patch in question [1] proposes allowing pre-merge functional api test 
maintenance 

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

[Maru Newby]

On Mar 26, 2014, at 12:59 PM, Joe Gordon  wrote:

> 
> 
> 
> On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby  wrote:
> 
> On Mar 21, 2014, at 9:01 AM, David Kranz  wrote:
> 
> > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
> >>
> >>> -Original Message-
> >>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com]
> >>> Sent: Thursday, March 20, 2014 12:13 PM
> >>>
> >>> 'project specific functional testing' in the Marconi context is
> >>> treating
> >>> Marconi as a complete system, making Marconi API calls & verifying the
> >>> response - just like an end user would, but without keystone. If one of
> >>> these tests fail, it is because there is a bug in the Marconi code ,
> >>> and
> >>> not because its interaction with Keystone caused it to fail.
> >>>
> >>> "That being said there are certain cases where having a project
> >>> specific
> >>> functional test makes sense. For example swift has a functional test
> >>> job
> >>> that
> >>> starts swift in devstack. But, those things are normally handled on a
> >>> per
> >>> case
> >>> basis. In general if the project is meant to be part of the larger
> >>> OpenStack
> >>> ecosystem then Tempest is the place to put functional testing. That way
> >>> you know
> >>> it works with all of the other components. The thing is in openstack
> >>> what
> >>> seems
> >>> like a project isolated functional test almost always involves another
> >>> project
> >>> in real use cases. (for example keystone auth with api requests)
> >>>
> >>> "
> 
> 
> 
> >>>
> >>> One of the concerns we heard in the review was 'having the functional
> >>> tests elsewhere (I.e within the project itself) does not count and they
> >>> have to be in Tempest'.
> >>> This has made us as a team wonder if we should migrate all our
> >>> functional
> >>> tests to Tempest.
> >>> But from Matt's response, I think it is reasonable to continue in our
> >>> current path & have the functional tests in Marconi coexist  along with
> >>> the tests in Tempest.
> >>>
> >> I think that what is being asked, really is that the functional tests 
> >> could be a single set of tests that would become a part of the tempest 
> >> repository and that these tests would have an ENV variable as part of the 
> >> configuration that would allow either "no Keystone" or "Keystone" or some 
> >> such, if that is the only configuration issue that separates running the 
> >> tests isolated vs. integrated.  The functional tests need to be as much as 
> >> possible a single set of tests to reduce duplication and remove the 
> >> likelihood of two sets getting out of sync with each other/development.  
> >> If they only run in the integrated environment, that's ok, but if you want 
> >> to run them isolated to make debugging easier, then it should be a 
> >> configuration option and a separate test job.
> >>
> >> So, if my assumptions are correct, QA only requires functional tests for 
> >> integrated runs, but if the project QAs/Devs want to run isolated for dev 
> >> and devtest purposes, more power to them.  Just keep it a single set of 
> >> functional tests and put them in the Tempest repository so that if a 
> >> failure happens, anyone can find the test and do the debug work without 
> >> digging into a separate project repository.
> >>
> >> Hopefully, the tests as designed could easily take a new configuration 
> >> directive and a short bit of work with OS QA will get the integrated FTs 
> >> working as well as the isolated ones.
> >>
> >> --Rocky
> > This issue has been much debated. There are some active members of our 
> > community who believe that all the functional tests should live outside of 
> > tempest in the projects, albeit with the same idea that such tests could be 
> > run either as part of today's "real" tempest runs or mocked in various ways 
> > to allow component isolation or better performance. Maru Newby posted a 
> > patch with an example of one way to do this but I think it expired and I 
> > don't have a pointer.
> 
> I think the best place for functional api tests to be maint

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

[Maru Newby]

On Mar 26, 2014, at 1:52 PM, Sean Dague  wrote:

> On 03/25/2014 04:49 AM, Maru Newby wrote:
>> 
>> On Mar 21, 2014, at 9:01 AM, David Kranz  wrote:
>> 
>>> On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
>>>> 
>>>>> -Original Message-
>>>>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com]
>>>>> Sent: Thursday, March 20, 2014 12:13 PM
>>>>> 
>>>>> 'project specific functional testing' in the Marconi context is
>>>>> treating
>>>>> Marconi as a complete system, making Marconi API calls & verifying the
>>>>> response - just like an end user would, but without keystone. If one of
>>>>> these tests fail, it is because there is a bug in the Marconi code ,
>>>>> and
>>>>> not because its interaction with Keystone caused it to fail.
>>>>> 
>>>>> "That being said there are certain cases where having a project
>>>>> specific
>>>>> functional test makes sense. For example swift has a functional test
>>>>> job
>>>>> that
>>>>> starts swift in devstack. But, those things are normally handled on a
>>>>> per
>>>>> case
>>>>> basis. In general if the project is meant to be part of the larger
>>>>> OpenStack
>>>>> ecosystem then Tempest is the place to put functional testing. That way
>>>>> you know
>>>>> it works with all of the other components. The thing is in openstack
>>>>> what
>>>>> seems
>>>>> like a project isolated functional test almost always involves another
>>>>> project
>>>>> in real use cases. (for example keystone auth with api requests)
>>>>> 
>>>>> "
>> 
>> 
>> 
>>>>> 
>>>>> One of the concerns we heard in the review was 'having the functional
>>>>> tests elsewhere (I.e within the project itself) does not count and they
>>>>> have to be in Tempest'.
>>>>> This has made us as a team wonder if we should migrate all our
>>>>> functional
>>>>> tests to Tempest.
>>>>> But from Matt's response, I think it is reasonable to continue in our
>>>>> current path & have the functional tests in Marconi coexist  along with
>>>>> the tests in Tempest.
>>>>> 
>>>> I think that what is being asked, really is that the functional tests 
>>>> could be a single set of tests that would become a part of the tempest 
>>>> repository and that these tests would have an ENV variable as part of the 
>>>> configuration that would allow either "no Keystone" or "Keystone" or some 
>>>> such, if that is the only configuration issue that separates running the 
>>>> tests isolated vs. integrated.  The functional tests need to be as much as 
>>>> possible a single set of tests to reduce duplication and remove the 
>>>> likelihood of two sets getting out of sync with each other/development.  
>>>> If they only run in the integrated environment, that's ok, but if you want 
>>>> to run them isolated to make debugging easier, then it should be a 
>>>> configuration option and a separate test job.
>>>> 
>>>> So, if my assumptions are correct, QA only requires functional tests for 
>>>> integrated runs, but if the project QAs/Devs want to run isolated for dev 
>>>> and devtest purposes, more power to them.  Just keep it a single set of 
>>>> functional tests and put them in the Tempest repository so that if a 
>>>> failure happens, anyone can find the test and do the debug work without 
>>>> digging into a separate project repository.
>>>> 
>>>> Hopefully, the tests as designed could easily take a new configuration 
>>>> directive and a short bit of work with OS QA will get the integrated FTs 
>>>> working as well as the isolated ones.
>>>> 
>>>> --Rocky
>>> This issue has been much debated. There are some active members of our 
>>> community who believe that all the functional tests should live outside of 
>>> tempest in the projects, albeit with the same idea that such tests could be 
>>> run either as part of today's "real" tempest runs or mocked in various ways 
>>> to allow component isolation or better pe

[openstack-dev] [neutron][stable] metadata agent performance

[Maru Newby]

We merged caching support for the metadata agent in juno, and backported to 
icehouse.  It was enabled by default in juno, but disabled by default in 
icehouse to satisfy the stable maint requirement of not changing functional 
behavior.

While performance of the agent was improved with caching enabled, it regressed 
a reported 8x when caching was disabled [1].  This means that by default, the 
caching backport severely impacts icehouse Neutron's performance.

So, what is the way forward?  We definitely need to document the problem for 
both icehouse and juno.  Is documentation enough?  Or can we enable caching by 
default in icehouse?  Or remove the backport entirely.

There is also a proposal to replace the metadata agent’s use of the neutron 
client in favor of rpc [2].  There were comments on an old bug suggesting we 
didn’t want to do this [3], but assuming that we want this change in Kilo, is 
backporting even a possibility given that it implies a behavioral change to be 
useful?

Thanks,


Maru



1: https://bugs.launchpad.net/cloud-archive/+bug/1361357
2: https://review.openstack.org/#/c/121782
3: https://bugs.launchpad.net/neutron/+bug/1092043
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][stable] metadata agent performance

[Maru Newby]

On Oct 22, 2014, at 12:53 AM, Jakub Libosvar  wrote:

> On 10/22/2014 02:26 AM, Maru Newby wrote:
>> We merged caching support for the metadata agent in juno, and backported to 
>> icehouse.  It was enabled by default in juno, but disabled by default in 
>> icehouse to satisfy the stable maint requirement of not changing functional 
>> behavior.
>> 
>> While performance of the agent was improved with caching enabled, it 
>> regressed a reported 8x when caching was disabled [1].  This means that by 
>> default, the caching backport severely impacts icehouse Neutron's 
>> performance.
>> 
>> So, what is the way forward?  We definitely need to document the problem for 
>> both icehouse and juno.  Is documentation enough?  Or can we enable caching 
>> by default in icehouse?  Or remove the backport entirely.
>> 
>> There is also a proposal to replace the metadata agent’s use of the neutron 
>> client in favor of rpc [2].  There were comments on an old bug suggesting we 
>> didn’t want to do this [3], but assuming that we want this change in Kilo, 
>> is backporting even a possibility given that it implies a behavioral change 
>> to be useful?
>> 
>> Thanks,
>> 
>> 
>> Maru
>> 
>> 
>> 
>> 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357
>> 2: https://review.openstack.org/#/c/121782
>> 3: https://bugs.launchpad.net/neutron/+bug/1092043
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> I thought the performance regression was caused by wrong keystone token
> caching leading to authentication per neutron client instance. Fix was
> backported to Icehouse [1].
> 
> Does it mean this patch hasn't solved the problem and regression is
> somewhere else?

As you say (and as per Ihar and Akihiro’s responses), the problem was fixed.  I 
was confused by the bug that Oleg’s RPC proposal hard targeted as a related bug 
and thought the problem wasn’t yet resolved, but looking at it again it appears 
the bug is a Ubuntu distro issue.  Accordingly, I’ve removed Neutron as a 
target for that bug.  Apologies for the confusion.


Maru 

> Kuba
> 
> [1] https://review.openstack.org/#/c/120418/
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [NFV] Patches tagging with NFVImpact

[Maru Newby]

Am I the only one wondering whether introducing arbitrary tagging into our 
commit messages sets a bad precedent?  Or was there a discussion on this topic 
that I missed? 


Maru

On Jun 17, 2014, at 2:27 PM, Sylvain Bauza  wrote:

> Hi,
> 
> There is an action for creating a Gerrit dashboard to show all
> NFV-related specs and patches.
> As it requires to have a specific label for discriminating them, I'm
> adding "NFVImpact" in all the patches proposed in [1].
> The side effect is that it creates another patchset and so runs another
> Jenkins check so don't worry about it.
> 
> I'll do a team status by tomorrow but meanwhile, please make sure that
> if you upload a new patch to Gerrit, it will be marked as "NFVImpact" in
> the commit message.
> 
> Many thanks,
> -Sylvain
> 
> 
> [1] https://wiki.openstack.org/wiki/Teams/NFV#Active_Blueprints
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron]issue about dvr flows and port timestamp

[Maru Newby]

On Oct 29, 2014, at 8:12 AM, Yangxurong  wrote:

> Hi,
>  
> I’m not sure whether following issue is problematic, and both, our team do 
> some effort, so I submit two blueprints:
> [1.] optimize dvr flows:
> Currently, accurate ovs flows in terms of full mac are used to communicate 
> among distributed router, but here comes problem : (1)the more distributed 
> router node, the more flows; (2)different distributed router across DC cannot 
> communicate through tunnel and additional operation under same mac prefix 
> configuration. So it would be useful to shift the complete-matching of mac to 
> fuzzy Matching, like prefix matching, reducing the number of flows and 
> leading to communicate among different DC though configuring same mac prefix 
> through tunnel.
> Link: https://blueprints.launchpad.net/neutron/+spec/optimize-dvr-flows

I think we need to focus on paying down technical debt (both in the code and on 
the testing side) related to dvr before we seriously consider the kind of 
optimization that you are proposing.  I’m also unclear as to why we would want 
to pursue a solution to a problem whose severity doesn’t appear to be clear 
("I’m not sure whether the following issue is problematic…").


Maru

>  
> [2.]add port timestamp:
> It would be worth adding timestamp fields including create_at, update_at and 
> delete_at in the table of port in neutron, so users can monitor port change 
> conveniently, for example portal or management center wants to query the 
> ports that have changed or refreshed during the latest 5sec in a large scale 
> application. If not, it's time-consuming and low effectiveness.
> Link: https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp
>  
> Any response I will appreciate.
>  
> Thanks,
> Xurong Yang
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [NFV] Patches tagging with NFVImpact

[Maru Newby]

On Oct 29, 2014, at 12:25 PM, Steve Gordon  wrote:

> - Original Message -
>> From: "Maru Newby" 
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> 
>> Am I the only one wondering whether introducing arbitrary tagging into our
>> commit messages sets a bad precedent?  Or was there a discussion on this
>> topic that I missed?
>> 
>> Maru
> 
> Any particular reason for digging this up from June? We already changed the 
> process so that we did not need to do this.
> 
> Thanks,
> 
> Steve

I had a mail filter applied when I looked at the list this morning and I failed 
to notice that this was an old thread until I had already sent my reply.  That 
said, we have Neutron specs up for review that still include this tag in the 
commit message and now I know at least that I can ask for their removal.


Maru



>> On Jun 17, 2014, at 2:27 PM, Sylvain Bauza  wrote:
>> 
>>> Hi,
>>> 
>>> There is an action for creating a Gerrit dashboard to show all
>>> NFV-related specs and patches.
>>> As it requires to have a specific label for discriminating them, I'm
>>> adding "NFVImpact" in all the patches proposed in [1].
>>> The side effect is that it creates another patchset and so runs another
>>> Jenkins check so don't worry about it.
>>> 
>>> I'll do a team status by tomorrow but meanwhile, please make sure that
>>> if you upload a new patch to Gerrit, it will be marked as "NFVImpact" in
>>> the commit message.
>>> 
>>> Many thanks,
>>> -Sylvain
>>> 
>>> 
>>> [1] https://wiki.openstack.org/wiki/Teams/NFV#Active_Blueprints
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> -- 
> Steve Gordon, RHCE
> Sr. Technical Product Manager,
> Red Hat Enterprise Linux OpenStack Platform
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron]issue about dvr flows and port timestamp

[Maru Newby]

On Oct 29, 2014, at 1:36 PM, Hly  wrote:

> 
> 
> Sent from my iPad
> 
> On 2014-10-29, at 下午6:33, Maru Newby  wrote:
> 
>> 
>> On Oct 29, 2014, at 8:12 AM, Yangxurong  wrote:
>> 
>>> Hi,
>>> 
>>> I’m not sure whether following issue is problematic, and both, our team do 
>>> some effort, so I submit two blueprints:
>>> [1.] optimize dvr flows:
>>> Currently, accurate ovs flows in terms of full mac are used to communicate 
>>> among distributed router, but here comes problem : (1)the more distributed 
>>> router node, the more flows; (2)different distributed router across DC 
>>> cannot communicate through tunnel and additional operation under same mac 
>>> prefix configuration. So it would be useful to shift the complete-matching 
>>> of mac to fuzzy Matching, like prefix matching, reducing the number of 
>>> flows and leading to communicate among different DC though configuring same 
>>> mac prefix through tunnel.
>>> Link: https://blueprints.launchpad.net/neutron/+spec/optimize-dvr-flows
>> 
>> I think we need to focus on paying down technical debt (both in the code and 
>> on the testing side) related to dvr before we seriously consider the kind of 
>> optimization that you are proposing.  I’m also unclear as to why we would 
>> want to pursue a solution to a problem whose severity doesn’t appear to be 
>> clear ("I’m not sure whether the following issue is problematic…").
>> 
> 
> DVR stability is the first class for sure, but if the code and logic could be 
> less and simpler, there is more chance of stability. By my understanding, 
> since DVR mac range has been configured as a prefix, so prefix based 
> judgement instead of one by one flow setup triggered by mesh-like message 
> notifying would simplify the code logic, thus indirectly contribute to 
> overall stability. Also, it would remove hundreds of flows in the ovs in a 
> middle scale cluster, very helpful for trouble shooting.

If we’re going to refactor/optimize/whatever any part of Neutron, the first 
requirement is that we can maintain the expectations of the code (i.e. ensure 
good test coverage) across changes.  DVR is no different from any other feature 
in this regard.  I would welcome any effort you want to devote to improving 
DVR’s testing story such that the kind of optimizations you are proposing would 
have less potential to cause regressions.

In addition to baseline test coverage, I would also expect to see test 
additions validating flow generation such that reviewers would be able to 
easily identify the benefit of your proposed optimizations by comparing the 
results before and after.


Maru


> Wu
> 
>> 
>> Maru
>> 
>>> 
>>> [2.]add port timestamp:
>>> It would be worth adding timestamp fields including create_at, update_at 
>>> and delete_at in the table of port in neutron, so users can monitor port 
>>> change conveniently, for example portal or management center wants to query 
>>> the ports that have changed or refreshed during the latest 5sec in a large 
>>> scale application. If not, it's time-consuming and low effectiveness.
>>> Link: https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp
>>> 
>>> Any response I will appreciate.
>>> 
>>> Thanks,
>>> Xurong Yang
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [nova] Weekly nova-network / neutron parity meeting

[Maru Newby]

On Jul 17, 2014, at 8:12 AM, Kyle Mestery  wrote:

> On Thu, Jul 17, 2014 at 6:42 AM, Thierry Carrez  wrote:
>> Kyle Mestery wrote:
>>> As we're getting down to the wire in Juno, I'd like to propose we have
>>> a weekly meeting on the nova-network and neutron parity effort. I'd
>>> like to start this meeting next week, and I'd like to propose
>>> Wednesday at 1500 UTC on #openstack-meeting-3 as the time and
>>> location.
>> 
>> This conflicts on the agenda with the "PHP SDK Team Meeting".
>> That said, I'm not sure they still run this one regularly.
>> 
> We could do Wednesday or Friday at 1500 UTC if that works better for
> folks. I'd prefer Wednesday, though. We could also do 1400UTC
> Wednesday.
> 
> Perhaps I'll split the difference and do 1430 UTC Wednesday. Does that
> sound ok to everyone?
> 
> Thanks,
> Kyle

+1 from me.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Tenant isolation gate failures?

[Maru Newby]

The tenant isolation gates that have been failing so frequently seem to be 
passing all of a sudden.  I didn't see any merges that claimed to fix the 
issue, so maybe this is just a lull due to a lower volume of gate jobs.  If it 
was intentional, though, I would appreciate knowing which patch or patches 
resolved the problem.

Thanks in advance,


Maru
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Networking] Basic Bug Etiquette

[Maru Newby]

I'd like to reminder all Neutron contributors that if you are not the assignee 
for a bug, you should be consulting the assignee before you consider 
reassigning the bug to yourself or others.  Failing to do so is not only 
disrespectful, but may also result in unnecessary work.  It could be that the 
assignee has been making progress on the problem - code or otherwise - that 
would be duplicated by the new assignee.  Or the assignee could have 
specialized knowledge that makes them best equipped to fix the bug.  In any 
case, please ensure that bugs are not reassigned without the involvement of the 
current assignee.

Thanks for reading,


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] L3 agent bug - metadata nat rule removal

[Maru Newby]

Hi Nachi,

The current neutron gate failure is due to the following nat rule being cleared 
from the router namespace when the l3 agent syncs the router:

-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 
-j REDIRECT --to-ports 9697

The only place the metadata nat rule appears to be applied is when a router is 
detected as being added by the l3 agent.

I'm unclear on whether the failure is due to not having the metadata nat rule 
added on sync, or if the sync is supposed to retain it.  Do you have any 
insight on this?

See the comments on the bug for more info: 
https://bugs.launchpad.net/neutron/+bug/1211829


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Code review study

[Maru Newby]

On Aug 15, 2013, at 12:50 PM, Joe Gordon  wrote:

>   • 
> On Thu, Aug 15, 2013 at 12:22 PM, Sam Harwell  
> wrote:
> I like to take a different approach. If my commit message is going to take 
> more than a couple lines for people to understand the decisions I made, I go 
> and make an issue in the issue tracker before committing locally and then 
> reference that issue in the commit message. This helps in a few ways:
> 
>  
> 
> 1.   If I find a technical or grammatical error in the commit message, it 
> can be corrected.
> 
> 2.   Developers can provide feedback on the subject matter independently 
> of the implementation, as well as feedback on the implementation itself.
> 
> 3.   I like the ability to include formatting and hyperlinks in my 
> documentation of the commit.
> 
>  
> 
> 
> This pattern has one slight issue, which is:
>  
>   • Do not assume the reviewer has access to external web services/site.
> In 6 months time when someone is on a train/plane/coach/beach/pub 
> troubleshooting a problem & browsing GIT history, there is no guarantee they 
> will have access to the online bug tracker, or online blueprint documents. 
> The great step forward with distributed SCM is that you no longer need to be 
> "online" to have access to all information about the code repository. The 
> commit message should be totally self-contained, to maintain that benefit.

I'm not sure I agree with this.  It can't be true in all cases, so it can 
hardly be considered a rule.  A guideline, maybe - something to strive for.  
But not all artifacts of the development process are amenable to being stuffed 
into code or the commits associated with them.  A dvcs is great and all, but 
unless one is working in a silo, online resources are all but mandatory.


m.

> 
> 
> https://wiki.openstack.org/wiki/GitCommitMessages#Information_in_commit_messages
> 
> 
> 
>  
> 
> Sam
> 
>  
> 
> From: Christopher Yeoh [mailto:cbky...@gmail.com] 
> Sent: Thursday, August 15, 2013 7:12 AM
> To: OpenStack Development Mailing List
> Subject: Re: [openstack-dev] Code review study
> 
>  
> 
>  
> 
> On Thu, Aug 15, 2013 at 11:42 AM, Robert Collins  
> wrote:
> 
> This may interest data-driven types here.
> 
> https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/
> 
> Note specifically the citation of 200-400 lines as the knee of the review 
> effectiveness curve: that's lower than I thought - I thought 200 was clearly 
> fine - but no.
> 
>  
> 
> Very interesting article. One other point which I think is pretty relevant is 
> point 4 about getting authors to annotate the code better (and for those who 
> haven't read it, they don't mean comments in the code but separately) because 
> it results in the authors picking up more bugs before they even submit the 
> code.
> 
> So I wonder if its worth asking people to write more detailed commit logs 
> which include some reasoning about why some of the more complex changes were 
> done in a certain way and not just what is implemented or fixed. As it is 
> many of the commit messages are often very succinct so I think it would help 
> on the review efficiency side too.
> 
>  
> 
> Chris
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Code review study

[Maru Newby]

On Aug 16, 2013, at 2:12 AM, Robert Collins  wrote:

> On 16 August 2013 20:15, Maru Newby  wrote:
> 
>>> This pattern has one slight issue, which is:
>>> 
>>>  • Do not assume the reviewer has access to external web services/site.
>>> In 6 months time when someone is on a train/plane/coach/beach/pub 
>>> troubleshooting a problem & browsing GIT history, there is no guarantee 
>>> they will have access to the online bug tracker, or online blueprint 
>>> documents. The great step forward with distributed SCM is that you no 
>>> longer need to be "online" to have access to all information about the code 
>>> repository. The commit message should be totally self-contained, to 
>>> maintain that benefit.
>> 
>> I'm not sure I agree with this.  It can't be true in all cases, so it can 
>> hardly be considered a rule.  A guideline, maybe - something to strive for.  
>> But not all artifacts of the development process are amenable to being 
>> stuffed into code or the commits associated with them.  A dvcs is great and 
>> all, but unless one is working in a silo, online resources are all but 
>> mandatory.
> 
> In a very strict sense you're right, but consider that for anyone
> doing fast iterative development the need to go hit a website is a
> huge slowdown : at least in most of the world :).

You're suggesting that it's possible to do _fast_ iterative development on a 
distributed system of immense and largely undocumented complexity (like 
openstack)?  I'd like to be working on the code you're working on!  ;) 


m.

> 
> So - while I agree that it's something to strive for, I think we
> should invert it and say 'not having everything in the repo is
> something we should permit occasional exceptions to'.
> 
> -Rob
> 
> -- 
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Gate breakage process - Let's fix! (related but not specific to neutron)

[Maru Newby]

Neutron has been in and out of the gate for the better part of the past month, 
and it didn't slow the pace of development one bit.  Most Neutron developers 
kept on working as if nothing was wrong, blithely merging changes with no 
guarantees that they weren't introducing new breakage.  New bugs were indeed 
merged, greatly increasing the time and effort required to get Neutron back in 
the gate.  I don't think this is sustainable, and I'd like to make a suggestion 
for how to minimize the impact of gate breakage.

For the record, I don't think consistent gate breakage in one project should be 
allowed to hold up the development of other projects.  The current approach of 
skipping tests or otherwise making a given job non-voting for innocent projects 
should continue.  It is arguably worth taking the risk of relaxing gating for 
those innocent projects rather than halting development unnecessarily.

However, I don't think it is a good idea to relax a broken gate for the 
offending project.  So if a broken job/test is clearly Neutron related, it 
should continue to gate Neutron, effectively preventing merges until the 
problem is fixed.  This would both raise the visibility of breakage beyond the 
person responsible for fixing it, and prevent additional breakage from slipping 
past were the gating to be relaxed.

Thoughts?


m.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gate breakage process - Let's fix! (related but not specific to neutron)

[Maru Newby]

On Aug 16, 2013, at 11:44 AM, Monty Taylor  wrote:

> 
> 
> On 08/16/2013 02:25 PM, Maru Newby wrote:
>> Neutron has been in and out of the gate for the better part of the
>> past month, and it didn't slow the pace of development one bit.  Most
>> Neutron developers kept on working as if nothing was wrong, blithely
>> merging changes with no guarantees that they weren't introducing new
>> breakage.  New bugs were indeed merged, greatly increasing the time
>> and effort required to get Neutron back in the gate.  I don't think
>> this is sustainable, and I'd like to make a suggestion for how to
>> minimize the impact of gate breakage.
>> 
>> For the record, I don't think consistent gate breakage in one project
>> should be allowed to hold up the development of other projects.  The
>> current approach of skipping tests or otherwise making a given job
>> non-voting for innocent projects should continue.  It is arguably
>> worth taking the risk of relaxing gating for those innocent projects
>> rather than halting development unnecessarily.
>> 
>> However, I don't think it is a good idea to relax a broken gate for
>> the offending project.  So if a broken job/test is clearly Neutron
>> related, it should continue to gate Neutron, effectively preventing
>> merges until the problem is fixed.  This would both raise the
>> visibility of breakage beyond the person responsible for fixing it,
>> and prevent additional breakage from slipping past were the gating to
>> be relaxed.
> 
> I do not know the exact implementation that would work here, but I do
> think it's worth discussing further. Essentially, a neutron bug killing
> the gate for a nova dev isn't necessarily going to help - because the
> nova dev doesn't necessarily have the background to fix it.
> 
> I want to be very careful that we don't wind up with an assymetrical
> gate though…

What are your concerns regarding an 'asymmetrical gate'?  By halting neutron 
development until neutron-caused breakage is fixed, there would presumably be 
sufficient motivation to ensure timely resolution.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gate breakage process - Let's fix! (related but not specific to neutron)

[Maru Newby]

On Aug 16, 2013, at 11:44 AM, Clint Byrum  wrote:

> Excerpts from Maru Newby's message of 2013-08-16 11:25:07 -0700:
>> Neutron has been in and out of the gate for the better part of the past 
>> month, and it didn't slow the pace of development one bit.  Most Neutron 
>> developers kept on working as if nothing was wrong, blithely merging changes 
>> with no guarantees that they weren't introducing new breakage.  New bugs 
>> were indeed merged, greatly increasing the time and effort required to get 
>> Neutron back in the gate.  I don't think this is sustainable, and I'd like 
>> to make a suggestion for how to minimize the impact of gate breakage.
>> 
>> For the record, I don't think consistent gate breakage in one project should 
>> be allowed to hold up the development of other projects.  The current 
>> approach of skipping tests or otherwise making a given job non-voting for 
>> innocent projects should continue.  It is arguably worth taking the risk of 
>> relaxing gating for those innocent projects rather than halting development 
>> unnecessarily.
>> 
>> However, I don't think it is a good idea to relax a broken gate for the 
>> offending project.  So if a broken job/test is clearly Neutron related, it 
>> should continue to gate Neutron, effectively preventing merges until the 
>> problem is fixed.  This would both raise the visibility of breakage beyond 
>> the person responsible for fixing it, and prevent additional breakage from 
>> slipping past were the gating to be relaxed.
>> 
>> Thoughts?
>> 
> 
> I think this is a cultural problem related to the code review discussing
> from earlier in the week.
> 
> We are not looking at finding a defect and reverting as a good thing where
> high fives should be shared all around. Instead, "you broke the gate"
> seems to mean "you are a bad developer". I have been a bad actor here too,
> getting frustrated with the gate-breaker and saying the wrong thing.
> 
> The problem really is "you _broke_ the gate". It should be "the gate has
> found a defect, hooray!". It doesn't matter what causes the gate to stop,
> it is _always_ a defect. Now, it is possible the defect is in tempest,
> or jenkins, or HP/Rackspace's clouds where the tests run. But it is
> always a defect that what worked before, does not work now.
> 
> Defects are to be expected. None of us can write perfect code. We should
> be happy to revert commits and go forward with an enabled gate while
> the team responsible for the commit gathers information and works to
> correct the issue.

You're preaching to the choir, and I suspect that anyone with an interest in 
software quality is likely to prefer problem solving to finger pointing.  
However, my intent with this thread was not to promote more constructive 
thinking about defect detection.  Rather, I was hoping to communicate a flaw in 
the existing process and seek consensus on how that process could best be 
modified to minimize the cost of resolving gate breakage.


> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Neutron + Grenade (or other upgrade testing)

[Maru Newby]

Is anyone working on/planning on adding support for neutron to grenade?  Or is 
there any other automated upgrade testing going on for neutron?

Thanks,


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] About multihost patch review

[Maru Newby]

On Aug 26, 2013, at 4:06 PM, Edgar Magana  wrote:

> Hi Developers,
> 
> Let me explain my point of view on this topic and please share your thoughts 
> in order to merge this new feature ASAP.
> 
> My understanding is that multi-host is nova-network HA  and we are 
> implementing this bp 
> https://blueprints.launchpad.net/neutron/+spec/quantum-multihost for the same 
> reason.
> So, If in neutron configuration admin enables multi-host:
> etc/dhcp_agent.ini
> 
> # Support multi host networks
> # enable_multihost = False
> 
> Why do tenants needs to be aware of this? They should just create networks in 
> the way they normally do and not by adding the "multihost" extension.

I was pretty confused until I looked at the nova-network HA doc [1].  The 
proposed design would seem to emulate nova-network's multi-host HA option, 
where it was necessary to both run nova-network on every compute node and 
create a network explicitly as multi-host.  I'm not sure why nova-network was 
implemented in this way, since it would appear that multi-host is basically 
all-or-nothing.  Once nova-network services are running on every compute node, 
what does it mean to create a network that is not multi-host?

So, to Edgar's question - is there a reason other than 'be like nova-network' 
for requiring neutron multi-host to be configured per-network?


m.

1: 
http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html


> I could be totally wrong and crazy, so please provide some feedback.
> 
> Thanks,
> 
> Edgar
> 
> 
> From: Yongsheng Gong 
> Date: Monday, August 26, 2013 2:58 PM
> To: "Kyle Mestery (kmestery)" , Aaron Rosen 
> , Armando Migliaccio , Akihiro 
> MOTOKI , Edgar Magana , Maru Newby 
> , Nachi Ueno , Salvatore Orlando 
> , Sumit Naiksatam , Mark 
> McClain , Gary Kotton , 
> Robert Kukura 
> Cc: OpenStack List 
> Subject: Re: About multihost patch review
> 
> Hi,
> Edgar Magana has commented to say:
> 'This is the part that for me is confusing and I will need some clarification 
> from the community. Do we expect to have the multi-host feature as an 
> extension or something that will natural work as long as the deployment 
> include more than one Network Node. In my opinion, Neutron deployments with 
> more than one Network Node by default should call DHCP agents in all those 
> nodes without the need to use an extension. If the community has decided to 
> do this by extensions, then I am fine' at
> https://review.openstack.org/#/c/37919/11/neutron/extensions/multihostnetwork.py
> 
> I have commented back, what is your opinion about it?
> 
> Regards,
> Yong Sheng Gong
> 
> 
> On Fri, Aug 16, 2013 at 9:28 PM, Kyle Mestery (kmestery)  
> wrote:
>> Hi Yong:
>> 
>> I'll review this and try it out today.
>> 
>> Thanks,
>> Kyle
>> 
>> On Aug 15, 2013, at 10:01 PM, Yongsheng Gong  wrote:
>> 
>> > The multihost patch is there for a long long time, can someone help to 
>> > review?
>> > https://review.openstack.org/#/c/37919/
>> 
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] About multihost patch review

[Maru Newby]

On Aug 26, 2013, at 9:39 PM, Yongsheng Gong  wrote:

> First 'be like nova-network' is a merit for some deployments.

I'm afraid 'merit' is a bit vague for me.  Would you please elaborate?
 

> second, To allow admin to decide which network will be multihosted at runtime 
> will enable the neutron to continue using the current network node (dhcp 
> agent) mode at the same time.

If multi-host and non- multi-host networks are permitted to co-exist (because 
configuration is per-network), won't compute nodes have to be allowed to be 
heterogenous (some multi-host capable, some not)?  And won't Nova then need to 
schedule VMs configured with multi-host networks on compatible nodes?  I don't 
recall mention of this issue in the blueprint or design doc, and would 
appreciate pointers to where this decision was documented.


> 
> If we force the network multihosted when the configuration enable_multihost 
> is true, and then administrator wants to transfer to normal neutron way, 
> he/she must modify the configuration item and then restart.

I'm afraid I don't follow - are you suggesting that configuring multi-host 
globally will be harder on admins than the change under review?  Switching to 
non multi-host under the current proposal involves reconfiguring and restarting 
of an awful lot of agents, to say nothing of the db changes.


m. 


> 
> 
> 
> On Tue, Aug 27, 2013 at 9:14 AM, Maru Newby  wrote:
> 
> On Aug 26, 2013, at 4:06 PM, Edgar Magana  wrote:
> 
> > Hi Developers,
> >
> > Let me explain my point of view on this topic and please share your 
> > thoughts in order to merge this new feature ASAP.
> >
> > My understanding is that multi-host is nova-network HA  and we are 
> > implementing this bp 
> > https://blueprints.launchpad.net/neutron/+spec/quantum-multihost for the 
> > same reason.
> > So, If in neutron configuration admin enables multi-host:
> > etc/dhcp_agent.ini
> >
> > # Support multi host networks
> > # enable_multihost = False
> >
> > Why do tenants needs to be aware of this? They should just create networks 
> > in the way they normally do and not by adding the "multihost" extension.
> 
> I was pretty confused until I looked at the nova-network HA doc [1].  The 
> proposed design would seem to emulate nova-network's multi-host HA option, 
> where it was necessary to both run nova-network on every compute node and 
> create a network explicitly as multi-host.  I'm not sure why nova-network was 
> implemented in this way, since it would appear that multi-host is basically 
> all-or-nothing.  Once nova-network services are running on every compute 
> node, what does it mean to create a network that is not multi-host?
> 
> So, to Edgar's question - is there a reason other than 'be like nova-network' 
> for requiring neutron multi-host to be configured per-network?
> 
> 
> m.
> 
> 1: 
> http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html
> 
> 
> > I could be totally wrong and crazy, so please provide some feedback.
> >
> > Thanks,
> >
> > Edgar
> >
> >
> > From: Yongsheng Gong 
> > Date: Monday, August 26, 2013 2:58 PM
> > To: "Kyle Mestery (kmestery)" , Aaron Rosen 
> > , Armando Migliaccio , Akihiro 
> > MOTOKI , Edgar Magana , Maru Newby 
> > , Nachi Ueno , Salvatore Orlando 
> > , Sumit Naiksatam , 
> > Mark McClain , Gary Kotton 
> > , Robert Kukura 
> > Cc: OpenStack List 
> > Subject: Re: About multihost patch review
> >
> > Hi,
> > Edgar Magana has commented to say:
> > 'This is the part that for me is confusing and I will need some 
> > clarification from the community. Do we expect to have the multi-host 
> > feature as an extension or something that will natural work as long as the 
> > deployment include more than one Network Node. In my opinion, Neutron 
> > deployments with more than one Network Node by default should call DHCP 
> > agents in all those nodes without the need to use an extension. If the 
> > community has decided to do this by extensions, then I am fine' at
> > https://review.openstack.org/#/c/37919/11/neutron/extensions/multihostnetwork.py
> >
> > I have commented back, what is your opinion about it?
> >
> > Regards,
> > Yong Sheng Gong
> >
> >
> > On Fri, Aug 16, 2013 at 9:28 PM, Kyle Mestery (kmestery) 
> >  wrote:
> >> Hi Yong:
> >>
> >> I'll review this and try it out today.
> >>
> >> Thanks,
> >> Kyle
> >>
> >> On Aug 15, 2013, at 10:01 PM, Yongsheng Gong  
> >> wrote:
> >>
> >> > The multihost patch is there for a long long time, can someone help to 
> >> > review?
> >> > https://review.openstack.org/#/c/37919/
> >>
> >
> 
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Neutron + Grenade (or other upgrade testing)

[Maru Newby]

On Aug 26, 2013, at 10:23 AM, Dean Troyer  wrote:

> On Mon, Aug 26, 2013 at 10:50 AM, Maru Newby  wrote:
> Is anyone working on/planning on adding support for neutron to grenade?  Or 
> is there any other automated upgrade testing going on for neutron?
> 
> We deliberately avoided migrations in Grenade (like Nova Volume -> Cinder) as 
> we wanted to focus on upgrades within projects.  Migrations will necessarily 
> be much more complicated, especially Nova Network -> Neutron.  At some point 
> Neutron should be added to Grenade, but only as a release upgrade step for 
> some basic configuration.
> 
> That said, I'm sure there would be great appreciation for a recipe to 
> duplicate an existing Nova Network config in Neutron.  We can debate if that 
> belongs in Grenade should it ever exist…

I was referring to upgrades within projects - in this case Quantum to Neutron.  
I'm assuming that belongs in grenade?


m. 



> dt
> 
> -- 
> 
> Dean Troyer
> dtro...@gmail.com
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] About multihost patch review

[Maru Newby]

On Aug 27, 2013, at 3:27 PM, Tom Fifield  wrote:

> On 27/08/13 15:23, Maru Newby wrote:
>> 
>> On Aug 26, 2013, at 9:39 PM, Yongsheng Gong  wrote:
>> 
>>> First 'be like nova-network' is a merit for some deployments.
>> 
>> I'm afraid 'merit' is a bit vague for me.  Would you please elaborate?
> 
> One area of 'merit' in this area is for migration from nova-network to
> neutron. If there's something exactly analogous to something that
> already exists, its easier to move across.

I apologize for being unclear, but I don't think there is any question that 
neutron needs a multi-host HA capability.  The question is not  one of 
function, but of implementation.  

I don't believe that the design of a feature being proposed for neutron should 
be acceptable simply because it reuses an implementation strategy used by 
nova-network.  Neutron's architecture may allow different decisions to be made, 
and we may have learned from nova-network's example.  In any case, reviewers 
need to understand the 'why' behind design decisions, and it doesn't appear to 
me that there is sufficient documentation justifying the current proposal's 
approach.  Only once we have more information will we be able to make an 
educated decision as to the quality of the proposal.


m.


> 
>> 
>>> second, To allow admin to decide which network will be multihosted at 
>>> runtime will enable the neutron to continue using the current network node 
>>> (dhcp agent) mode at the same time.
>> 
>> If multi-host and non- multi-host networks are permitted to co-exist 
>> (because configuration is per-network), won't compute nodes have to be 
>> allowed to be heterogenous (some multi-host capable, some not)?  And won't 
>> Nova then need to schedule VMs configured with multi-host networks on 
>> compatible nodes?  I don't recall mention of this issue in the blueprint or 
>> design doc, and would appreciate pointers to where this decision was 
>> documented.
>> 
>> 
>>> 
>>> If we force the network multihosted when the configuration enable_multihost 
>>> is true, and then administrator wants to transfer to normal neutron way, 
>>> he/she must modify the configuration item and then restart.
>> 
>> I'm afraid I don't follow - are you suggesting that configuring multi-host 
>> globally will be harder on admins than the change under review?  Switching 
>> to non multi-host under the current proposal involves reconfiguring and 
>> restarting of an awful lot of agents, to say nothing of the db changes.
>> 
>> 
>> m. 
>> 
>> 
>>> 
>>> 
>>> 
>>> On Tue, Aug 27, 2013 at 9:14 AM, Maru Newby  wrote:
>>> 
>>> On Aug 26, 2013, at 4:06 PM, Edgar Magana  wrote:
>>> 
>>>> Hi Developers,
>>>> 
>>>> Let me explain my point of view on this topic and please share your 
>>>> thoughts in order to merge this new feature ASAP.
>>>> 
>>>> My understanding is that multi-host is nova-network HA  and we are 
>>>> implementing this bp 
>>>> https://blueprints.launchpad.net/neutron/+spec/quantum-multihost for the 
>>>> same reason.
>>>> So, If in neutron configuration admin enables multi-host:
>>>> etc/dhcp_agent.ini
>>>> 
>>>> # Support multi host networks
>>>> # enable_multihost = False
>>>> 
>>>> Why do tenants needs to be aware of this? They should just create networks 
>>>> in the way they normally do and not by adding the "multihost" extension.
>>> 
>>> I was pretty confused until I looked at the nova-network HA doc [1].  The 
>>> proposed design would seem to emulate nova-network's multi-host HA option, 
>>> where it was necessary to both run nova-network on every compute node and 
>>> create a network explicitly as multi-host.  I'm not sure why nova-network 
>>> was implemented in this way, since it would appear that multi-host is 
>>> basically all-or-nothing.  Once nova-network services are running on every 
>>> compute node, what does it mean to create a network that is not multi-host?
>>> 
>>> So, to Edgar's question - is there a reason other than 'be like 
>>> nova-network' for requiring neutron multi-host to be configured per-network?
>>> 
>>> 
>>> m.
>>> 
>>> 1: 
>>> http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.h

Re: [openstack-dev] [neutron] Proposing Assaf Muller for the Neutron Core Reviewer Team

[Maru Newby]

+1 from me, long overdue! 


> On May 28, 2015, at 9:42 AM, Kyle Mestery  wrote:
> 
> Folks, I'd like to propose Assaf Muller to be a member of the Neutron core 
> reviewer team. Assaf has been a long time contributor in Neutron, and he's 
> also recently become my testing Lieutenant. His influence and knowledge in 
> testing will be critical to the team in Liberty and beyond. In addition to 
> that, he's done some fabulous work for Neutron around L3 HA and DVR. Assaf 
> has become a trusted member of our community. His review stats place him in 
> the pack with the rest of the Neutron core reviewers.
> 
> I'd also like to take this time to remind everyone that reviewing code is a 
> responsibility, in Neutron the same as other projects. And core reviewers are 
> especially beholden to this responsibility. I'd also like to point out that 
> +1/-1 reviews are very useful, and I encourage everyone to continue reviewing 
> code even if you are not a core reviewer.
> 
> Existing Neutron cores, please vote +1/-1 for the addition of Assaf to the 
> core reviewer team.
> 
> Thanks!
> Kyle
> 
> [1] http://stackalytics.com/report/contribution/neutron-group/180
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] The future of Xen + OVS support

[Maru Newby]

There have been a couple of patches [1] [2] proposed to neutron recently that 
could impact our support for Xen+OVS, but I don’t see an easy way for us to 
validate such changes.  I’m not aware of a 3rd party CI job that runs against 
Xen, and I don’t know of any active contributors able to manually validate 
changes.  Unless this changes, I think we should consider deprecating support 
for Xen in kilo.  We can’t afford to block changes for fear of breaking 
something that nobody seems to care about.

I’m hoping this post serves as a wake-up call to anyone that wants to see 
neutron maintain its support for Xen, and that they will be willing to devote 
resources to ensure that support for it continues.  I’ve also added this issue 
to next week’s irc meeting [3].


Maru

1: https://review.openstack.org/#/c/148969/
2: https://review.openstack.org/#/c/158805 
3: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] The future of Xen + OVS support

[Maru Newby]

> On Mar 3, 2015, at 1:06 PM, Kevin Benton  wrote:
> 
> It might make sense to send this to the users list as well. There may be a 
> large deployment out there with resources willing to at least test changes 
> even if they don't have any upstream development resources.

Good call, I’ve cross-posted to the user list as you suggest.

> 
> On Tue, Mar 3, 2015 at 12:50 PM, Maru Newby  wrote:
> There have been a couple of patches [1] [2] proposed to neutron recently that 
> could impact our support for Xen+OVS, but I don’t see an easy way for us to 
> validate such changes.  I’m not aware of a 3rd party CI job that runs against 
> Xen, and I don’t know of any active contributors able to manually validate 
> changes.  Unless this changes, I think we should consider deprecating support 
> for Xen in kilo.  We can’t afford to block changes for fear of breaking 
> something that nobody seems to care about.
> 
> I’m hoping this post serves as a wake-up call to anyone that wants to see 
> neutron maintain its support for Xen, and that they will be willing to devote 
> resources to ensure that support for it continues.  I’ve also added this 
> issue to next week’s irc meeting [3].
> 
> 
> Maru
> 
> 1: https://review.openstack.org/#/c/148969/
> 2: https://review.openstack.org/#/c/158805
> 3: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] The future of Xen + OVS support

[Maru Newby]

> On Mar 3, 2015, at 1:44 PM, Bob Ball  wrote:
> 
> We certainly care about this but converting the XenServer CI to Neutron+OVS 
> has been hitting a number of issues (notably a few concurrency ones – 
> although there are a few missing features) that we’ve been trying to sort 
> through.
>  
> I am certainly hoping that we’ll have everything stable enough to set up a CI 
> specifically for this combination before the Kilo release and would be very 
> keen not to deprecate XenAPI support in Kilo.
> Certainly even if we don’t make the Kilo release for the CI we will still be 
> pushing for it in L.
>  
> Bob

I guess that answers the question of whether anyone cares enough to work on 
enabling 3rd party CI, and that it may be premature to consider deprecation.  

Until CI is live, though, and in the absence of resources committed to fixing 
breakage when it is detected, I’m not sure it’s reasonable that core reviewers 
have to consider Xen support in determining whether a patch is ready to merge.  
Given that, it’s entirely possible that we’ll end up shipping broken Xen 
support for kilo, especially given the changes coming to rootwrap and ovs 
interaction.  Should we as a community consider that acceptable with the hope 
that fixes will be proposed and backported in a timely-enough fashion?  Or 
maybe we should consider moving Xen support (which is centered on the ovs 
agent) out of the tree so that it can be maintained independent of the 
OpenStack release cycle? 


Maru

> From: Kevin Benton [mailto:blak...@gmail.com] 
> Sent: 03 March 2015 21:06
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] The future of Xen + OVS support
>  
> It might make sense to send this to the users list as well. There may be a 
> large deployment out there with resources willing to at least test changes 
> even if they don't have any upstream development resources.
>  
> On Tue, Mar 3, 2015 at 12:50 PM, Maru Newby  wrote:
> There have been a couple of patches [1] [2] proposed to neutron recently that 
> could impact our support for Xen+OVS, but I don’t see an easy way for us to 
> validate such changes.  I’m not aware of a 3rd party CI job that runs against 
> Xen, and I don’t know of any active contributors able to manually validate 
> changes.  Unless this changes, I think we should consider deprecating support 
> for Xen in kilo.  We can’t afford to block changes for fear of breaking 
> something that nobody seems to care about.
> 
> I’m hoping this post serves as a wake-up call to anyone that wants to see 
> neutron maintain its support for Xen, and that they will be willing to devote 
> resources to ensure that support for it continues.  I’ve also added this 
> issue to next week’s irc meeting [3].
> 
> 
> Maru
> 
> 1: https://review.openstack.org/#/c/148969/
> 2: https://review.openstack.org/#/c/158805
> 3: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
>  
> -- 
> Kevin Benton
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Proposal to add Ihar Hrachyshka as a Neutron Core Reviewer

[Maru Newby]

+1 from me, Ihar has been doing great work and it will be great to have him 
finally able to merge!

> On Mar 4, 2015, at 11:42 AM, Kyle Mestery  wrote:
> 
> I'd like to propose that we add Ihar Hrachyshka to the Neutron core reviewer 
> team. Ihar has been doing a great job reviewing in Neutron as evidence by his 
> stats [1]. Ihar is the Oslo liaison for Neutron, he's been doing a great job 
> keeping Neutron current there. He's already a critical reviewer for all the 
> Neutron repositories. In addition, he's a stable maintainer. Ihar makes 
> himself available in IRC, and has done a great job working with the entire 
> Neutron team. His reviews are thoughtful and he really takes time to work 
> with code submitters to ensure his feedback is addressed.
> 
> I'd also like to again remind everyone that reviewing code is a 
> responsibility, in Neutron the same as other projects. And core reviewers are 
> especially beholden to this responsibility. I'd also like to point out and 
> reinforce that +1/-1 reviews are super useful, and I encourage everyone to 
> continue reviewing code across Neutron as well as the other OpenStack 
> projects, regardless of your status as a core reviewer on these projects.
> 
> Existing Neutron cores, please vote +1/-1 on this proposal to add Ihar to the 
> core reviewer team.
> 
> Thanks!
> Kyle
> 
> [1] http://stackalytics.com/report/contribution/neutron-group/90
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][neutron] Moving network api test development to Neutron repo

[Maru Newby]

tl;dr; As per a discussion in Paris [1], development of Neutron's
API tests is moving from the Tempest repo to the Neutron repo.
If you are developing API tests for Neutron in Tempest, please be
advised that, effective immediately, your efforts should be
directed towards the Neutron repo.

The current set of Neutron API tests in Tempest has been
copied (along with supporting infrastructure) to
neutron/tests/tempest [1].  Tests in this path are run as part of
the neutron-dsvm-api job, which will shortly be gating [2].  Test
changes that previously targeted the tempest/api/network path in
the Tempest repo should target neutron/tests/tempest/network/api
in the Neutron repo until further notice.

Automatic conversion from a Tempest change to a Neutron change is
possible:

 - cd [path to neutron repo]
 - ./tools/copy_api_tests_from_tempest.sh [path to tempest working directory]

As per the Tempest guidelines for test removal [3], the tests
currently in Tempest will remain in Tempest and continue to run
as part of the existing jobs until we can target tests in the
Neutron repo against stable branches and enable the use of the
in-repo tests by defcore/refstack.

Finally, guidelines for API test development in the Neutron repo are
in the works and will be proposed shortly.  The guidelines will
define policy intended to protect against backwards compatible
changes to our API.

Thanks,


Maru

1: 
https://etherpad.openstack.org/p/kilo-crossproject-move-func-tests-to-projects
2: https://github.com/openstack/neutron/tree/master/neutron/tests/tempest
3: https://review.openstack.org/#/c/164886
4: https://wiki.openstack.org/wiki/QA/Tempest-test-removal


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra] What do people think of YAPF (like gofmt, for python)?

[Maru Newby]

I am excited by the release of YAPF [1], a gofmt-like too for python.  I think 
it has the potential to simplify style enforcement, and as much as I appreciate 
our current hacking checks, I’d be much happier not requiring developers to 
manually conform to them.  Maybe we can consider automation in a manner similar 
to that employed by the go codereview tool [2]?

Cheers,


Maru

1: https://github.com/google/yapf
2: https://godoc.org/golang.org/x/review/git-codereview#hdr-Gofmt
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] What do people think of YAPF (like gofmt, for python)?

[Maru Newby]

> On Mar 25, 2015, at 4:22 PM, Monty Taylor  wrote:
> 
> On 03/25/2015 05:50 PM, Maru Newby wrote:
>> I am excited by the release of YAPF [1], a gofmt-like too for python.
>> I think it has the potential to simplify style enforcement, and as
>> much as I appreciate our current hacking checks, I’d be much happier
>> not requiring developers to manually conform to them.  Maybe we can
>> consider automation in a manner similar to that employed by the go
>> codereview tool [2]?
> 
> I played with it for a few minutes and although it's configurable, it's
> still pretty limited in terms of expressiveness.
> 
> That said - although I do appreciate the theory of auto-formatting
> (seriously, one less thing, right?) I think it would be problematic for
> us. You can't ship git hooks in a git repo, so we can't help our users
> know to run it before pushing. In a world where getting set up in
> openstack is already non-trivial, I think requiring 2500 developers to
> add a new git hook to every repo that they do something with would be a
> bit of a disaster. When you combine that with the people who won't know,
> will make a patch, send it up, and have it rejected --- oy. Chaos.
> 
> git review is used by a ton of people who write in non-python. I think
> adding openstack-specific style enforcement to it would make it way less
> generally useful.
> 
> In general, if you find it interesting, there's certainly nothing wrong
> with tracking it and poking at it from time to time. But I honestly
> think that the logistical issue is pretty large, and one that the payoff
> from solving would not be worth the effort.

I agree with points raised by you and jeblair regarding the
potential for problems in applying auto-formatting without
developer intervention.  I apologize if my use of the word
'automation' muddied the waters, as the go codereview tool's
support of the 'gofmt' command is limited to explicit invocation
rather than being triggered in a hook.  It doesn't appear
acceptable for git-review to be granted similar capability, but
really that is a nicety.  Individual projects can as easily
provide their own style definitions, include YAPF as a test
dependency, and document how to invoke it.

I think there is value in extending our current solution of
automated style enforcement to provide a component that can be
manually invoked to auto-format to the enforced style.  I don't
see virtue in applying manual effort to maintaining code style.
YAPF may not be ready for our use just yet, but I think it's
worth pursuing. 


Maru


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The Evolution of core developer to maintainer?

[Maru Newby]

> On Apr 1, 2015, at 6:09 AM, Jeremy Stanley  wrote:
> 
> And here is the crux of the situation, which I think bears
> highlighting. These "empowered groups" are (or at least started out
> as) nothing more than an attempt to map responsibilities onto the
> ACLs available to our projects in the tools we use to do the work.
> Coming up with some new pie-in-the-sky model of leadership hierarchy
> is an interesting thought exercise, but many people in this
> discussion are losing sight of the fact that the model we have is
> determined to a great extent by the tools we use. Change the tools
> and you may change the model, but changing the model doesn't
> automatically change the tools to support it (and those proposing a
> new model need to pony up the resources to implement it in
> _reality_, not just in _thought_).

> Responsibilities not tied to specific controls in our tools do exist
> in abundance, but they tend to be more fluid and ad-hoc because in
> most cases there's been no need to wrap authorization/enforcement
> around them. What I worry is happening is that as a community we're
> enshrining the arbitrary constructs which we invented to be able to
> configure our tools sanely. I see this discussion as an attempt to
> recognize those other responsibilities as well, but worry that
> creation of additional unnecessary authorization/enforcement process
> will emerge as a "solution" and drive us further into pointless
> bureaucracy.

Given how important trust and relationships are to the
functioning of individual projects, I think we’re past the point
where we should allow our tooling to be the limiting factor in
how we structure ourselves.  Do we need finer-grained permissions
in gerrit to enable something like subtree maintainers?  I don't
believe we do.  In large projects like Neutron, there is no such
thing as someone who knows everything anymore, so we all need to
be aware of our limitations and know not to merge things we don't
understand without oversight from those of our peers that do.
Responsibility in this case could be subject to social rather than
tool-based oversight.


Maru
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The Evolution of core developer to maintainer?

[Maru Newby]

> On Apr 1, 2015, at 2:52 AM, Thierry Carrez  wrote:
> 
> - Some people are core reviewers and maintainers (or "drivers", to reuse
> the openstack terminology we already have for that)
> - Some people are core reviewers only (because they can't commit 90% of
> their work time to work on project priorities)
> - Some people are maintainers/drivers only (because their project duties
> don't give them enough time to also do reviewing)
> - Some people are casual developers / reviewers (because they can't
> spend more than 30% of their day on project stuff)
> 
> All those people are valuable. Simply renaming "core reviewers" to
> "maintainers" (creating a single super-developer class) just excludes
> valuable people.

I hear that you believe that the proposal to rename 'core
reviewer' to 'maintainer' in Neutron was intended to entrench
privilege.  Nothing could be further from the truth - it was
actually intended to break it down.

As per Joe’s recent reply, ‘drivers’ in Nova have to be core
reviewers.  This is true in Neutron as well.  I think a more
accurate taxonomy, at least in Neutron, is the following:

- Everyone that participates in the project is a 'contributor'

- Some contributors are 'core reviewers' - members of the team
  with merge rights on a primary repo and a responsibility to
  actively review for that repo.

- Some core reviewers are 'drivers' - members of a team with
  merge rights on the spec repo and a responsibility to actively
  review for that repo.

This is obviously a gross simplification, but it should serve for
what I'm trying to communicate.  Many of us in the Neutron
community find this taxonomy restrictive and not representative
of all the work that makes the project possible.  Worse, 'cores'
are put on a pedastal, and not just in the project.  Every summit
a 'core reviewer dinner' is held that underscores the
glorification of this designation.  By proposing to rename 'core
reviewer' to 'maintainer' the goal was to lay the groundwork for
broadening the base of people whose valuable contribution could
be recognized.  The goal was to recognize not just review-related
contributors, but also roles like doc/bug/test czar and cross-project
liaison.  The statue of the people filling these roles today is less 
if they are not also ‘core’, and that makes the work less attractive 
to many.

Given the TC's apparent mandate to define the organizational
taxonomy that a project like Neutron is allowed to use, I would
ask you and your fellow committee members to consider addressing
the role that the current taxonomy plays in valuing reviewing
ahead of other forms of contribution.  It provides disincentive against
other forms of contribution, since they aren’t recognized on an equal 
footing, and I think this needs to change if we want to ensure the 
long-term viability of projects like Neutron (if not OpenStack as a whole).


Maru
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The Evolution of core developer to maintainer?

[Maru Newby]

> On Apr 1, 2015, at 1:47 PM, Jeremy Stanley  wrote:
> 
> On 2015-04-01 12:00:53 -0700 (-0700), Maru Newby wrote:
>> Given how important trust and relationships are to the functioning
>> of individual projects, I think we’re past the point where we
>> should allow our tooling to be the limiting factor in how we
>> structure ourselves.
> 
> I'm definitely not suggesting that either, merely pointing out that
> if you have an ACL which, for example, defines the set of people
> able to push a particular button then it's helpful to have a term
> for that set of people. As soon as you start to conflate that
> specific permission with other roles and responsibilities then the
> term for it gets overloaded. To me a "core reviewer" is just that:
> people with accounts in the .*-core Gerrit groups granted the
> ability to push a review button indicating that a proposed change is
> suitable to merge. Whether or not those same people are also
> afforded permissions outside that system is orthogonal.

I find your perspective on the term ‘core reviewer’ to be interesting indeed,
and for me it underscores the need to consider whether using the term
outside of gerrit is justified.


Maru


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The Evolution of core developer to maintainer?

[Maru Newby]

> On Apr 2, 2015, at 3:26 AM, Thierry Carrez  wrote:
> 
> Maru Newby wrote:
>> [...] Many of us in the Neutron
>> community find this taxonomy restrictive and not representative
>> of all the work that makes the project possible.
> 
> We seem to be after the same end goal. I just disagree that renaming
> "core reviewers" to "maintainers" is a positive step toward that goal.
> 
>> Worse, 'cores'
>> are put on a pedastal, and not just in the project.  Every summit
>> a 'core reviewer dinner' is held that underscores the
>> glorification of this designation.
> 
> I deeply regret that, and communicated to the sponsor holding it the
> problem with this "+2 dinner" the very first time it was held. FWIW it's
> been renamed to "VIP dinner" and no longer limited to core reviewers,
> but I'd agree with you that the damage was already done.
> 
>> By proposing to rename 'core
>> reviewer' to 'maintainer' the goal was to lay the groundwork for
>> broadening the base of people whose valuable contribution could
>> be recognized.  The goal was to recognize not just review-related
>> contributors, but also roles like doc/bug/test czar and cross-project
>> liaison.  The statue of the people filling these roles today is less 
>> if they are not also ‘core’, and that makes the work less attractive 
>> to many.
> 
> That's where we disagree. You see renaming "core reviewer" to
> "maintainer" has a way to recognize a broader type of contributions. I
> see it as precisely resulting in the opposite.
> 
> Simply renaming "core reviewers" to "maintainers" just keeps us using a
> single term (or class) to describe project leadership. And that class
> includes +2 reviewing duties. So you can't be a maintainer if you don't
> do core reviewing. That is exclusive, not inclusive.

The important part of my statement above was ‘lay the groundwork for’.
We were intended to change the name as a _precursor_ to changing the
role itself to encompass more than just those with +2 rights.  Nobody
in their right mind would assume that changing the name by itself could
fix the situation, but we thought it would be a good signal as to our
intent to broaden the scope of recognized contribution.


> What we need to do instead is reviving the "drivers" concept (we can
> rename it "maintainers" if you really like that term), separate from the
> "core reviewers" concept. One can be a project "driver" and a "core
> reviewer". And one can be a project "driver" *without* being a "core
> reviewer". Now *that* allows to recognize all valuable contributions,
> and to be representative of all the work that makes the project possible.

As Joe and I have said, Nova and Neutron already have drivers teams and 
they fill a different role from what you are suggesting.  Can you think of a 
more
appropriate name that isn’t already in use for what you are proposing?


Maru
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev