Re: [openstack-dev] [Fuel] Python code in fuel-library

[Sebastian Kalinowski]

Hi,

I've created a patchset that introduces a script to run Python tests in
fuel-library [1] and a
Jenkins job [2]

I also would like to backport those tests to stable branches to assure that
any fix backported to stable branches will be also
checked.

Best,
Sebastian

[1] https://review.openstack.org/#/c/157319/
[2] https://review.fuel-infra.org/#/c/3810/

2015-02-18 16:01 GMT+01:00 Aleksandr Didenko :

> Hi,
>
> I agree that we need a better testing for python tasks/code. There should
> be no problems adding py.test tests into fuel-library CI, we already have
> one [1] up and running. So I'm all in and ready to help with such testing
> implementation.
>
> [1] https://fuel-jenkins.mirantis.com/job/fuellib_tasks_graph_check/
>
> Regards,
> Aleksandr
>
> On Wed, Feb 18, 2015 at 4:02 PM, Vladimir Kuklin 
> wrote:
>
>> Hi, Seb
>>
>> Very fair point, thank you. We need to add this to our jobs for unittests
>> run and syntax check. I am adding Aleksandr Didenko into the loop as he is
>> currently working on the similar task.
>>
>> On Wed, Feb 18, 2015 at 4:53 PM, Jay Pipes  wrote:
>>
>>> On 02/18/2015 04:57 AM, Sebastian Kalinowski wrote:
>>>
 Hello Fuelers,

 There is more and more Python code appearing in fuel-library [1] that is
 used in our Puppet manifests. Now, with introduction of Granular
 Deployment feature it could appear more often as
 writing some tasks as a Python script is a nice option.

 First problem that I see is that in some cases this code is getting
 merged without a positive review from a Python developer from Fuel team.
 My proposition of the solution is simple:
 fuel-library core reviewers shouldn't merge such code if there is no a
 +1 from a Python developer from fuel-core group [2].

 Second problem is that there are no automatic tests for this code.
 Testing it manually and by running deployment when that code is used is
 not enough since those scripts could be quite large and complicated and
 some of them are executed in specific situations so it is hard for
 reviewers to check how they will work.
 In fuel-library we already have tests for Puppet modules: [3].
 I suggest that we should introduce similar checks for Python code:
   - there will be one global 'test-requirements.txt' file (if there will
 be a need to, we could introduce more granular split, like per module)
   - py.test [4] will be used as a test runner
   - (optional, but advised) flake8+hacking checks [5] (could be limited
 to just run flake8/pyflakes checks)

 Looking forward to your opinions on those two issues.

>>>
>>> Hi Seba,
>>>
>>> All those suggestions look fine to me. I'd also add to improve the
>>> documentation on how to write and run Python tests to help out those
>>> developers who are not as familiar with Python as Ruby or other languages.
>>>
>>> Best,
>>> -jay
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
>>> unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Yours Faithfully,
>> Vladimir Kuklin,
>> Fuel Library Tech Lead,
>> Mirantis, Inc.
>> +7 (495) 640-49-04
>> +7 (926) 702-39-68
>> Skype kuklinvv
>> 45bk3, Vorontsovskaya Str.
>> Moscow, Russia,
>> www.mirantis.com 
>> www.mirantis.ru
>> vkuk...@mirantis.com
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] The strange case of osapi_compute_unique_server_name_scope

[Matthew Booth]

On 20/02/15 20:15, Andrew Bogott wrote:
> On 2/20/15 9:06 AM, Mike Dorman wrote:
>> I can report that we do use this option (‘global' setting.)  We have to
>> enforce name uniqueness for instances’ integration with some external
>> systems (namely AD and Spacewalk) which require unique naming.
>>
>> However, we also do some external name validation which I think
>> effectively enforces uniqueness as well.  So if this were deprecated, I
>> don’t know if it would directly affect us for our specific situation.
>>
>> Other operators, anyone else using
>> osapi_compute_unique_server_name_scope?
> I use it!  And, in fact, added it in the first place :(
> 
> I have no real recall of what we concluded when originally discussing
> the associated race.  The feature is useful to me and I'd love it if it
> could be moved into the db to fix the race, but I concede that it's a
> pretty big can o' worms, so if no one else cries out in pain I can live
> with it being deprecated.

Ok, with 2 identified users I'm going to abandon my patch to deprecate
with no replacement. My current plan is to leave the race there with a
comment in the code and the config documentation. Perhaps we can fix
this properly at some point when we get the online schema changes, but
for the moment it seems like a lot of complication for a relatively
small problem.

Do you use the global or project scope, btw?

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Raphael Glon]

On 02/19/2015 12:45 PM, Richard W.M. Jones wrote:

On Wed, Feb 18, 2015 at 07:23:52PM +0100, Raphael Glon wrote:

I entcountered a similar case more recently on powerkvm 2.1.0
(defect with the libguestfs)

What's the actual bug?  We've worked hard, with IBM, to make
libguestfs work on POWER 7 and POWER 8 systems.  I have full access to
those systems through Red Hat.  If there's a new bug I'm sure we'll be
able to fix it.

Rich.


Hi, thank you for all your answers.

Not saying there are "actual" bugs (anyway I'm stuck here because i 
would need to find time+environment to recheck all/reproduce) -> i 
haven't even deployed juno on pkvm yet


We've talked with ibm (and they have likely been working with you) and 
they are really responsive in fixing defects with their distribution


We've entcountered two problems with powerkvm regarding nova + libguestfs.

1. since pkvm 2.1.x is forked from a Fedo 19, we fell back to this Red 
Hat bug you fixed regarding the attach method


Note that one of the workaround proposed was

sudo sysctl -w fs.protected_hardlinks=0 + common user nova/qemu


-> Not a specialist here, but seems like to be able to use libguestfs to 
avoid "potential" issues with fuse mounts, we open other "potential" 
holes somewhere else


2. because pkvm 2.1.x is forked from fedo 19 it embeds rather old 
versions of libguestfs and libvirt.


We also entcountered the following issue(as you see from the dates, it 
is rather "old"). About this, i was not perfectly accurate saying it was 
a libguestfs defect, it was a pkvm defect embedding an old version of 
libguestfs and anyway it also has been fixed quickly


http://paste.ubuntu.com/8465699/

Sep 30 14:25:33 host-power8-1 libvirtd[15894]: Domain id=2 
name='guestfs-xv6mh1nvhr17ktj6' 
uuid=341b09bc-6583-4b72-9df8-dc9b18116303 is tainted: custom-argv
Sep 30 14:25:33 host-power8-1 libvirtd[15894]: Unable to read from 
monitor: Connection reset by peer
Sep 30 14:25:33 host-power8-1 kernel: [  878.869394] 
qemu-system-ppc[16250]: unhandled signal 11 at 00d8 nip 
3070c0ac lr 3070bff4 code 30001
Sep 30 14:25:33 host-power8-1 libvirtd[15894]: cannot lookup default 
selinux label for /tmp/libguestfsrOhcjP/console.sock
Sep 30 14:25:33 host-power8-1 libvirtd[15894]: cannot lookup default 
selinux label for /tmp/libguestfsrOhcjP/guestfsd.sock
Sep 30 14:25:33 host-power8-1 libvirtd[15894]: cannot lookup default 
selinux label for /var/tmp/.guestfs-0/kernel.16152
Sep 30 14:25:33 host-power8-1 libvirtd[15894]: cannot lookup default 
selinux label for /var/tmp/.guestfs-0/initrd.16152



3. Had no time to investigate on this but when using libguestfs with 
nova, the ghost was not always destroyed after file injection. 
Sometimes, you could find an instance spawned with the libguestfs ghost 
still running in the same time. Anyway, I've got no logs to detailthis 
issue, i'll try to get some one day


So summing up this patch was about:

File injection with libguestfs not working in some specific environment 
(dist pkvm 2.1.0 + libguestfs pkvm packaged version + nova havana) and i 
supposed i was not the only one concerned


On our side we had to temporarily disable file injection using libguestfs

Since nova still considers fuse mounts as acceptableit would have been 
practical if, at the time, it had been flexible on the fact to use or 
not libguestfs when this one is installed. (By the way, correct me if 
wrong, but there are no current open issues with fuse mounts, and if 
there are, why is it still proposed in nova ? would even say this is the 
default/most used method because there is no dist considering libguestfs 
is a dependency for the nova-compute package)


Get your reluctance. Giving up with the patch.

regards

raphael


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bp serial-ports *partly* implemented?

[Sahid Orentino Ferdjaoui]

On Fri, Feb 20, 2015 at 06:03:46PM +0100, Markus Zoeller wrote:
> It seems to me that the blueprint serial-ports[1] didn't implement
> everything which was described in its spec. If one of you could have a 
> look at the following examples and help me to understand if these 
> observations are right/wrong that would be great.
> 
> Example 1:
> The flavor provides the extra_spec "hw:serial_port_count" and the image
> the property "hw_serial_port_count". This is used to decide how many
> serial devices (with different ports) should be defined for an instance.
> But the libvirt driver returns always only the *first* defined port 
> (see method "get_serial_console" [2]). I didn't find anything in the 
> code which uses the other defined ports.

The method you are referencing [2] is used to return the first well
binded and not connected port in the domain.

When defining the domain '{hw_|hw:}serial_port_count' are well take
into account as you can see:

   
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3702

(The method looks to have been refactored and include several parts
not related to serial-console)

> Example 2:
> "If a user is already connected, then reject the attempt of a second
> user to access the console, but have an API to forceably disconnect
> an existing session. This would be particularly important to cope
> with hung sessions where the client network went away before the
> console was cleanly closed." [1]
> I couldn't find the described API. If there is a hung session one cannot
> gracefully recover from that. This could lead to a bad UX in horizons
> serial console client implementation[3].

This API is not implemented, I will see what I can do on that
part. Thanks for this.

> [1] nova bp serial-ports;
> 
> https://github.com/openstack/nova-specs/blob/master/specs/juno/implemented/serial-ports.rst
> [2] libvirt driver; return only first port; 
> 
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2518
> [3] horizon bp serial-console; 
> https://blueprints.launchpad.net/horizon/+spec/serial-console
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Daniel P. Berrange]

On Mon, Feb 23, 2015 at 11:08:31AM +0100, Raphael Glon wrote:
> On 02/19/2015 12:45 PM, Richard W.M. Jones wrote:
> >On Wed, Feb 18, 2015 at 07:23:52PM +0100, Raphael Glon wrote:
> >>I entcountered a similar case more recently on powerkvm 2.1.0
> >>(defect with the libguestfs)
> >What's the actual bug?  We've worked hard, with IBM, to make
> >libguestfs work on POWER 7 and POWER 8 systems.  I have full access to
> >those systems through Red Hat.  If there's a new bug I'm sure we'll be
> >able to fix it.
> >
> >Rich.
> >
> Hi, thank you for all your answers.
> 
> Not saying there are "actual" bugs (anyway I'm stuck here because i would
> need to find time+environment to recheck all/reproduce) -> i haven't even
> deployed juno on pkvm yet
> 
> We've talked with ibm (and they have likely been working with you) and they
> are really responsive in fixing defects with their distribution
> 
> We've entcountered two problems with powerkvm regarding nova + libguestfs.
> 
> 1. since pkvm 2.1.x is forked from a Fedo 19, we fell back to this Red Hat
> bug you fixed regarding the attach method
> 
> Note that one of the workaround proposed was
> 
> sudo sysctl -w fs.protected_hardlinks=0 + common user nova/qemu
> 
> 
> -> Not a specialist here, but seems like to be able to use libguestfs to
> avoid "potential" issues with fuse mounts, we open other "potential" holes
> somewhere else

The alternative Nova implementation is *not* using fuse, it is using real
mounts on the host FS. This is not a potential issue, it is an *actual*
issue. There have been bugs in Linux filesystem drivers, including ext4,
that would have allowed a malicous kernel image to crash and/or exploit
the host kernel if mounted.

  http://libguestfs.org/guestfs.3.html#security-of-mounting-filesystems

The libguestfs architecture is explicitly designed such that any security
critical tasks take place inside an unprivileged KVM guest. So unless Nova
is using libguestfs in a broken way, the security of libguestfs is effectively
equivalent to the security of KVM in general. This is a faaar better security
architecture design


> 2. because pkvm 2.1.x is forked from fedo 19 it embeds rather old versions
> of libguestfs and libvirt.

Fedora 19 is end of life so not really relevant any more as a target.
If there are bugs you find in current versions of Fedora please file
bugs so they can be addressed.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Raphael Glon]

On 02/23/2015 11:23 AM, Daniel P. Berrange wrote:

The alternative Nova implementation is*not*  using fuse, it is using real
mounts on the host FS. This is not a potential issue, it is an*actual*
issue. There have been bugs in Linux filesystem drivers, including ext4,
that would have allowed a malicous kernel image to crash and/or exploit
the host kernel if mounted.

   http://libguestfs.org/guestfs.3.html#security-of-mounting-filesystems


Ok noted -> so why is losetup or qemu-nbd still proposed by nova and 
still the default method ?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Raphael Glon]

On 02/23/2015 11:23 AM, Daniel P. Berrange wrote:

Fedora 19 is end of life so not really relevant any more as a target.

 powerkvm 2.1.x is not eol

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Daniel P. Berrange]

On Mon, Feb 23, 2015 at 11:54:40AM +0100, Raphael Glon wrote:
> On 02/23/2015 11:23 AM, Daniel P. Berrange wrote:
> >Fedora 19 is end of life so not really relevant any more as a target.
>  powerkvm 2.1.x is not eol

Well Fedora 19 is end of life, so if powerkvm is using Fedora 19, then
it is running on software which has no further update support. So if
powerkvm team find bugs in it, they'll have to take responsibility for
fixing them, as Fedora maintainers won't be doing any fixes for it.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Daniel P. Berrange]

On Mon, Feb 23, 2015 at 11:52:29AM +0100, Raphael Glon wrote:
> On 02/23/2015 11:23 AM, Daniel P. Berrange wrote:
> >The alternative Nova implementation is*not*  using fuse, it is using real
> >mounts on the host FS. This is not a potential issue, it is an*actual*
> >issue. There have been bugs in Linux filesystem drivers, including ext4,
> >that would have allowed a malicous kernel image to crash and/or exploit
> >the host kernel if mounted.
> >
> >   http://libguestfs.org/guestfs.3.html#security-of-mounting-filesystems
> 
> Ok noted -> so why is losetup or qemu-nbd still proposed by nova and still
> the default method ?

Libguestfs method takes priority if it is installed on the host, but
the legacy code still exists for benefit of existing deployed setups
and drivers which don't have qemu/kvm available, eg LXC containers.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Raphael Glon]

On 02/23/2015 11:23 AM, Daniel P. Berrange wrote:

The alternative Nova implementation is*not*  using fuse,


yeah you're right -> read your page too quickly 
https://www.berrange.com/tags/nbd/


added the term fuse where it should not have been
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Separating granular tasks validator

[Kamil Sambor]

Thank you guys for response.
There is no cons, so we migrate validation to separate repo.

Best regards,
Kamil Sambor

On Wed, Feb 18, 2015 at 9:34 AM, Evgeniy L  wrote:

> +1 to extract validators for granular deployment tasks
>
> Dmitry, do you mean that we should create some cli to generate graph
> picture? Or just make it as a module and then use it in Nailgun?
>
> Thanks,
>
> On Tue, Feb 17, 2015 at 4:31 PM, Dmitriy Shulyak 
> wrote:
>
>> +1 for separate tasks/graph validation library
>>
>> In my opinion we may even migrate graph visualizer to this library, cause
>> it is most usefull during development and to demand installed fuel with
>> nailgun feels a bit suboptimal
>>
>>
>> On Tue, Feb 17, 2015 at 12:58 PM, Kamil Sambor 
>> wrote:
>>
>>> Hi all,
>>>
>>> I want to discuss separating validation from our repositories to one. On
>>> this moment in fuel we have validation for  granular deployment tasks in 3
>>> separate repositories so we need to maintain very similar code in all of
>>> them. New idea that we discussed with guys assumes to keep this code in one
>>> place. Below are more details.
>>>
>>> Schema validator should be in separate repo, we will install validator
>>> in fuel-plugin, fuel-lib, fuel-nailgun. Validator should support versions
>>> (return schemas and validate them for selected version).
>>> Reasons why we should have validation in all three repositories:
>>> nailgun: we need validation in api because we are able to send our own
>>> tasks to nailgun and execute them (now we validate type of tasks in
>>> deployment graph and  during installation of plugin)
>>> fuel-library: we need to check if tasks schema is correct defined in
>>> task.yaml files and if tasks not create cycles (actually we do both things)
>>> fuel-plugin: we need to check if defined tasks are supported by selected
>>> version of nailgun (now we check if task type are the same with hardcoded
>>> types in fuel-plugin, we not update this part since a while and now there
>>> are only 2 type of tasks: shell and puppet)
>>> With versioning we shouldn't have conflicts between nailgun
>>> serialization and fuel-plugin because plugin will be able to use schemas
>>> for specified version of nailgun.
>>>
>>> As a core reviewers of repositories we should keep the same reviewers as
>>> we have in fuel-core.
>>>
>>> How validator should looks like:
>>> separate repo, to install using pip
>>> need to return correct schema for selected version of fuel
>>> should be able to validate schema for selected version and ignore
>>> selected fields
>>> validate graph from selected tasks
>>>
>>> Pros and cons of this solution:
>>> pros:
>>> one place to keep validation
>>> less error prone - we will eliminate errors caused by not updating one
>>> of the repos, also it will be easy to test if changes are correct and
>>> compatible with all repos
>>> easier to develop (less changes in cases when we add new type of task or
>>> we change schemas of tasks - we edit just one place)
>>> easy distribution of code between repositories and easy to use by
>>> external developers
>>> cons:
>>> new repository that needs to be managed (and included into CI/QA/release
>>> cycle)
>>> new dependency for fuel-library, fuel-web, fuel-plugins (fuel plugin
>>> builder) of which developer need to be aware of
>>>
>>> Please comments and give opinions.
>>>
>>> Best regards,
>>> Kamil Sambor
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic "The Heat Orchestration Template Builder: A demonstration"

[Aggarwal, Nikunj]

Hi Angus,

I am working Timur and Drago on HOT builder. I am using this -> 
https://github.com/rackerlabs/hotbuilder as the base and made some improvements 
over the existing code and wish to give a demonstration on how easy it is to 
create a HOT template from Horizon.

Regards,
Nikunj

From: Angus Salkeld [mailto:asalk...@mirantis.com]
Sent: Monday, February 23, 2015 4:52 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic 
"The Heat Orchestration Template Builder: A demonstration"

On Sat, Feb 21, 2015 at 4:21 AM, Aggarwal, Nikunj 
mailto:nikunj.aggar...@hp.com>> wrote:
Hi,

I have submitted  presentations for Openstack L summit :


The Heat Orchestration Template Builder: A 
demonstration



Hi
Nice to see work on a HOT builder progressing, but..
are you planning on integrating this with the other HOT builder efforts?
Is the code public (link)?

This is more of a framework to make these easier to build:
https://github.com/stackforge/merlin
https://wiki.openstack.org/wiki/Merlin

Timur (who works on Merlin) is working with Rackers to build this upstream - I 
am not sure on the progress.
https://github.com/rackerlabs/hotbuilder
https://github.com/rackerlabs/foundry

It would be nice if we could all work together (I am hoping you are already)?
Hopefully some of the others that are working on this can chip in and say where 
they
are.

-Angus


Please cast your vote if you feel it is worth for presentation.

Thanks & Regards,
Nikunj


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][vmware][ironic] Configuring active/passive HA Nova compute

[Matthew Booth]

On 20/02/15 11:48, Matthew Booth wrote:
> Gary Kotton came across a doozy of a bug recently:
> 
> https://bugs.launchpad.net/nova/+bug/1419785
> 
> In short, when you start a Nova compute, it will query the driver for
> instances and compare that against the expected host of the the instance
> according to the DB. If the driver is reporting an instance the DB
> thinks is on a different host, it assumes the instance was evacuated
> while Nova compute was down, and deletes it on the hypervisor. However,
> Gary found that you trigger this when starting up a backup HA node which
> has a different `host` config setting. i.e. You fail over, and the first
> thing it does is delete all your instances.
> 
> Gary and I both agree on a couple of things:
> 
> 1. Deleting all your instances is bad
> 2. HA nova compute is highly desirable for some drivers
> 
> We disagree on the approach to fixing it, though. Gary posted this:
> 
> https://review.openstack.org/#/c/154029/
> 
> I've already outlined my objections to this approach elsewhere, but to
> summarise I think this fixes 1 symptom of a design problem, and leaves
> the rest untouched. If the value of nova compute's `host` changes, then
> the assumption that instances associated with that compute can be
> identified by the value of instance.host becomes invalid. This
> assumption is pervasive, so it breaks a lot of stuff. The worst one is
> _destroy_evacuated_instances(), which Gary found, but if you scan
> nova/compute/manager for the string 'self.host' you'll find lots of
> them. For example, all the periodic tasks are broken, including image
> cache management, and the state of ResourceTracker will be unusual.
> Worse, whenever a new instance is created it will have a different value
> of instance.host, so instances running on a single hypervisor will
> become partitioned based on which nova compute was used to create them.
> 
> In short, the system may appear to function superficially, but it's
> unsupportable.
> 
> I had an alternative idea. The current assumption is that the `host`
> managing a single hypervisor never changes. If we break that assumption,
> we break Nova, so we could assert it at startup and refuse to start if
> it's violated. I posted this VMware-specific POC:
> 
> https://review.openstack.org/#/c/154907/
> 
> However, I think I've had a better idea. Nova creates ComputeNode
> objects for its current configuration at startup which, amongst other
> things, are a map of host:hypervisor_hostname. We could assert when
> creating a ComputeNode that hypervisor_hostname is not already
> associated with a different host, and refuse to start if it is. We would
> give an appropriate error message explaining that this is a
> misconfiguration. This would prevent the user from hitting any of the
> associated problems, including the deletion of all their instances.

I have posted a patch implementing the above for review here:

https://review.openstack.org/#/c/158269/

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] db-level locks, non-blocking algorithms, active/active DB clusters and IPAM

[Salvatore Orlando]

Lazy-Stacker summary:
I am doing some work on Neutron IPAM code for IP Allocation, and I need to
found whether it's better to use db locking queries (SELECT ... FOR UPDATE)
or some sort of non-blocking algorithm.
Some measures suggest that for this specific problem db-level locking is
more efficient even when using multi-master DB clusters, which kind of
counters recent findings by other contributors [2]... but also backs those
from others [7].

The long and boring thing:

In the past few months there's been a fair amount of discussion concerning
the use of locking queries (ie: SELECT ... FOR UPDATE) in various OpenStack
projects, especially in connection with the fact that this kind of queries
is likely to trigger write set certification failures in synchronous db
replication engines such as MySQL Galera - as pointed out in [1] and
thoroughly analysed in [2].
Neutron, in particular, uses this construct often - and the correctness of
the whole IPAM logic depends on it. On this regard Neutron is also guilty
of not implementing any sort of retry mechanism. Eugene, Rossella, and
others have been working towards fixing this issue. Some examples of their
work can be found at [3] and [4].

However, techniques such as "compare and swap" described in [2] can be used
for implementing non-blocking algorithms which avoid at all the write-set
certification issue.
A neutron example of such technique can be found in [5]. However, a bug in
this process found by Attila [6], and the subsequent gerrit discussion [7],
raised further questions on locking vs non-blocking solutions.
At this stage, we know that using database-level locks triggers write-set
certification failures in synchronous multi master modes. This failures are
reported as deadlock errors. Such errors can be dealt with retrying the
transaction. However, this is claimed as not efficient in [2], whereas
Attila in [7] argues the opposite.

As I'm rewriting Neutron's IPAM logic as a part of [8], which heavily
relies on this construct, I decided to spend some time to put everything to
the test.
To this aim I devised 3 algorithms.

A-1) The classic one which locks the availability ranges for a subnet until
allocation is complete, augmented with a simple and brutal retry mechanism
[9].
A-2) A wait-free 2 step process, in which the first one creates the IP
allocation entry, and the second completes it by adjusting availability
ranges [10]. Primary key violations will trigger retries.
A-3) A wait-free, lock-free 3 step process [11], which adopts
compare-and-swap [2] and the same election process as the bully algorithm
[12].

Unfortunately neutron does not create IP records for every address in a
subnet's CIDR - this would be fairly bad for IPv6 networks. Therefore we
cannot simply apply the simple CAS technique introduced in [2].

I then tested them under arbitrary concurrent load from multiple workers.
The tests were performed first on a single node backend, and then a 3-node
mysql Galera cluster, installed and configured using the official
documentation [13].
The single-node tests showed, as it might have been expected, that the
algorithm leveraging db-level locks is way more efficient [14]
While it's pointless discussing the full results, one important aspect here
is that with db-level locks are a lot more "gentle" with the database, and
this result in consistently faster execution times.
With 20 concurrent threads, every thread completed in about 0.06 seconds
with db-level locks (A-1). With A-2 it took about 0.45 seconds, and with
A-3 0.32. With A-2 we saw over 500 queries, a little more than 200 with
A-3, and just a bit more than 100 with A-1.
It's worth noting that what makes A-3 faster than A-2 is a random
allocation strategy which drastically reduces collisions. A-3's performance
with sequential allocations are much worse (the avg. thread runtime with 20
concurrent threads is about 0.75 seconds)

With the test on the Galera cluster I was expecting a terrible slowdown in
A-1 because of deadlocks caused by certification failures. I was extremely
disappointed that the slowdown I measured however does not make any of the
other algorithms a viable alternative.
On the Galera cluster I did not run extensive collections for A-2. Indeed
primary key violations seem to triggers db deadlock because of failed write
set certification too (but I have not yet tested this).
I run tests with 10 threads on each node, for a total of 30 workers. Some
results are available at [15]. There was indeed a slow down in A-1 (about
20%), whereas A-3 performance stayed pretty much constant. Regardless, A-1
was still at least 3 times faster than A-3.
As A-3's queries are mostly select (about 75% of them) use of caches might
make it a lot faster; also the algorithm is probably inefficient and can be
optimised in several areas. Still, I suspect it can be made faster than
A-1. At this stage I am leaning towards adoption db-level-locks with
retries for Neutron's IPAM. However, since I never trus

Re: [openstack-dev] [nova][vmware][ironic] Configuring active/passive HA Nova compute

[Gary Kotton]

On 2/23/15, 2:05 PM, "Matthew Booth"  wrote:

>On 20/02/15 11:48, Matthew Booth wrote:
>> Gary Kotton came across a doozy of a bug recently:
>> 
>> https://bugs.launchpad.net/nova/+bug/1419785
>> 
>> In short, when you start a Nova compute, it will query the driver for
>> instances and compare that against the expected host of the the instance
>> according to the DB. If the driver is reporting an instance the DB
>> thinks is on a different host, it assumes the instance was evacuated
>> while Nova compute was down, and deletes it on the hypervisor. However,
>> Gary found that you trigger this when starting up a backup HA node which
>> has a different `host` config setting. i.e. You fail over, and the first
>> thing it does is delete all your instances.
>> 
>> Gary and I both agree on a couple of things:
>> 
>> 1. Deleting all your instances is bad
>> 2. HA nova compute is highly desirable for some drivers
>> 
>> We disagree on the approach to fixing it, though. Gary posted this:
>> 
>> https://review.openstack.org/#/c/154029/
>> 
>> I've already outlined my objections to this approach elsewhere, but to
>> summarise I think this fixes 1 symptom of a design problem, and leaves
>> the rest untouched. If the value of nova compute's `host` changes, then
>> the assumption that instances associated with that compute can be
>> identified by the value of instance.host becomes invalid. This
>> assumption is pervasive, so it breaks a lot of stuff. The worst one is
>> _destroy_evacuated_instances(), which Gary found, but if you scan
>> nova/compute/manager for the string 'self.host' you'll find lots of
>> them. For example, all the periodic tasks are broken, including image
>> cache management, and the state of ResourceTracker will be unusual.
>> Worse, whenever a new instance is created it will have a different value
>> of instance.host, so instances running on a single hypervisor will
>> become partitioned based on which nova compute was used to create them.
>> 
>> In short, the system may appear to function superficially, but it's
>> unsupportable.
>> 
>> I had an alternative idea. The current assumption is that the `host`
>> managing a single hypervisor never changes. If we break that assumption,
>> we break Nova, so we could assert it at startup and refuse to start if
>> it's violated. I posted this VMware-specific POC:
>> 
>> https://review.openstack.org/#/c/154907/
>> 
>> However, I think I've had a better idea. Nova creates ComputeNode
>> objects for its current configuration at startup which, amongst other
>> things, are a map of host:hypervisor_hostname. We could assert when
>> creating a ComputeNode that hypervisor_hostname is not already
>> associated with a different host, and refuse to start if it is. We would
>> give an appropriate error message explaining that this is a
>> misconfiguration. This would prevent the user from hitting any of the
>> associated problems, including the deletion of all their instances.
>
>I have posted a patch implementing the above for review here:
>
>https://review.openstack.org/#/c/158269/

I have to look at what you have posted. I think that this topic is
something that we should speak about at the summit and this should fall
under some BP and well defined spec. I really would not like to see
existing installations being broken if and when this patch lands. It may
also affect Ironic as it works on the same model.

>
>Matt
>-- 
>Matthew Booth
>Red Hat Engineering, Virtualisation Team
>
>Phone: +442070094448 (UK)
>GPG ID:  D33C3490
>GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] New version of python-neutronclient release: 2.3.11

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/20/2015 11:01 PM, Matt Riedemann wrote:
> 
> 
> On 2/20/2015 6:23 AM, Alan Pevec wrote:
 https://review.openstack.org/#/c/157535/
>>> 
>>> Do you know where it ends ? We could set up Depends lines on
>>> those requirements stable/* reviews and line them up so that
>>> they are ready to merge when openstackclient is fixed in
>>> devstack.
>> 
>> Alternative workaround is https://review.openstack.org/157654
>> which is blocked on swift-dsvm-functional issue fixed by 
>> https://review.openstack.org/157670 which is blocked on
>> neutronclient i.e. we got a cyclic dep here which will require
>> ninja merge to resolve.
>> 
>> I suggest to start with ninja merging 157670 which looks the
>> most innocent.
>> 
>> Once we get icehouse working again we can look at backporting
>> venv patch series to devstack icehouse.
>> 
>> 
>> Cheers, Alan
>> 
>> __
>>
>>
>> 
OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>> 
> We disabled the swift functional job on stable/icehouse since it
> doesn't have the tox target and shouldn't have been running on
> stable/icehouse:
> 
> https://review.openstack.org/#/c/157825/
> 
> That freed up Adam's devstack workaround change:
> 
> https://review.openstack.org/#/c/157654/
> 
> So we can now focus on the neutronclient cap:
> 
> https://review.openstack.org/#/c/157606/
> 
> That hit a problem with an uncapped pyghmi version in the gate so
> we're also now capping that library as well.
> 

UPD: pyghmi cap didn't work, and it seems that we hit a bug in
'git-diff' that makes it return wrong exit code when used with --quiet
argument. The workaround is to switch to --exit-code argument:

https://review.openstack.org/158247

Once it's merged, we hit another issue - requirements integration
tools trying to sync stable/icehouse requirements for pycadf that does
not maintain a stable branch. The proper fix for this issue is to
avoid integration run for those projects for stable branches:

https://review.openstack.org/158086

/Ihar
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJU6xpTAAoJEC5aWaUY1u57TkgIAKnys1hfuscwDTyErlbsvu6q
JnKkkHSfnn/so36xCt65r6mq1NeBUF+Wn4CFeoMwh6eIbql4aeXJC0voBwH5ScpL
HUKZtz2eL088wJfwu5yhn0TQleh1C5GCQf2rutQv1bcqdwhRbo1MLf0Dda34dJSg
HL6gNfYTm3TckgNvWujNxGOluEUg7It0a/myf9Hn0xDhyLCvdF9+7DMzvotmEai+
HuoGhaFBGjyEYgS/aCRjg0c0qtBxAk4pITQwuIvUzkDWLYQLuW6S4pzaySGDXxP0
zVbqMoReUtyCv1f6t9k43K+K1nj2yloBtlzBsUUF04n9YbGlfVqlWtHu2yAc86c=
=aMhI
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][cisco][apic] entry point for APIC agent

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Ivar,

Thanks! Please add me to review list.

is the agent leaving the tree this cycle? If so, there is no big
reason to introduce a new agent in master just to yank it several
weeks ago.

/Ihar

On 02/21/2015 01:41 AM, Ivar Lazzaro wrote:
> Hi Ihar,
> 
> That was missed for the Juno release, I'll post a patch on master
> and then backport it to stable/juno.
> 
> Thanks for catching that, Ivar.
> 
> On Fri Feb 20 2015 at 11:06:11 AM Ihar Hrachyshka
> mailto:ihrac...@redhat.com>> wrote:
> 
> Hi,
> 
> does anyone know why we don't maintain an entry point for APIC
> agent in setup.cfg? The code in [1] looks like there is a main()
> function for the agent, but for some reason it's not exposed to
> any console_script during installation of neutron.
> 
> Is there any reason not to do it?
> 
> [1]: 
> http://git.openstack.org/cgit/openstack/neutron/tree/__neutron__/plugins/ml2/drivers/__cisco/__apic/apic_topology.py#__n320
>
> 

> 
> /Ihar
> 
> __
>
> 
OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request@lists.__op__enstack.org?subject:__unsubscrib__e
> 
>
> 
http://lists.openstack.org/__cgi__-bin/mailman/listinfo/__openstac__k-dev
> 
>
> 
> 
> 
> __
>
> 
OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJU6xqrAAoJEC5aWaUY1u57GXYIAMAiXzgvK/QCHmaQtp4YWMvJ
SQeG7Ql/Yv/j7ew1a5sLeQvI/9qjsmFJ4l2ePAYPxuw5VMENISguCn/5HVIUPIru
iuuS9jvQXbIM5X0ug8y6OVDpKEiSPYFHG05HZ/atsH5D2UjdFF24SaHa9IWJyo4y
mzH/QIkGuo/hjIxfcwm0vWB7+5axrHGLu5Km8PFxQADGbii7IJcVU3BsFdvjyZld
LWDwKESwN3KTj7a9yGssDIZ8q5/o/8DwHSG/izcBY5C8SHC7iH2tnsY5F0/p56B6
Sy+IFh/pSffAfXevJhrqKawlWH4GfLGytaJ6pHBwI6FOAV6pMDl1bTUY+t96PF0=
=Dach
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Priority tracking

[Gary Kotton]

Hi,
Not sure if everyone is aware of this. There is a ether pad where the priority 
of the project appears: 
https://etherpad.openstack.org/p/kilo-nova-priorities-tracking. This has the 
major bullets for the K cycle.
So in short - if you reviews are not on the page they may not get review cycles 
:)
Thanks
Gary
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic "The Heat Orchestration Template Builder: A demonstration"

[Angus Salkeld]

On Mon, Feb 23, 2015 at 9:18 PM, Aggarwal, Nikunj 
wrote:

>  Hi Angus,
>
>  I am working Timur and Drago on HOT builder. I am using this ->
> https://github.com/rackerlabs/hotbuilder as the base and made some
> improvements over the existing code and wish to give a demonstration on how
> easy it is to create a HOT template from Horizon.
>
>
That's awesome, looking forward to see it!

-Angus


>
>
> Regards,
>
> Nikunj
>
>
>
> *From:* Angus Salkeld [mailto:asalk...@mirantis.com]
> *Sent:* Monday, February 23, 2015 4:52 AM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit
> topic "The Heat Orchestration Template Builder: A demonstration"
>
>
>
> On Sat, Feb 21, 2015 at 4:21 AM, Aggarwal, Nikunj 
> wrote:
>
>  Hi,
>
>
>
> I have submitted  presentations for Openstack L summit :
>
>
>
> The Heat Orchestration Template Builder: A demonstration
> 
>
>
>
>
>
> Hi
>
> Nice to see work on a HOT builder progressing, but..
>
> are you planning on integrating this with the other HOT builder efforts?
>
> Is the code public (link)?
>
>
> This is more of a framework to make these easier to build:
> https://github.com/stackforge/merlin
> https://wiki.openstack.org/wiki/Merlin
>
> Timur (who works on Merlin) is working with Rackers to build this upstream
> - I am not sure on the progress.
> https://github.com/rackerlabs/hotbuilder
> https://github.com/rackerlabs/foundry
>
>
>
> It would be nice if we could all work together (I am hoping you are
> already)?
>
> Hopefully some of the others that are working on this can chip in and say
> where they
>
> are.
>
>
> -Angus
>
>
>
>
>
> Please cast your vote if you feel it is worth for presentation.
>
>
>
> Thanks & Regards,
>
> Nikunj
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic "The Heat Orchestration Template Builder: A demonstration"

[Aggarwal, Nikunj]

I am also looking forward to present it. Please support this by giving your 
vote ☺

Regards,
Nikunj

From: Angus Salkeld [mailto:asalk...@mirantis.com]
Sent: Monday, February 23, 2015 5:53 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic 
"The Heat Orchestration Template Builder: A demonstration"

On Mon, Feb 23, 2015 at 9:18 PM, Aggarwal, Nikunj 
mailto:nikunj.aggar...@hp.com>> wrote:
Hi Angus,
I am working Timur and Drago on HOT builder. I am using this -> 
https://github.com/rackerlabs/hotbuilder as the base and made some improvements 
over the existing code and wish to give a demonstration on how easy it is to 
create a HOT template from Horizon.

That's awesome, looking forward to see it!
-Angus


Regards,
Nikunj

From: Angus Salkeld [mailto:asalk...@mirantis.com]
Sent: Monday, February 23, 2015 4:52 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic 
"The Heat Orchestration Template Builder: A demonstration"

On Sat, Feb 21, 2015 at 4:21 AM, Aggarwal, Nikunj 
mailto:nikunj.aggar...@hp.com>> wrote:
Hi,

I have submitted  presentations for Openstack L summit :


The Heat Orchestration Template Builder: A 
demonstration



Hi
Nice to see work on a HOT builder progressing, but..
are you planning on integrating this with the other HOT builder efforts?
Is the code public (link)?

This is more of a framework to make these easier to build:
https://github.com/stackforge/merlin
https://wiki.openstack.org/wiki/Merlin

Timur (who works on Merlin) is working with Rackers to build this upstream - I 
am not sure on the progress.
https://github.com/rackerlabs/hotbuilder
https://github.com/rackerlabs/foundry

It would be nice if we could all work together (I am hoping you are already)?
Hopefully some of the others that are working on this can chip in and say where 
they
are.

-Angus


Please cast your vote if you feel it is worth for presentation.

Thanks & Regards,
Nikunj


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][vmware][ironic] Configuring active/passive HA Nova compute

[Matthew Booth]

On 23/02/15 12:13, Gary Kotton wrote:
> 
> 
> On 2/23/15, 2:05 PM, "Matthew Booth"  wrote:
> 
>> On 20/02/15 11:48, Matthew Booth wrote:
>>> Gary Kotton came across a doozy of a bug recently:
>>>
>>> https://bugs.launchpad.net/nova/+bug/1419785
>>>
>>> In short, when you start a Nova compute, it will query the driver for
>>> instances and compare that against the expected host of the the instance
>>> according to the DB. If the driver is reporting an instance the DB
>>> thinks is on a different host, it assumes the instance was evacuated
>>> while Nova compute was down, and deletes it on the hypervisor. However,
>>> Gary found that you trigger this when starting up a backup HA node which
>>> has a different `host` config setting. i.e. You fail over, and the first
>>> thing it does is delete all your instances.
>>>
>>> Gary and I both agree on a couple of things:
>>>
>>> 1. Deleting all your instances is bad
>>> 2. HA nova compute is highly desirable for some drivers
>>>
>>> We disagree on the approach to fixing it, though. Gary posted this:
>>>
>>> https://review.openstack.org/#/c/154029/
>>>
>>> I've already outlined my objections to this approach elsewhere, but to
>>> summarise I think this fixes 1 symptom of a design problem, and leaves
>>> the rest untouched. If the value of nova compute's `host` changes, then
>>> the assumption that instances associated with that compute can be
>>> identified by the value of instance.host becomes invalid. This
>>> assumption is pervasive, so it breaks a lot of stuff. The worst one is
>>> _destroy_evacuated_instances(), which Gary found, but if you scan
>>> nova/compute/manager for the string 'self.host' you'll find lots of
>>> them. For example, all the periodic tasks are broken, including image
>>> cache management, and the state of ResourceTracker will be unusual.
>>> Worse, whenever a new instance is created it will have a different value
>>> of instance.host, so instances running on a single hypervisor will
>>> become partitioned based on which nova compute was used to create them.
>>>
>>> In short, the system may appear to function superficially, but it's
>>> unsupportable.
>>>
>>> I had an alternative idea. The current assumption is that the `host`
>>> managing a single hypervisor never changes. If we break that assumption,
>>> we break Nova, so we could assert it at startup and refuse to start if
>>> it's violated. I posted this VMware-specific POC:
>>>
>>> https://review.openstack.org/#/c/154907/
>>>
>>> However, I think I've had a better idea. Nova creates ComputeNode
>>> objects for its current configuration at startup which, amongst other
>>> things, are a map of host:hypervisor_hostname. We could assert when
>>> creating a ComputeNode that hypervisor_hostname is not already
>>> associated with a different host, and refuse to start if it is. We would
>>> give an appropriate error message explaining that this is a
>>> misconfiguration. This would prevent the user from hitting any of the
>>> associated problems, including the deletion of all their instances.
>>
>> I have posted a patch implementing the above for review here:
>>
>> https://review.openstack.org/#/c/158269/
> 
> I have to look at what you have posted. I think that this topic is
> something that we should speak about at the summit and this should fall
> under some BP and well defined spec. I really would not like to see
> existing installations being broken if and when this patch lands. It may
> also affect Ironic as it works on the same model.

This patch will only affect installations configured with multiple
compute hosts for a single hypervisor. These are already broken, so this
patch will at least let them know if they haven't already noticed.

It won't affect Ironic, because they configure all compute hosts to have
the same 'host' value. An Ironic user would only notice this patch if
they accidentally misconfigured it, which is the intended behaviour.

Incidentally, I also support more focus on the design here. Until we
come up with a better design, though, we need to do our best to prevent
non-trivial corruption from a trivial misconfiguration. I think we need
to merge this, or something like it, now and still have a summit discussion.

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Temporary freeze on API changes

[Matthew Gilliard]

The devref has merged:
http://docs.openstack.org/developer/nova/devref/api_microversions.html

There are a few corner cases being worked on now, but the information
in there should be enough to add a new API change & version.

MG

On Fri, Feb 20, 2015 at 12:33 AM, Christopher Yeoh  wrote:
> Hi,
>
> The Nova API subgroup is planning on releasing V2.1 on Monday (changing its
> status from experimental to CURRENT) and merging the first patch which uses
> microversions on top of 2.1 on the Wednesday.
>
> In the meantime we'd appreciate it if reviewers did not +A any patchset
> which changes the REST API (as it potentially means we'd have to sync
> changes to 2.1 as well. Any future api changes should only be applied to the
> v2.1/v3 tree and use the microversions mechanism.
>
> When https://review.openstack.org/#/c/140313/ is merged, that will signal
> that microversions
> is enabled and we are open for patches to v2.1 microversions. At this point
> no changes should be accepted to the old v2 api code and anything to the
> v2.1 code base should be done using the microversions mechanism.
>
> There is devref docs on how to use microversions either merged or working
> its way through gerrit.
>
> Regards,
>
> Chris
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel][Nova][Cinder] Operations: adding new nodes in "disabled" state, allowed for test tenant only

[Bogdan Dobrelya]

+ [Fuel] tag
+ openstack-operators ML

Joe Gordon joe.gordon0 at gmail.com
Thu Dec 4 13:26:59 UTC 2014

>On Wed, Dec 3, 2014 at 3:31 PM, Mike Scherbakov 
>wrote:

>> Hi all,
>> enable_new_services in nova.conf seems to allow add new compute nodes in
>> disabled state:
>>
>>
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L507-L508,
>> so it would allow to check everything first, before allowing production
>> workloads host a VM on it. I've filed a bug to Fuel to use this by
default
>> when we scale up the env (add more computes) [1].
>>
>> A few questions:
>>
>>1. can we somehow enable compute service for test tenant first? So
>>cloud administrator would be able to run test VMs on the node, and
after
>>ensuring that everything is fine - to enable service for all tenants
>>
>>

>Although there may be more then one way to set this up in nova, this can
>definitely be done via nova host aggregates. Put new compute services into
>an aggregate that only specific tenants can access (controlled via
scheduler filter).

Looks reasonable, +1 for Nova host aggregates [0].

There is still a question, though, about an enable_new_services
parameter, cinder and other OpenStack services. It is not clear how to
use this parameter from an operator perspective, for example:
1) While deploying or scaling the OpenStack environment, we should set
enable_new_services=false for all services which support it.
2) Once the deploy/scale is done, re-enable disabled services. But how
exactly that should be done?
* Set enable_new_services=True and restart the schedulers and conductor
services? Or API services as well?
* Keep enable_new_services=false in configs, but issue - for nova
exmaple - 'nova-manage service enable ...' commands for added compute
nodes? And what about cinder and other ones (there are no *-manage
service enable commands)?
* Some another way?

And regarding plans for implementing this improvement in Fuel, I believe
for nova computes it could be done as a workaround for the 6.1 release
with the help of enable_new_services configuration parameter and a
separate Fuel post-deploy granular task which should re-enable the
disabled compute services. But this should be re-implemented later with
separate host aggregate for deployment and health checks, see the
related blueprint [1].

[0]
http://docs.openstack.org/havana/config-reference/content/host-aggregates.html
[1] https://blueprints.launchpad.net/fuel/+spec/disable-new-computes

>
>1.
>2. What about Cinder? Is there a similar option / ability?
>3. What about other OpenStack projects?
>
> What is your opinion, how we should approach the problem (if there is a
> problem)?
>
> [1] https://bugs.launchpad.net/fuel/+bug/1398817
> --
> Mike Scherbakov
> #mihgen
>

-- 
Best regards,
Bogdan Dobrelya,
Skype #bogdando_at_yahoo.com
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Priority tracking

[Gary Kotton]

Just for a few clarifications:

  1.  The link below is for the nova priority tracking
  2.  It was requested that we do not add bug fixes to that list. There are 
other tracking tools for that
  3.  The list does have a "Quick/Trivial Hit Bugs". Please read the 
description before you start to add to that list. Feel free to reach out to the 
guys who regularly contribute to this list if you feel that you want to dive in

A luta continua
Gary

From: Gary Kotton mailto:gkot...@vmware.com>>
Reply-To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, February 23, 2015 at 2:19 PM
To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [nova] Priority tracking

Hi,
Not sure if everyone is aware of this. There is a ether pad where the priority 
of the project appears: 
https://etherpad.openstack.org/p/kilo-nova-priorities-tracking. This has the 
major bullets for the K cycle.
So in short - if you reviews are not on the page they may not get review cycles 
:)
Thanks
Gary
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Outcome of the nova FFE meeting for Kilo

[Nikola Đipanov]

On 02/20/2015 11:33 PM, Sourabh Patwardhan (sopatwar) wrote:
> Nova core reviewers,
> 
> May I request an FFE for Cisco VIF driver:
> https://review.openstack.org/#/c/157616/
> 
> This is a small isolated change similar to the vhostuser / open contrail
> vif drivers for which FFE has been granted.
> 
> Thanks,
> Sourabh
> 
> 

Hey Sourabh,

Sorry that you didn't get any responses sooner.

Actually the FFEs get decided by a subset of the Nova core team called
the nova-drivers. You can see it briefly mentioned here [1]. (*)

You can see an ethepad that hosts the minutes of the meeting where
nova-drivers were deciding on FFEs at [2] which may give you more
insight into why your BP did not make the cut.

Once again - apologies for any poor experience you may have had trying
to contribute to Nova.

N.

[1] https://wiki.openstack.org/wiki/Nova
[2] https://etherpad.openstack.org/p/kilo-nova-ffe-requests

(*) On a side note this is an example of us not making the process and
the players clear to our contributors, so we should probably try to
document the role of the nova-drivers better at the very least.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Libguestfs: possibility not to use it, even when installed ?

[Richard W.M. Jones]

On Mon, Feb 23, 2015 at 11:08:31AM +0100, Raphael Glon wrote:
> sudo sysctl -w fs.protected_hardlinks=0 + common user nova/qemu

We fixed this a while back (in July 2013 in fact).

I think if you forked at Fedora 19 then you're probably using
libguestfs 1.24 + supermin 4.  I'd definitely recommend updating to at
least libguestfs >= 1.26 + supermin 5 since that fixes a bunch of bugs
around building the appliance, and is also faster.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] A question about strange behavior of oslo.config in eclipse

[Doug Hellmann]

On Mon, Feb 23, 2015, at 01:29 AM, Joshua Zhang wrote:
> Hi Doug,
> 
>  I can't find above error when using the latest codes (2015-02-22),
>  but
> new error occur ( see [1] ).  I feel it has no concern with the
> code(ConfigOpts.import_opt), the same code can be run in both bash and
> pycharm, it just didn't work in eclipse+pydev. It looks like there are
> some
> conflicts between oslo/monkey patch and pydev. You are python expert,
> could
> you give me some input by the following traceback ?

Unfortunately, I don't know much about eclipse or pydev, so I'm not sure
how much help I can be. The traceback is saying that the MultiString
class is missing from oslo_config.types. Is that true? Do you somehow
have a version of the file without that class? Is there another version
of oslo.config installed elsewhere that could be interfering with the
imports?

> 
>  Another question,  'Gevent compatible debugging' feature both in
> eclipse and pycharm doesn't work, changing 'thread=False' for monkey
> patch
> may cause the error [2], so now I have to get back to use pdb to debug
> openstack. Could you have some idea to make oslo/monkey patch is more
> friendly for IDE ? many thanks.

This is outside of my area of expertise. Maybe another pydev user can
share information about how they have configured it?

> 
> *[1]*, traceback when running 'neutron-server --config-file
> /etc/neutron/neutron.conf --config-file
> /etc/neutron/plugins/ml2/ml2_conf.ini' in eclipse+pydev env. while we can
> run this command well in bash and pycharm.
> 
> Traceback (most recent call last):
>   File "/usr/local/bin/neutron-server", line 9, in 
> load_entry_point('neutron==2015.1.dev202', 'console_scripts',
> 'neutron-server')()
>   File
>   "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 521, in load_entry_point
> return get_distribution(dist).load_entry_point(group, name)
>   File
>   "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 2632, in load_entry_point
> return ep.load()
>   File
>   "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 2312, in load
> return self.resolve()
>   File
>   "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py",
> line 2318, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
>   File "/bak/openstack/neutron/neutron/cmd/eventlet/server/__init__.py",
> line 13, in 
> from neutron import server
>   File "/bak/openstack/neutron/neutron/server/__init__.py", line 26, in
> 
> from neutron.common import config
>   File "/bak/openstack/neutron/neutron/common/config.py", line 25, in
> 
> import oslo_messaging
>   File
>   "/usr/local/lib/python2.7/dist-packages/oslo_messaging/__init__.py",
> line 18, in 
> from .notify import *
>   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/__init__.py",
> line 23, in 
> from .notifier import *
>   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/notifier.py",
> line 32, in 
> help='Driver or drivers to handle sending notifications.'),
>   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line
> 1108, in __init__
> item_type=types.MultiString(),
> AttributeError: 'module' object has no attribute 'MultiString'
> 
> *[2] *
> 
> 2015-02-22 14:56:24.117 ERROR oslo_messaging.rpc.dispatcher
> [req-65501748-3db5-43b6-b00e-732565d2192a TestNetworkVPNaaS-1393357785
> TestNetworkVPNaaS-1147259963] Exception during message handling:
> _oslo_messaging_localcontext_9bb7d928d1a042e085f354eb118e98a0
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher
> Traceback
> (most recent call last):
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
> line 142, in _dispatch_and_reply
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher
> executor_callback))
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
> line 188, in _dispatch
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher
> localcontext.clear_local_context()
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/localcontext.py",
> line 55, in clear_local_context
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher
> delattr(_STORE, _KEY)
> 2015-02-22 14:56:24.117 28042 TRACE oslo_messaging.rpc.dispatcher
> AttributeError:
> _oslo_messaging_localcontext_9bb7d928d1a042e085f354eb118e98a0
> 
> 
> 
> On Sat, Feb 14, 2015 at 7:05 AM, Doug Hellmann 
> wrote:
> 
> >
> >
> > On Thu, Feb 12, 2015, at 07:19 AM, Joshua Zhang wrote:
> > > Hi Doug,
> > >
> > >Thank you very much for your reply. I don't have any codes, so no any
> > > special codes as well.
> > >Only thing I d

Re: [openstack-dev] [nova] Outcome of the nova FFE meeting for Kilo

[Daniel P. Berrange]

On Mon, Feb 23, 2015 at 03:47:20PM +0100, Nikola Đipanov wrote:
> On 02/20/2015 11:33 PM, Sourabh Patwardhan (sopatwar) wrote:
> > Nova core reviewers,
> > 
> > May I request an FFE for Cisco VIF driver:
> > https://review.openstack.org/#/c/157616/
> > 
> > This is a small isolated change similar to the vhostuser / open contrail
> > vif drivers for which FFE has been granted.
> 
> Actually the FFEs get decided by a subset of the Nova core team called
> the nova-drivers. You can see it briefly mentioned here [1]. (*)
> 
> You can see an ethepad that hosts the minutes of the meeting where
> nova-drivers were deciding on FFEs at [2] which may give you more
> insight into why your BP did not make the cut.

Unfortunately the Cisco VIF driver was not on the list of features
considered in the first place, since (AFAICT) this FFE request was
only made after the FFEs had all been decided for Kilo.

Personally I think the Nova FFE process is too inflexible, and thus
I would support inclusion of this (and any) VIF driver into Kilo
regardless of FFE status for pragmatic reasons, but this appears to
be a minority view right now.

> (*) On a side note this is an example of us not making the process and
> the players clear to our contributors, so we should probably try to
> document the role of the nova-drivers better at the very least.

As a member of Nova drivers, I personally think that the team should
never have existed in the first place, and should cease to exist at
the soonest opportunity. That it exists at all is just an accident
of history due the use of launchpad for blueprint approval. IMHO it
makes no sense to restrict which cores can participate in feature
review & approval - any Nova core should be able to be choose to
participate as their time allows, without needing to be granted
access to yet another special group.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] New version of python-neutronclient release: 2.3.11

[Doug Hellmann]

On Mon, Feb 23, 2015, at 07:17 AM, Ihar Hrachyshka wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 02/20/2015 11:01 PM, Matt Riedemann wrote:
> > 
> > 
> > On 2/20/2015 6:23 AM, Alan Pevec wrote:
>  https://review.openstack.org/#/c/157535/
> >>> 
> >>> Do you know where it ends ? We could set up Depends lines on
> >>> those requirements stable/* reviews and line them up so that
> >>> they are ready to merge when openstackclient is fixed in
> >>> devstack.
> >> 
> >> Alternative workaround is https://review.openstack.org/157654
> >> which is blocked on swift-dsvm-functional issue fixed by 
> >> https://review.openstack.org/157670 which is blocked on
> >> neutronclient i.e. we got a cyclic dep here which will require
> >> ninja merge to resolve.
> >> 
> >> I suggest to start with ninja merging 157670 which looks the
> >> most innocent.
> >> 
> >> Once we get icehouse working again we can look at backporting
> >> venv patch series to devstack icehouse.
> >> 
> >> 
> >> Cheers, Alan
> >> 
> >> __
> >>
> >>
> >> 
> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: 
> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >> 
> > We disabled the swift functional job on stable/icehouse since it
> > doesn't have the tox target and shouldn't have been running on
> > stable/icehouse:
> > 
> > https://review.openstack.org/#/c/157825/
> > 
> > That freed up Adam's devstack workaround change:
> > 
> > https://review.openstack.org/#/c/157654/
> > 
> > So we can now focus on the neutronclient cap:
> > 
> > https://review.openstack.org/#/c/157606/
> > 
> > That hit a problem with an uncapped pyghmi version in the gate so
> > we're also now capping that library as well.
> > 
> 
> UPD: pyghmi cap didn't work, and it seems that we hit a bug in
> 'git-diff' that makes it return wrong exit code when used with --quiet
> argument. The workaround is to switch to --exit-code argument:
> 
> https://review.openstack.org/158247
> 
> Once it's merged, we hit another issue - requirements integration
> tools trying to sync stable/icehouse requirements for pycadf that does
> not maintain a stable branch. The proper fix for this issue is to
> avoid integration run for those projects for stable branches:
> 
> https://review.openstack.org/158086

We will also need to start creating stable branches for all libraries
with caps. It might make sense to do that retroactively for pycadf and
any others for our existing stable branches.

Doug

> 
> /Ihar
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1
> 
> iQEcBAEBAgAGBQJU6xpTAAoJEC5aWaUY1u57TkgIAKnys1hfuscwDTyErlbsvu6q
> JnKkkHSfnn/so36xCt65r6mq1NeBUF+Wn4CFeoMwh6eIbql4aeXJC0voBwH5ScpL
> HUKZtz2eL088wJfwu5yhn0TQleh1C5GCQf2rutQv1bcqdwhRbo1MLf0Dda34dJSg
> HL6gNfYTm3TckgNvWujNxGOluEUg7It0a/myf9Hn0xDhyLCvdF9+7DMzvotmEai+
> HuoGhaFBGjyEYgS/aCRjg0c0qtBxAk4pITQwuIvUzkDWLYQLuW6S4pzaySGDXxP0
> zVbqMoReUtyCv1f6t9k43K+K1nj2yloBtlzBsUUF04n9YbGlfVqlWtHu2yAc86c=
> =aMhI
> -END PGP SIGNATURE-
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Cancelling team meeting today on 02/23/2015

[Renat Akhmerov]

Sorry for the late notice. We’re cancelling our team meeting today. Several 
team members are on official holidays.

Renat Akhmerov
@ Mirantis Inc.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova]

[Adam Young]

On 02/16/2015 05:00 AM, Nikolay Makhotkin wrote:
Well, if we use trust-scoped token for getting server-list from nova 
(simply use nova.servers.list() ),


Novaclient somehow tries to get another token: 
https://github.com/openstack/python-novaclient/blob/master/novaclient/client.py#L690-L724


Actually, novaclient does this request: (found from debug of novaclient)
So this sounds like a bug in Nova client.  Jamie Lennox has been working 
with the various client teams to get the  using the Auth plugin 
architecture and session management from Keystone client to try and make 
the usage consistent.






REQ: curl -i 'http://:5000/v2.0/tokens' -X POST -H "Accept: 
application/json" -H "Content-Type: application/json" -H "User-Agent: 
python-novaclient" -d '{"auth": {"token": {"id": 
"78c71fb549244075b3a5d994baa326b3"}, "tenantName": 
"b0c9bbb541d541b098c3c0a92412720d"}}'


I.e., this is the request for another auth token from keystone. 
Keystone here returns 403 because token in request is trust-scoped.


Why I can't do this simple command using trust-scoped token?

Note: Doing the keystone --os-token 5483086d91094a3886ccce1442b538a0 
--os-endpoint http://:5000/v2.0 tenant-list, it returns 
tenant-list (not 403).
Note2: Doing the server-list request directly to api with trust-scoped 
token, it returns 200, not 403:


curl -H "X-Auth-Token: 5483086d91094a3886ccce1442b538a0" 
http://192.168.0.2:8774/v3/servers


{
"servers": [  ]
}

How I can use trust-scoped tokrn via client?

On Fri, Feb 13, 2015 at 9:16 PM, Alexander Makarov 
mailto:amaka...@mirantis.com>> wrote:


Adam, Nova client does it for some reason during a call to
nova.servers.list()


On Thu, Feb 12, 2015 at 10:03 PM, Adam Young mailto:ayo...@redhat.com>> wrote:

On 02/12/2015 10:40 AM, Alexander Makarov wrote:

A trust token cannot be used to get another token:

https://github.com/openstack/keystone/blob/master/keystone/token/controllers.py#L154-L156
You have to make your Nova client use the very same trust
scoped token obtained from authentication using trust without
trying to authenticate with it one more time.



Actually, there have been some recent changes to allow
re-delegation of Trusts, but for older deployments, you are
correct.  I hadn't seen anywhere here that he was trying to
use a trust token to get another token, though.




On Wed, Feb 11, 2015 at 9:10 PM, Adam Young
mailto:ayo...@redhat.com>> wrote:

On 02/11/2015 12:16 PM, Nikolay Makhotkin wrote:

No, I just checked it. Nova receives trust token and
raise this error.

In my script, I see:

http://paste.openstack.org/show/171452/

And as you can see, token from trust differs from direct
user's token.


The original user needs to have the appropriate role to
perform the operation on the specified project.  I see
the admin role is created on the trust. If the trustor
did not have that role, the trustee would not be able to
exececute the trust and get a token.  It looks like you
were able to execute the trust and get a token,  but I
would like you to confirm that, and not just trust the
keystone client:  either put debug statements in Keystone
or call the POST to tokens from curl with the appropriate
options to get a trust token.  In short, make sure you
have not fooled yourself.  You can also look in the token
table inside Keystone to see the data for the trust
token, or validate the token  via curl to see the data in
it.  In all cases, there should be an OS-TRUST stanza in
the token data.


If it is still failing, there might be some issue on the
Policy side.  I have been assuming that you are running
with the default policy for Nova.


http://git.openstack.org/cgit/openstack/nova/tree/etc/nova/policy.json

I'm not sure which rule matches for list servers (Nova
developer input would be appreciated)  but I'm guessing
it is executing the rule
|
"admin_or_owner": "is_admin:True or
project_id:%(project_id)s",

Since that is the default. I am guessing that the
project_id in question comes from the token here, as that
seems to be common, but if not, it might be that the two
values are mismatched. Perhaps there Proejct ID value
from the client env var is sent, and matches what the
trustor normally works as, not the project in question. 
If these two values don't match, then, yes, the rule

would fail.
|





On Wed, Feb 11, 2015 at 7:55 PM, Adam Young
mailto:ayo...@redha

Re: [openstack-dev] [horizon][keystone]

[Adam Young]

On 02/18/2015 12:02 PM, David Chadwick wrote:

I think this GUI is not intuitive to users and therefore should not be
encouraged or supported.
It is a fist hack.  I think you don't mean "any gui"  just that there 
are some warning flags raised by this design?




If you ask a user "what does authenticate via a Discovery Service mean?"
I think you will get some very strange answers. The same goes for
"Authenticate using Default Protocol". Users will have no idea what that
means.


He's not a native English speaker.  We'll need to craft the text here, 
and also carefully review the nuances.  Then there are the international 
translations




There has been a lot of research into how to support federated
authentication and there is a lot of practical experience across the
academic world from dozens of countries for many years. Most
universities now use federated login on a daily basis. We should use
this experience and follow best practise (which sadly does not involve
the screens that are being proposed here).

If you want to read more you can read a Good Practice Guide here

https://discovery.refeds.org/

It should help you to redesign the login page
THanks for the links.  Very helpful.  Some of that might require some 
changes from Horizon, but Jamie has seen those issues also come up in 
the context of the Kerberos login, and we can work to make this a 
smoother experience.


David, we certainly could use some UX  feedback here. Unfortunately, my 
GO-TO team member has decided to GO-TO a trip around the world...who can 
we pull in to make this flow in with the rest of Horizon?






regards

David

On 18/02/2015 16:06, Dolph Mathews wrote:

On Fri, Feb 6, 2015 at 12:47 PM, Adam Young mailto:ayo...@redhat.com>> wrote:

 On 02/04/2015 03:54 PM, Thai Q Tran wrote:

 Hi all,

 I have been helping with the websso effort and wanted to get some
 feedback.
 Basically, users are presented with a login screen where they can
 select: credentials, default protocol, or discovery service.
 If user selects credentials, it works exactly the same way it
 works today.
 If user selects default protocol or discovery service, they can
 choose to be redirected to those pages.

 Keep in mind that this is a prototype, early feedback will be good.
 Here are the relevant patches:
 https://review.openstack.org/#/c/136177/
 https://review.openstack.org/#/c/136178/
 https://review.openstack.org/#/c/151842/

 I have attached the files and present them below:



 Replace the dropdown with a specific link for each protocol type:

 SAML and OpenID  are the only real contenders at the moment, but we
 will not likely have so many that it will clutter up the page.


Agree, but the likelihood that a single IdP will support multiple
protocols is probably low. Keystone certainly supports that from an API
perspective, but I don't think it should be the default UX. Choose a
remote IdP first, and then if *that* IdP supports multiple federation
protocols, present them.
  



 Thanks for doing this.






 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 

 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Vote for talk on using Metatemplates for heat

[Pratik Mallya]

Hey Folks,

In case you missed it, many of you might find the work we’ve done in maintaing 
catalogs of heat templates interesting.
The proposal is here: 
https://www.openstack.org/vote-vancouver/presentation/one-heat-template-to-rule-them-all

Please take a look if you haven’t already, and vote accordingly :-).

Thanks,
Pratik


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][keystone]

[David Chadwick]

Hi Adam

there is some work being done on this by HP, Intel and IBM, and they
have some designs at

http://invis.io

pieter.c.kruithof...@hp.com  can send you the details as he invited me
to comment on the designs, which I have done.

As you know, we already have our own federated Horizon login screen, and
this summer I am hoping to have another student integrate this into the
current release, as well as make some enhancements to it (like
type-ahead searching for IdPs)

regards

David

On 23/02/2015 15:54, Adam Young wrote:
> On 02/18/2015 12:02 PM, David Chadwick wrote:
>> I think this GUI is not intuitive to users and therefore should not be
>> encouraged or supported.
> It is a fist hack.  I think you don't mean "any gui"  just that there
> are some warning flags raised by this design?
> 
>>
>> If you ask a user "what does authenticate via a Discovery Service mean?"
>> I think you will get some very strange answers. The same goes for
>> "Authenticate using Default Protocol". Users will have no idea what that
>> means.
> 
> He's not a native English speaker.  We'll need to craft the text here,
> and also carefully review the nuances.  Then there are the international
> translations
> 
>>
>> There has been a lot of research into how to support federated
>> authentication and there is a lot of practical experience across the
>> academic world from dozens of countries for many years. Most
>> universities now use federated login on a daily basis. We should use
>> this experience and follow best practise (which sadly does not involve
>> the screens that are being proposed here).
>>
>> If you want to read more you can read a Good Practice Guide here
>>
>> https://discovery.refeds.org/
>>
>> It should help you to redesign the login page
> THanks for the links.  Very helpful.  Some of that might require some
> changes from Horizon, but Jamie has seen those issues also come up in
> the context of the Kerberos login, and we can work to make this a
> smoother experience.
> 
> David, we certainly could use some UX  feedback here. Unfortunately, my
> GO-TO team member has decided to GO-TO a trip around the world...who can
> we pull in to make this flow in with the rest of Horizon?
> 
> 
> 
>>
>> regards
>>
>> David
>>
>> On 18/02/2015 16:06, Dolph Mathews wrote:
>>> On Fri, Feb 6, 2015 at 12:47 PM, Adam Young >> > wrote:
>>>
>>>  On 02/04/2015 03:54 PM, Thai Q Tran wrote:
  Hi all,

  I have been helping with the websso effort and wanted to get some
  feedback.
  Basically, users are presented with a login screen where they can
  select: credentials, default protocol, or discovery service.
  If user selects credentials, it works exactly the same way it
  works today.
  If user selects default protocol or discovery service, they can
  choose to be redirected to those pages.

  Keep in mind that this is a prototype, early feedback will be
 good.
  Here are the relevant patches:
  https://review.openstack.org/#/c/136177/
  https://review.openstack.org/#/c/136178/
  https://review.openstack.org/#/c/151842/

  I have attached the files and present them below:
>>>
>>>
>>>  Replace the dropdown with a specific link for each protocol type:
>>>
>>>  SAML and OpenID  are the only real contenders at the moment, but we
>>>  will not likely have so many that it will clutter up the page.
>>>
>>>
>>> Agree, but the likelihood that a single IdP will support multiple
>>> protocols is probably low. Keystone certainly supports that from an API
>>> perspective, but I don't think it should be the default UX. Choose a
>>> remote IdP first, and then if *that* IdP supports multiple federation
>>> protocols, present them.
>>>  
>>>
>>>  Thanks for doing this.





 
 __

  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> 
>>> __
>>>
>>>  OpenStack Development Mailing List (not for usage questions)
>>>  Unsubscribe:
>>>  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> 
>>> 
>>>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openst

Re: [openstack-dev] [horizon][heat]Vote for Openstack L summit topic "The Heat Orchestration Template Builder: A demonstration"

[Yves-Gwenaël Bourhis]

We have a library called "flame" :

https://github.com/cloudwatt/flame
https://pypi.python.org/pypi/python-flameclient/0.1.0

It generates heat templates from "existing" resources:

http://dev.cloudwatt.com/en/blog/introducing-flame-automatic-heat-template-generation.html

Could it help?

Le 23/02/2015 00:21, Angus Salkeld a écrit :
> On Sat, Feb 21, 2015 at 4:21 AM, Aggarwal, Nikunj
> mailto:nikunj.aggar...@hp.com>> wrote:
> 
> Hi, 
> 
> __ __
> 
> I have submitted  presentations for Openstack L summit :
> 
> __ __
> 
> The Heat Orchestration Template Builder: A demonstration
> 
> 
> 
> __ 
> 
> 
> Hi
> 
> Nice to see work on a HOT builder progressing, but..
> are you planning on integrating this with the other HOT builder efforts?
> Is the code public (link)?
> 
> This is more of a framework to make these easier to build:
> https://github.com/stackforge/merlin
> https://wiki.openstack.org/wiki/Merlin
> 
> Timur (who works on Merlin) is working with Rackers to build this
> upstream - I am not sure on the progress.
> https://github.com/rackerlabs/hotbuilder
> https://github.com/rackerlabs/foundry
>  
> It would be nice if we could all work together (I am hoping you are
> already)?
> Hopefully some of the others that are working on this can chip in and
> say where they
> are.
> 
> -Angus
> 
> __
> 
> __ __
> 
> Please cast your vote if you feel it is worth for presentation.
> 
> __ __
> 
> Thanks & Regards,
> 
> Nikunj
> 
> __ __
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Yves-Gwenaël Bourhis

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Vote for talk on using Metatemplates for heat

[Anita Kuno]

On 02/23/2015 11:16 AM, Pratik Mallya wrote:
> Hey Folks,
> 
> In case you missed it, many of you might find the work we’ve done in 
> maintaing catalogs of heat templates interesting.
> The proposal is here: 
> https://www.openstack.org/vote-vancouver/presentation/one-heat-template-to-rule-them-all
> 
> Please take a look if you haven’t already, and vote accordingly :-).
> 
> Thanks,
> Pratik
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
Please don't spam the list with "vote for me" requests like this.

The community knows how to evaluate talk proposals and vote. If they
don't they are welcome to ask in #openstack-dev if they need support.

Posts like this don't add to our overall community benefit or discussion.

Thank you,
Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable][requirements] External dependency caps introduced in 499db6b

[Joe Gordon]

On Mon, Feb 23, 2015 at 8:49 AM, Ihar Hrachyshka 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 02/20/2015 07:16 PM, Joshua Harlow wrote:
> > Sean Dague wrote:
> >> On 02/20/2015 12:26 AM, Adam Gandelman wrote:
> >>> Its more than just the naming.  In the original proposal,
> >>> requirements.txt is the compiled list of all pinned deps
> >>> (direct and transitive), while
> >>> requirements.in  reflects what people
> >>> will actually use.  Whatever is in requirements.txt affects the
> >>> egg's requires.txt. Instead, we can keep requirements.txt
> >>> unchanged and have it still be the canonical list of
> >>> dependencies, while
> >>> reqiurements.out/requirements.gate/requirements.whatever is an
> >>> upstream utility we produce and use to keep things sane on our
> >>> slaves.
> >>>
> >>> Maybe all we need is:
> >>>
> >>> * update the existing post-merge job on the requirements repo
> >>> to produce a requirements.txt (as it does now) as well the
> >>> compiled version.
> >>>
> >>> * modify devstack in some way with a toggle to have it process
> >>> dependencies from the compiled version when necessary
> >>>
> >>> I'm not sure how the second bit jives with the existing
> >>> devstack installation code, specifically with the libraries
> >>> from git-or-master but we can probably add something to warm
> >>> the system with dependencies from the compiled version prior to
> >>> calling pip/setup.py/etc 
> >>
> >> It sounds like you are suggesting we take the tool we use to
> >> ensure that all of OpenStack is installable together in a unified
> >> way, and change it's installation so that it doesn't do that any
> >> more.
> >>
> >> Which I'm fine with.
> >>
> >> But if we are doing that we should just whole hog give up on the
> >> idea that OpenStack can be run all together in a single
> >> environment, and just double down on the devstack venv work
> >> instead.
> >
> > It'd be interesting to see what a distribution (canonical,
> > redhat...) would think about this movement. I know yahoo! has been
> > looking into it for similar reasons (but we are more flexibly then
> > I think a packager such as canonical/redhat/debian/... would/culd
> > be). With a move to venv's that seems like it would just offload
> > the work to find the set of dependencies that work together (in a
> > single-install) to packagers instead.
> >
> > Is that ok/desired at this point?
> >
>
> Honestly, I failed to track all the different proposals. Just saying
> from packager perspective: we absolutely rely on requirements.txt not
> being a list of hardcoded values from pip freeze, but present us a
> reasonable freedom in choosing versions we want to run in packaged
> products.
>
>
in short the current proposal for stable branches is:

keep requirements.txt as is, except maybe put some upper bounds on the
requirements.

Add requirements.gate to specify the *exact* versions we are gating against
(this would be a full list including all transitive dependencies).


> That's why I asked before we should have caps and not pins.
>
> /Ihar
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJU61oJAAoJEC5aWaUY1u57T7cIALySnlpLV0tjrsTH2gZxskH+
> zY+L6E/DukFNZsWxB2XSaOuVdVaP3Oj4eYCZ2iL8OoxLrBotiOYyRFH29f9vjNSX
> h++dErBr0SwIeUtcnEjbk9re6fNP6y5Hqhk1Ac+NSxwL75KlS3bgKnJAhLA08MVB
> 5xkGRR7xl2cuYf9eylPlQaAy9rXPCyyRdxZs6mNjZ2vlY6hZx/w/Y7V28R/V4gO4
> qsvMg6Kv+3urDTRuJdEsV6HbN/cXr2+o543Unzq7gcPpDYXRFTLkpCRV2k8mnmA1
> pO9W10F1FCQZiBnLk0c6OypFz9rQmKxpwlNUN5MTMF15Et6DOxGBxMcfr7TpRaQ=
> =WHOH
> -END PGP SIGNATURE-
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] ML2 versus core plugin for OVN

[Ben Pfaff]

[branching off a discussion on ovs-dev at this point:
http://openvswitch.org/pipermail/dev/2015-February/051609.html]

On Fri, Feb 20, 2015 at 6:56 PM, Kyle Mestery  wrote:
> One thing to keep in mind, this ties somewhat into my response to Russell
> earlier on the decision around ML2 vs. core plugin. If we do ML2, there are
> type drivers for VLAN, VXLAN, and GRE tunnels. There is no TypeDriver for
> STT tunnels upstream now. It's just an item we need on the TODO list if we
> go down the STT tunnel path.

It was suggested to me off-list that this part of the discussion should be on
openstack-dev, so here it is ;-)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Outcome of the nova FFE meeting for Kilo

[Dan Smith]

> What's the alternative proposed release model?

I deleted a couple paragraphs of the above before sending this, thinking
(like Joe) that there probably needs to be a specific discussion aimed
at this topic. But:

> What's the compatibility story with Glance / Neutron / Cinder in
> whatever that model is?

I think we end up making the tight integrations where we need to.
Presumably Nova and Neutron still need to coordinate tightly. I'm not
sure Nova and Glance do at this point.

> What's the sliding upgrade window look like?

Right now we're maintaining some level of RPC compatibility to a point
far earlier than Juno. The six month releases give us the *only* points
at which we can (currently, by our existing rules) break compatibility.
Presumably we just need to define the minimum timeframe, and then
iterate our interfaces when it makes sense (and fits a policy). Maybe
that's six months; maybe longer, maybe shorter.

> What's the stable maint story look like? the security fix story?

Aren't you the one that wants to get rid of stable-maint? How we handle
security issues is certainly something worthy of a lot of discussion. If
the community only maintains a single stream, then the answer is "move
forward". If the only people maintaining slower-moving streams are
vendors or distros then I'd say the answer is "ask your vendor."

> I get being frustrated with a thing, but there are a lot of
> considerations before redoing the release cycle.

Well, I'll refer to my original words:

> On 02/23/2015 03:45 PM, Dan Smith wrote:
>> I've been wondering this myself for quite a while now. I'm really
>> interested to hear what things would look like in a no-release model.

That's all. Just interested. I do think it's the sane way forward,
long-term.

--Dan



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Outcome of the nova FFE meeting for Kilo

[Jay Pipes]

On 02/23/2015 04:02 PM, Sean Dague wrote:

On 02/23/2015 03:45 PM, Dan Smith wrote:

Seriously, what is the point of 6-month releases again? We are a
free-form open source set of projects, with a lot of intelligent
engineers. Why are we stuck using an outdated release model?


I've been wondering this myself for quite a while now. I'm really
interested to hear what things would look like in a no-release model.
I'm sure it would be initially met with a lot of resistance, but I think
that in the end, it makes more sense to move to that sort of model and
let vendors/deployers more flexibly decide when to roll out new stuff
based on what has changed and what they value.


What's the alternative proposed release model?


Release at-will. No pre-determined date-based releases. Smaller list of 
features in each tagged release. Focus on API versioning, not releases.



What's the compatibility story with Glance / Neutron / Cinder in
whatever that model is?


Exactly the same as it is now: everything is based on the server 
supported API version and the python-xxxclient release that understands 
that server API version, and integrating support for that client release 
into Nova. There's really very little different at this point between 
our usage of python-neutronclient and our usage of the requests Python 
library. We should pin to a particular library version that we make use 
of the features from, and upgrade the pinning when we utilize 
functionality that is a newer release.



What's the sliding upgrade window look like?


Whatever we want to make it. :) With our RPC and object versioning 
functionality, we can drop support for some old RPC API and object 
versions after some arbitrary amount of time (check with operators and 
get sign off, then just do it).



What's the stable maint story look like? the security fix story?


"stable maintenance" should be entirely left up to the distributions, IMO.

Security fix story should be up to the distributions to track when their 
products are patched. Upstream should be concerned with quick fixes into 
the upstream branches and coordinated communication with distributions.



I get being frustrated with a thing, but there are a lot of
considerations before redoing the release cycle.


Understood completely. But I'm curious to see just how much grief we put 
ourselves through in trying to do date-based releases when our 
contributor community just isn't assembled in a way that makes those 
releases seem reasonable.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] tracking bugs superseded by blueprints

[Mike Scherbakov]

Bogdan,
I think we should keep bugs open and not "supersed" them by blueprint. I
see following reasons for it.

Often, we can find workaround in order to fix the bug. Even if bug
naturally seems to be falling into some blueprint's scope. Then problem is
that when you close the bug, you don't even try to think about workaround -
and project gets shipped with some serious gaps from release to release.

Another issue is that you lose real technical requirements for blueprints.
If you keep bugs open and associated with blueprint, you will pass by bugs
a few times before you deliver blueprint's functionality, and finally close
bugs if code covers bug's case. At least, I'd like it to be so.

Finally, QA and users will keep opening duplicates, as no one will be happy
with won't fix. You can vote for bug (by "affecting" it), and you can't for
blueprint in LP, unfortunately. This just keeps door open for getting
feedback.

I don't really see any cons of moving bugs into Won't fix state instead.

Examples of bugs which I would certainly avoid putting into Won't fix:
https://bugs.launchpad.net/bugs/1398817 - disable computes by default
during scale up
https://bugs.launchpad.net/fuel/+bug/1422856 - separate /var & /var/log on
master node

Thanks,

On Wed, Feb 18, 2015 at 8:46 PM, Andrew Woodward  wrote:

> Bogdan,
>
> Yes I think tracking the bugs like this would be beneficial. We should
> also link them from the BP so that the imperilmenter can track them. It
> adds "related blueprints" in the bottom of the right column under the
> subscribers so we probably should also edit the description so that the
> data is easy to see
>
> On Wed, Feb 18, 2015 at 8:12 AM, Bogdan Dobrelya 
> wrote:
>
>> Hello.
>> There is inconsistency in the triage process for Fuel bugs superseded by
>> blueprints.
>> The current approach is to set "won't fix" status for such bugs.
>> But there are some cases we should clarify [0], [1].
>>
>> I vote to not track superseded bugs separately and keep them as "won't
>> fix" but update the status back to "confirmed" in case of regression
>> discovered. And if we want to backport an improvement tracked by a
>> blueprint (just for an exceptional case) let's assign milestones for
>> related bugs.
>>
>> If we want to change the triage rules, let's announce that so the people
>> won't get confused.
>>
>> [0] https://bugs.launchpad.net/fuel/+bug/1383741
>> [1] https://bugs.launchpad.net/fuel/+bug/1422856
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Skype #bogdando_at_yahoo.com 
>> Irc #bogdando
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Andrew
> Mirantis
> Fuel community ambassador
> Ceph community
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Mike Scherbakov
#mihgen
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver

[Clark Boylan]

On Fri, Feb 20, 2015, at 11:24 AM, Philipp Marek wrote:
> Hi all,
> 
> this is a reflection of the discussion I just had on #openstack-infra;
> it's 
> about (re-)using the central CI infrastructure for our Open-Source DRBD 
> driver too.
> 
> 
> The current status is:
>  * The DRBD driver is already in Cinder, so DRBD-replicated Cinder
>  storage
>using iSCSI to the hypervisors does work out-of-the-box.
>  * The Nova-parts didn't make it in for Kilo; we'll try to get them into
>  L.
>  * I've got a lib/backends/drbd for devstack, that together with a
>  matching 
>local.conf can set up a node - at least for a limited set of 
>distributions (as DRBD needs a kernel module, Ubuntu/Debian via DKMS
>are 
>the easy way).
>[Please note that package installation is not yet done in this script 
>yet - I'm not sure whether I can/may/should simply add an 
>apt-repository.]
> 
> 
> Now, clarkb told me about two caveats:
> 
>   «Yup, so the two things I will start with is that multinode testing is 
>still really rudimentary, we only just got tempest sort of working
>with 
>it. So I might suggest running on a single node first to get the
>general 
>thing working.
You can follow the current state of multinode work at
https://review.openstack.org/#/c/141530/
> 
>The other thing is that we don't have the zuul code to vote with 
>a different account deployed/merged yet. So initially you could run
>your 
>job but it wouldn't vote against, say, cinder.»
The stack that adds the necessary zuul stuff ends with
https://review.openstack.org/#/c/121528/
> 
> 
> Cinder has a deadline for CI: March 19th; upon relaying that fact (resp. 
> nearly correct date) clarkb said
> 
>   «thats about 3 weeks... probably at least for the zuul thing.»
> 
> So, actually it's nearly 4 weeks, let's hope that it all works out.
> 
> 
> Actually, the multi-node testing will only be needed when we get the Nova 
> parts in, because then it would make sense to test (Nova) via both iSCSI 
> and the DRBD transport; for Cinder CI a single-node setup is sufficient.
> 
> 
> My remaining questions are:
>  * Is it possible to have our driver tested via the common
>  infrastructure?
I think it should be, initially just reporting against your devstack
plugin while things get sorted out then once zuul has the appropriate
bits reporting like a third party test account but running upstream.
>  * Is it okay to setup another apt-repository during the devstack run,
>to install the needed packages? I'm not sure whether our servers
>would simply be accessible, some firewall or filtering proxy could
>break such things easily.
It is not ok in a gating job to do that but that isn't your intention so
should be fine. That said, isn't drbd available in existing apt/yum
repos? What special sauce do you need to pull in?
>  * Apart from the cinder-backend script in devstack (which I'll have to 
>finish first, see eg. package installation), is any other information 
>needed from us?
Other than following this thread and writing that devstack plugin
probably not.
> 
> 
Hope this helps,
Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable][requirements] External dependency caps introduced in 499db6b

[Joe Gordon]

On Mon, Feb 23, 2015 at 11:04 AM, Doug Hellmann 
wrote:

>
>
> On Mon, Feb 23, 2015, at 12:26 PM, Joe Gordon wrote:
> > On Mon, Feb 23, 2015 at 8:49 AM, Ihar Hrachyshka 
> > wrote:
> >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA1
> > >
> > > On 02/20/2015 07:16 PM, Joshua Harlow wrote:
> > > > Sean Dague wrote:
> > > >> On 02/20/2015 12:26 AM, Adam Gandelman wrote:
> > > >>> Its more than just the naming.  In the original proposal,
> > > >>> requirements.txt is the compiled list of all pinned deps
> > > >>> (direct and transitive), while
> > > >>> requirements.in  reflects what people
> > > >>> will actually use.  Whatever is in requirements.txt affects the
> > > >>> egg's requires.txt. Instead, we can keep requirements.txt
> > > >>> unchanged and have it still be the canonical list of
> > > >>> dependencies, while
> > > >>> reqiurements.out/requirements.gate/requirements.whatever is an
> > > >>> upstream utility we produce and use to keep things sane on our
> > > >>> slaves.
> > > >>>
> > > >>> Maybe all we need is:
> > > >>>
> > > >>> * update the existing post-merge job on the requirements repo
> > > >>> to produce a requirements.txt (as it does now) as well the
> > > >>> compiled version.
> > > >>>
> > > >>> * modify devstack in some way with a toggle to have it process
> > > >>> dependencies from the compiled version when necessary
> > > >>>
> > > >>> I'm not sure how the second bit jives with the existing
> > > >>> devstack installation code, specifically with the libraries
> > > >>> from git-or-master but we can probably add something to warm
> > > >>> the system with dependencies from the compiled version prior to
> > > >>> calling pip/setup.py/etc 
> > > >>
> > > >> It sounds like you are suggesting we take the tool we use to
> > > >> ensure that all of OpenStack is installable together in a unified
> > > >> way, and change it's installation so that it doesn't do that any
> > > >> more.
> > > >>
> > > >> Which I'm fine with.
> > > >>
> > > >> But if we are doing that we should just whole hog give up on the
> > > >> idea that OpenStack can be run all together in a single
> > > >> environment, and just double down on the devstack venv work
> > > >> instead.
> > > >
> > > > It'd be interesting to see what a distribution (canonical,
> > > > redhat...) would think about this movement. I know yahoo! has been
> > > > looking into it for similar reasons (but we are more flexibly then
> > > > I think a packager such as canonical/redhat/debian/... would/culd
> > > > be). With a move to venv's that seems like it would just offload
> > > > the work to find the set of dependencies that work together (in a
> > > > single-install) to packagers instead.
> > > >
> > > > Is that ok/desired at this point?
> > > >
> > >
> > > Honestly, I failed to track all the different proposals. Just saying
> > > from packager perspective: we absolutely rely on requirements.txt not
> > > being a list of hardcoded values from pip freeze, but present us a
> > > reasonable freedom in choosing versions we want to run in packaged
> > > products.
> > >
> > >
> > in short the current proposal for stable branches is:
> >
> > keep requirements.txt as is, except maybe put some upper bounds on the
> > requirements.
> >
> > Add requirements.gate to specify the *exact* versions we are gating
> > against
> > (this would be a full list including all transitive dependencies).
>
> The gate syncs requirements into projects before installing them. Would
> we change the sync script for the gate to work from the
> requirements.gate file, or keep it pulling from requirements.txt?
>

We would only add requirements.gate for stable branches (because we don't
want to cap/pin  things on master). So I think the answer is sync script
should work for both.  I am not sure on the exact mechanics of how this
would work. Whoever ends up driving this bit of work (I think Adam G), will
have to sort out the details.


> Doug
>
> >
> >
> > > That's why I asked before we should have caps and not pins.
> > >
> > > /Ihar
> > > -BEGIN PGP SIGNATURE-
> > > Version: GnuPG v1
> > >
> > > iQEcBAEBAgAGBQJU61oJAAoJEC5aWaUY1u57T7cIALySnlpLV0tjrsTH2gZxskH+
> > > zY+L6E/DukFNZsWxB2XSaOuVdVaP3Oj4eYCZ2iL8OoxLrBotiOYyRFH29f9vjNSX
> > > h++dErBr0SwIeUtcnEjbk9re6fNP6y5Hqhk1Ac+NSxwL75KlS3bgKnJAhLA08MVB
> > > 5xkGRR7xl2cuYf9eylPlQaAy9rXPCyyRdxZs6mNjZ2vlY6hZx/w/Y7V28R/V4gO4
> > > qsvMg6Kv+3urDTRuJdEsV6HbN/cXr2+o543Unzq7gcPpDYXRFTLkpCRV2k8mnmA1
> > > pO9W10F1FCQZiBnLk0c6OypFz9rQmKxpwlNUN5MTMF15Et6DOxGBxMcfr7TpRaQ=
> > > =WHOH
> > > -END PGP SIGNATURE-
> > >
> > >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> _

[openstack-dev] [stable][all] Revisiting the 6 month release cycle

[Joe Gordon]

Was:
http://lists.openstack.org/pipermail/openstack-dev/2015-February/057578.html

There has been frustration with our current 6 month development cadence.
This is an attempt to explain those frustrations and propose a very rough
outline of a possible alternative.


Currently we follow a 6 month release cadence, with 3 intermediate
milestones (6 weeks apart), as explained here:
https://wiki.openstack.org/wiki/Kilo_Release_Schedule


Current issues

   - 3 weeks of feature freeze for all projects at the end of each cycle (3
   out of the 26 feature blocked)
   - 3 weeks of release candidates. Once a candidate is cut development is
   open for next release. While this is good in theory, not much work actually
   starts on next release.
   - some projects have non priority feature freezes and at Milestone 2 (so
   9 out of 26 weeks restricted in those projects)
   - vicious development circle
  - vicious circle:
 - big push right to land lots of features right before the release
 - a lot of effort is spent getting the release ready
 - after the release people are a little burnt out and take it easy
 until the next summit
 - Blueprints have to be re-discussed and re-approved for the next
 cycle
 - makes it hard to land blueprints early in the cycle casing the
 bug rush at the end of the cycle, see step 1
  - Makes it hard to plan things across a release
  - This actually destabilizes things right as we go into the
  stabilization period (We actually have great data on this too)
  - Means postponing blueprints that miss a deadline several months


Requirements for a new model

   - Keep 6 month release cadence. Not everyone is willing to deploy from
   trunk
   - Keep stable branches for at least 6 months. In order to test upgrades
   from the last stable branch, we need a stable branch to test against
   - Keep supporting continuous deployment. Some people really want to
   deploy from trunk


What We can drop

   - While we need to keep releasing a stable branch every 6 months, we
   don't have to do all of our development planning around it. We can treat it
   as just another milestone


I think a lot of the frustration with our current cadence comes out of the
big stop everything (development, planning etc.), and stabilize the release
process. Which in turn isn't really making things more stable. So I propose
we keep the 6 month release cycle, but change the development cycle from a
6 month one with 3 intermediate milestones to a 6 week one with a milestone
at the end of it.

What this actually means:

   - Stop approving blueprints for specific stable releases, instead just
   approve them and target them to milestones.
  - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just
  become 1,2,3,4,5,6,7,8,9 etc.
  - If something misses what was previously known as Kilo-3 it has to
  wait a week for what for milestone 4.
   - Development focuses on milestones only. So 6 week cycle with say 1
   week of stabilization, finish things up before each milestone
   - Process of cutting a stable branch (planning, making the branch, doing
   release candidates, testing etc.) is done by a dedicated stable branch
   team. And it should be done based on a specific milestone
   - Goal: Change the default development planning mode to ignore stable
   branches, and allow for us to think of things in terms of the number of
   milestones needed, not will it make the stable branch or not


Gotchas, questions etc:

   - Some developers will still care about getting a feature into a
   specific stable release, so we may still get a small rush for the milestone
   before each stable branch
   - Requires significantly more work from the stable maintenance team
   - Naming the stable branches becomes less fun, as we refer to the stable
   branches less
   - Since planning is continuous 6 month cycle for summits becomes a
   little strange. This is still an open ended question to me.



Anyway, that has been rattling around my head since Paris and I needed to
write it down somewhere. Thanks for reading. Thoughts, comments, concerns
welcome.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Glance] [all] python-glanceclient release 0.16.0

[Nikhil Komawar]

The python-glanceclient release management team is pleased to announce:
python-glanceclient version 0.16.0 has been released on Tuesday, Feb 24th 
around 05:50 UTC.

For more information, please find the details at:

https://launchpad.net/python-glanceclient/+milestone/v0.16.0

Please report the issues through launchpad:

https://bugs.launchpad.net/python-glanceclient

Thanks,
-Nikhil
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev