date:20151130


On 11/28/2015 02:48 PM, Doug Hellmann wrote:

Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:

Liaisons,

We're making good progress on adding reno to service projects as
we head to the Mitaka-1 milestone. Thank you!

We also need to add reno to all of the other deliverables with
changes that might affect deployers. That means clients and other
libraries, SDKs, etc. with configuration options or where releases
can change deployment behavior in some way. Now that most teams
have been through this conversion once, it should be easy to replicate
for the other repositories in a similar way.

Libraries have 2 audiences for release notes: developers consuming
the library and deployers pushing out new versions of the libraries.
To separate the notes for the two audiences, and avoid doing manually
something that we have been doing automatically, we can use reno
just for deployer release notes (changes in support for options,
drivers, etc.). That means the library repositories that need reno
should have it configured just like for the service projects, with
the separate jobs and a publishing location different from their
existing developer documentation. The developer docs can continue
to include notes for the developer audience.


I've had a couple of questions about this split for release notes. The
intent is for developer-focused notes to continue to come from commit
messages and in-tree documentation, while using reno for new and
additional deployer-focused communication. Most commits to libraries
won't need reno release notes.


This looks like unnecessary overcomplication. Why not use such a 
convenient tool for both kinds of release notes instead of having us 
invent and maintain one more place to put release notes, now for 
developers? It's already not so easy to explain reno to newcomers, this 
idea makes it even harder...




Doug



After we start using reno for libraries, the release announcement
email tool will be updated to use those same notes to build the
message in addition to looking at the git change log. This will be
a big step toward unifying the release process for services and
libraries, and will allow us to make progress on completing the
automation work we have planned for this cycle.

It's not necessary to add reno to the liberty branch for library
projects, since we tend to backport far fewer changes to libraries.
If you maintain a library that does see a lot of backports, by all
means go ahead and add reno, but it's not a requirement. If you do
set up multiple branches, make sure you have one page that uses the
release-notes directive without specifing a branch, as in the
oslo.config example, to build notes for the "current" branch to get
releases from master and to serve as a test for rendering notes
added to stable branches.

Thanks,
Doug



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

2015-11-30 Thread 少合冯

2015-11-30 16:19 GMT+08:00 Koniszewski, Pawel :

> 2015-11-30 14:45 GMT+08:00 Koniszewski, Pawel  >:
> > -Original Message-
> > From: Murray, Paul (HP Cloud) [mailto:pmur...@hpe.com]
> > Sent: Friday, November 27, 2015 4:29 PM
> > To: Daniel P. Berrange; Carlton, Paul (Cloud Services)
> > Cc: 少合冯; OpenStack Development Mailing List (not for usage questions);
> > John Garbutt; Koniszewski, Pawel; Jin, Yuntong; Feng, Shaohe; Qiao,
> Liyong
> > Subject: RE: [nova] [RFC] how to enable xbzrle compress for live
> migration
> >
> >
> >
> > > -Original Message-
> > > From: Daniel P. Berrange [mailto:berra...@redhat.com]
> > > Sent: 26 November 2015 17:58
> > > To: Carlton, Paul (Cloud Services)
> > > Cc: 少合冯; OpenStack Development Mailing List (not for usage
> > questions);
> > > John Garbutt; pawel.koniszew...@intel.com; yuntong@intel.com;
> > > shaohe.f...@intel.com; Murray, Paul (HP Cloud); liyong.q...@intel.com
> > > Subject: Re: [nova] [RFC] how to enable xbzrle compress for live
> > > migration
> > >
> > > On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> > > > Seems to me the prevailing view is that we should get live migration
> > > > to figure out the best setting for itself where possible.  There was
> > > > discussion of being able have a default policy setting that will
> > > > allow the operator to define balance between speed of migration and
> > > > impact on the instance.  This could be a global default for the
> > > > cloud with overriding defaults per aggregate, image, tenant and
> > > > instance as well as the ability to vary the setting during the
> migration
> > operation.
> > > >
> > > > Seems to me that items like compression should be set in
> > > > configuration files based on what works best given the cloud
> operator's
> > environment?
> > >
> > > Merely turning on use of compression is the "easy" bit - there needs
> > > to be a way to deal with compression cache size allocation, which
> > > needs to have some smarts in Nova, as there's no usable "one size fits
> > > all" value for the compression cache size. If we did want to hardcode
> > > a compression cache size, you'd have to pick set it as a scaling factor
> against
> > the guest RAM size.
> > > This is going to be very heavy on memory usage, so there needs careful
> > > design work to solve the problem of migration compression triggering
> > > host OOM scenarios, particularly since we can have multiple concurrent
> > > migrations.
> > >
> >
> >
> > Use cases for live migration generally fall into two types:
> >
> > 1. I need to empty the host (host maintenance/reboot)
> >
> > 2. I generally want to balance load on the cloud
> >
> > The first case is by far the most common need right now and in that case
> the
> > node gets progressively more empty as VMs are moved off. So the resources
> > available for caching etc. grow as the process goes on.
> >I'd rather say that these resources might shrink. You need to turn off one
> compute node, stack more VMs on remaining compute nodes and you need to
> allocate cache on both sides, source and destination.
>
> >why do we need on destination?
>
> XBZRLE sends only a delta over network and it works in two phases:
> compressing and decompressing. During compression the original page and
> updated page are XORed together and resulting information is passed over to
> the RLE algorithm - the output is the delta page which is sent over network
> to destination host. During decompression run length decodes each pair of
> symbol-counter and the original page is XORed with the result from the run
> length decoding - the output is the updated page. It means that it needs to
> allocate cache on source and destination node.
>

But I think the RAM on the destination is the  original  page .
Just decompression
with the delta.

It does not need extra cache.


> > The second case is less likely to be urgent from the operators point of
> view,
> > so doing things more slowly may not be a problem.
> >
> > So looking at how much resource is available at the start of a migration
> and
> > deciding then what to do on a per VM basis is probably not a bad idea.
> > Especially if we can differentiate between the two cases.
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][vpnaas] "SKIPPED: Neutron support is required" while running tox

2015-11-30 Thread Ihar Hrachyshka


bharath  wrote:


Hi,

when running tox "sudo -u stack -H tox -e api  
neutron.tests.api.test_vpnaas_extensions"
test cases are failing with Error " setUpClass  
(neutron.tests.api.test_vpnaas_extensions.VPNaaSTestJSON) ... SKIPPED:  
Neutron support is required"


I even tried setting tempest_config_dir as below, still i am hitting the  
issue.


[testenv:api]
basepython = python2.7
passenv = {[testenv]passenv} TEMPEST_CONFIG_DIR
setenv = {[testenv]setenv}
 OS_TEST_PATH=./neutron/tests/api
 TEMPEST_CONFIG_DIR={env:TEMPEST_CONFIG_DIR:/opt/stack/tempest/etc}
 OS_TEST_API_WITH_REST=1


Can someone help me out



Does your /opt/stack/tempest/etc have tempest configured to run tests for  
neutron?


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] IRC Bot issues

2015-11-30 Thread Joshua Hesketh

Hi all,

Freenode are currently experiencing a severe DDoS attack that are having an
effect on our bots. As such the meetbot, irc logging and gerrit watcher are
interminably available.

We expect the bots to resume their normal function once Freenode has
recovered. For now, meetings may have to be postponed or minuted by hand.

Cheers,
Josh
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread hao wang

Hi, Duncan

2015-11-30 15:54 GMT+08:00 Duncan Thomas :
> Hi WangHao
>
> This was quite thoroughly discussed during the early discussions on
> replication. The general statement was 'not yet'. Getting any kind of
> workable replication API has proven to be very, very difficult to get right
> - we won't know for another full cycle whether we've actually gotten it
> somewhere near right, as operators start to deploy it. Piling more feature
> in the replication API before a) it has been used by operators and b)
> storage vendors have implemented what we already has would IMO be a mistake.

I agree with you, in my mind, using replication what we have is first
thing we should done,
improve it much better is second thing, and then we will add another
new features
one by one stably.

> None of this means that more DR interfaces don't belong in cinder, just that
> getting them right, getting them universal and getting them useful is quite
> a hard problem, and not one we should be in a rush to solve. Particularly as
> DR and replication is still a niche area of cinder, and we still have major
> issues in our basic functionality.

Yes, this will convince me about DR in Cinder, very clearly, thanks.
>
> On 30 November 2015 at 03:45, hao wang  wrote:
>>
>> Sean and Michal,
>>
>> In fact, there is a reason that I ask this question. Recently I have a
>> confusion about if cinder should provide the ability of Disaster
>> Recovery to storage resources, like volume. I mean we have volume
>> replication v1&v2, but for DR, specially DR between two independent
>> OpenStack sites(production and DR site), I feel we still need more
>> features to support it, for example consistency group for replication,
>> etc. I'm not sure if those features belong in Cinder or some new
>> project for DR.
>>
>> BR
>> WangHao
>>
>> 2015-11-30 3:02 GMT+08:00 Sean McGinnis :
>> > On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>> >> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>> >> > Hi guys,
>> >> >
>> >> > I notice nova have a clarification of project scope:
>> >> > http://docs.openstack.org/developer/nova/project_scope.html
>> >> >
>> >> > I want to find cinder's, but failed,  do you know where to find it?
>> >> >
>> >> > It's important to let developers know what feature should be
>> >> > introduced into cinder and what shouldn't.
>> >> >
>> >> > BR
>> >> > Wang Hao
>> >>
>> >> I believe Nova team needed to formalize the scop to have an explanation
>> >> for all the "this doesn't belong in Nova" comments on feature requests.
>> >> Does Cinder suffer from similar problems? From my perspective it's not
>> >> critically needed.
>> >
>> > I agree. I haven't seen a need for something like that with Cinder. Wang
>> > Hao, is there a reason you feel you need that?
>> >
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> --
> Duncan Thomas
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Magnum] Liberty RPMs for RDO

2015-11-30 Thread Mathieu Velten

Hi,

Let me first introduce myself : I am currently working at CERN to help
evaluate and deploy Magnum.

In this regard Ricardo recently sends an email regarding Puppet
modules, this one is about RPMs of Magnum for CentOS with RDO.

You can find here a repository containing the source and binary RPMs
for magnum and python-magnumclient.
http://linuxsoft.cern.ch/internal/repos/magnum7-testing/

The version 1.0.0.0b2.dev4 is the Magnum Liberty release and the
1.1.0.0-5 version is the Mitaka M1 release using Liberty dependencies
(one client commit regarding keystone auth and one server commit
regarding oslo.config have been reverted).

Let me know how I can contribute the spec files to somewhere more
suitable.

Regards,

Mathieu Velten
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][RFC] delete instance device

2015-11-30 Thread 少合冯

Hi all,

I'd like to talk about the delete instance device of nova.

Here is the libvirt doc string to describe it underly function
detachDeviceFlags.
http://paste.openstack.org/show/480330/

It says:

detaching a device from a running domain may be asynchronous.

*and it suggests:*

To check whether the device was successfully removed, either recheck domain

configuration using virDomainGetXMLDesc() or add handler for

VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED


Also Daniel elaborated it, and gave us more some scenarios about it.

it is not guaranteed to succeed. What happens is that the hypervisor
injects an ACPI request to unplug the device. The guest OS must co-operate
by releasing the device, before the hypervisor will complete the action of
physically removing it. So you require a guest OS that supports ACPI unplug
of course, and if the guest is crashed or being malicious there is no
guarantee the unplug will succeed. Libvirt will wait a short while for
success, but you must monitor for libvirt events to see if/when it finally
completes. This delayed release has implications for when Nova can mark the
PCI device as unused and available for other guests to assign.


Now I have checked the code, both detach volume or detach interface call
   5   1220  nova/virt/libvirt/driver.py <>
 guest . detach_device ( conf , persistent = True , live = live
)
   6   1280  nova/virt/libvirt/driver.py <>
 guest . detach_device ( cfg , persistent = True , live = live )
   7   3016  nova/virt/libvirt/driver.py <<_detach_pci_devices>>
 guest . detach_device ( self . _get_guest_pci_device ( dev ) ,
live = True )
   8   3105  nova/virt/libvirt/driver.py <<_detach_sriov_ports>>
 guest . detach_device ( cfg , live = True )

And for detach_interface in nova/compute/manager.py:

@wrap_exception()
@wrap_instance_fault
def detach_interface(self, context, instance, port_id):
"""Detach an network adapter from an instance."""
network_info = instance.info_cache.network_info
condemned = None
for vif in network_info:
if vif['id'] == port_id:
condemned = vif
break
if condemned is None:
raise exception.PortNotFound(_("Port %s is not "
   "attached") % port_id)
try:
self.driver.detach_interface(instance, condemned)
except exception.NovaException as ex:
LOG.warning(_LW("Detach interface failed, port_id=%(port_id)s,"
" reason: %(msg)s"),
{'port_id': port_id, 'msg': ex}, instance=instance)
raise exception.InterfaceDetachFailed(instance_uuid=instance.uuid)
else:
try:
self.network_api.deallocate_port_for_instance(
context, instance, port_id)
except Exception as ex:
with excutils.save_and_reraise_exception():
# Since this is a cast operation, log the failure for
# triage.
LOG.warning(_LW('Failed to deallocate port %(port_id)s '
'for instance. Error: %(error)s'),
{'port_id': port_id, 'error': ex},
instance=instance)



It just detach_interface, no double check the device is detached finally.

Now I will support the detach SRIOV code.
https://review.openstack.org/#/c/139910/
I'm not sure should I need to double check the device is finally detached.

If yes.

What should I support?

3 options:
1. Just ignored it. key the nova code.

2. sync check.
as the libvirt suggests: use virDomainGetXMLDesc()

def detach_interface(self, context, instance, port_id):

   self.driver.detach_interface(instance, condemned)

   # just *pseudo-code*

*   for i in range(1, 51):*

  if not(virDomainGetXMLDesc()):

   sleep(1)

   else if i == 51：

   raise exception

   else:

   break

   self.network_api.deallocate_port_for_instance(
   context, instance, port_id)



3. async notification.
 as the libvirt suggests:

add event handler for VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED.


call network_api.deallocate_port_for_instance in a backend task.
The backend receives the event result from event handler by AMPQ and filter
the
device is the the expected interface device, not the volume device.

Then backend call network_api.deallocate_port_for_instance to deallocate
the port.


I have not check the volume detach, not sure it has the same issues.

Beside this issue:
But from the libvirt Doc string,

hypervisors may prevent this operation if there is a currentblock copy
operation on the device being detached;
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsub

Re: [openstack-dev] 答复: [congress]role in Congress

2015-11-30 Thread Masahito MUROI


Hi Yali,

I'm not sure which your concern is about UI by Horizon or for a policy 
in Congress.


On 2015/11/30 16:27, zhangyali (D) wrote:

Hi Tim,

In the implementation of Congress UI, I have realized that we need to
add role assignment. In many systems and cases, RBAC (Role-Based Access
Control) is an vital function which are beneficial to the division of work.

I think we should have “admin” and “tenant” user at least.  The admin
user could define predicates or complicated relations used in the
violations, and tenant user could just use the predicates defined by
admin user.

A typical example is that, the owner of network connected to VM should
be in the same group with the owner of this VM. In this example,
same_group is a predicates or complicated relations. It should be
I think this depends on how admin writes policy and what service works 
as Congress datasource. If admin want to adopt the policy to specific 
tenant, user or group, admin writes an additional policy to affect them 
by the policy.




defined by admin user. And this predicate could be used by any tenant
user. In this way, some typical or common predicates could be defined in
a individual admin board, and other tenant user could choose which one
to use.
On the other hand, this mentions about which user has permission to edit 
and to see a policy by Horizon.


best regard,
Masahito



Yali

*发件人:*Tim Hinrichs [mailto:t...@styra.com]
*发送时间:*2015年11月30日10:37
*收件人:*zhangyali (D); OpenStack Development Mailing List (not for
usage questions)
*主题:*Re: [openstack-dev][congress]role in Congress

Hi Yali,

We didn't discuss role assignment for service users in Tokyo.  Could you
explain a bit more what you need?

Tim

On Sun, Nov 29, 2015 at 7:33 PM zhangyali (D) mailto:zhangyali...@huawei.com>> wrote:

Hi Tim and All,

I remember there is a topic named “role assignment for service
users”in the Tokyo Summit. Since I have not heard any message of
this topic. Does anyone could contribute some information for me? I
think it is vital for my design of Congress UI in horizon. Thanks a lot!

Best Regards,

Yali



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
室井 雅仁(Masahito MUROI)
Software Innovation Center, NTT
Tel: +81-422-59-4539，FAX: +81-422-59-2699



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] Autoscaling both clusters and containers

2015-11-30 Thread Peng Zhao


Ryan,
That's where Hyper could help. This blog talks about wasted capacity issue and
the solution:
http://thenewstack.io/hypernetes-brings-multi-tenancy-microservices/
Best Peng
- Hyper - Make VM run like 
Container


On Wed, Nov 18, 2015 at 6:03 AM, Ryan Rossiter < rlros...@linux.vnet.ibm.com > 
wrote:
Hi all,

I was having a discussion with a teammate with respect to container scaling. He
likes the aspect of nova-docker that allows you to scale (essentially)
infinitely almost instantly, assuming you are using a large pool of compute
hosts. In the case of Magnum, if I'm a container user, I don't want to be paying
for a ton of vms that just sit idle, but I also want to have enough vms to
handle my scale when I infrequently need it. But above all, when I need scale, I
don't want to suddenly have to go boot vms and wait for them to start up when I
really need it.

I saw [1] which discusses container scaling, but I'm thinking we can take this
one step further. If I don't want to pay for a lot of vms when I'm not using
them, could I set up an autoscale policy that allows my cluster to expand when
my container concentration gets too high on my existing cluster? It's kind of a
case of nested autoscaling. The containers are scaled based on request demand,
and the cluster vms are scaled based on container count.

I'm unsure of the details of Senlin, but at least looking at Heat autoscaling
[2], this would not be very hard to add to the Magnum templates, and we would
forward those on through the bay API. (I figure we would do this through the
bay, not baymodel, because I can see similar clusters that would want to be
scaled differently).

Let me know if I'm totally crazy or if this is a good idea (or if you guys have
already talked about this before). I would be interested in your feedback.

[1] http://lists.openstack.org/pip ermail/openstack-dev/2015-Nove 
mber/078628.html
[2] https://wiki.openstack.org/wik i/Heat/AutoScaling#AutoScaling _API

--
Thanks,

Ryan Rossiter (rlrossit)


__ __ __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.op enstack.org?subject:unsubscrib e
http://lists.openstack.org/cgi -bin/mailman/listinfo/openstac k-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Steven Hardy

On Mon, Nov 30, 2015 at 10:03:29AM +0100, Lennart Regebro wrote:
> I'm tasked to implement a command that shows error messages when a
> deployment has failed. I have a vague memory of having seen scripts
> that do something like this, if that exists, can somebody point me in
> teh right direction?

I wrote a super simple script and put it in a blog post a while back:

http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html

All it does is find the failed SoftwareDeployment resources, then do heat
deployment-show on the resource, so you can see the stderr associated with
the failure.

Having tripleoclient do that by default would be useful.

> Any opinions on what that should do, specifically? Traverse failed
> resources to find error messages, I assume. Anything else?

Yeah, but I think for this to be useful, we need to go a bit deeper than
just showing the resource error - there are a number of typical failure
modes, and I end up repeating the same steps to debug every time.

1. SoftwareDeployment failed (mentioned above).  Every time, you need to
see the name of the SoftwareDeployment which failed, figure out if it
failed on one or all of the servers, then look at the stderr for clues.

2. A server failed to build (OS::Nova::Server resource is FAILED), here we
need to check both nova and ironic, looking first to see if ironic has the
node(s) in the wrong state for scheduling (e.g nova gave us a no valid
host error), and then if they are OK in ironic, do nova show on the failed
host to see the reason nova gives us for it failing to go ACTIVE.

3. A stack timeout happened.  IIRC when this happens, we currently fail
with an obscure keystone related backtrace due to the token expiring.  We
should instead catch this error and show the heat stack status_reason,
which should say clearly the stack timed out.

If we could just make these three cases really clear and easy to debug, I
think things would be much better (IME the above are a high proportion of
all failures), but I'm sure folks can come up with other ideas to add to
the list.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][taas] discussion about port security setting

2015-11-30 Thread Soichi Shigeta



 Hi,

   I have a discussion about port security setting.

 The recommended sequence of operations:
  1. Create a neutron port (with port security disabled).
  2. Launch the monitoring VM and attach it to this port.
  3. Create a tap-service instance whose destination port
 is the monitoring VM's port.

 But, a monitoring VM can receive mirrored packets without
 disabling port security in our site.

 What I found:
  1) In case of port security is enabled, entries to enforce
 anti IP spoofing are set into iptables of a linux bridge
 when a VM is launched.

 It looks like this:
  INPUT:
   Chain neutron-openvswi-s12345678-9 (1 references)
   RETURN  all  -- 192.168.1.10 anywhere  MAC aa:bb:cc:dd:ee:ff /* 
Allow traffic from defined IP/MAC pairs. */
   DROPall  -- anywhere  anywhere  /* Drop traffic without 
an IP/MAC allow rule. */


 Note that these entries are effective for only egress
 direction from the VM.

  2) On the other hand, mac learning mechanism will drop
 ingress packets if destination mac address doesn't match
 the monitoring VM.

 During tap-service creation process, mac address learning
 is disabled (at line 251 in 
neutron_taas/services/taas/drivers/linux/ovs_taas.py).

 Therefore, a monitoring VM can receive mirrored packets
 from source VMs.

 As a result, I think the 1st operation (disabling port security)
 is not required for a monitoring VM to receive mirrored packets.

 Is my understand right?

 Regards,
 Soichi Shigeta



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

2015-11-30 Thread Koniszewski, Pawel

> -Original Message-
> From: Murray, Paul (HP Cloud) [mailto:pmur...@hpe.com]
> Sent: Friday, November 27, 2015 4:29 PM
> To: Daniel P. Berrange; Carlton, Paul (Cloud Services)
> Cc: 少合冯; OpenStack Development Mailing List (not for usage questions);
> John Garbutt; Koniszewski, Pawel; Jin, Yuntong; Feng, Shaohe; Qiao, Liyong
> Subject: RE: [nova] [RFC] how to enable xbzrle compress for live migration
>
>
>
> > -Original Message-
> > From: Daniel P. Berrange [mailto:berra...@redhat.com]
> > Sent: 26 November 2015 17:58
> > To: Carlton, Paul (Cloud Services)
> > Cc: 少合冯; OpenStack Development Mailing List (not for usage
> questions);
> > John Garbutt; pawel.koniszew...@intel.com; yuntong@intel.com;
> > shaohe.f...@intel.com; Murray, Paul (HP Cloud); liyong.q...@intel.com
> > Subject: Re: [nova] [RFC] how to enable xbzrle compress for live
> > migration
> >
> > On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> > > Seems to me the prevailing view is that we should get live migration
> > > to figure out the best setting for itself where possible.  There was
> > > discussion of being able have a default policy setting that will
> > > allow the operator to define balance between speed of migration and
> > > impact on the instance.  This could be a global default for the
> > > cloud with overriding defaults per aggregate, image, tenant and
> > > instance as well as the ability to vary the setting during the
migration
> operation.
> > >
> > > Seems to me that items like compression should be set in
> > > configuration files based on what works best given the cloud
operator's
> environment?
> >
> > Merely turning on use of compression is the "easy" bit - there needs
> > to be a way to deal with compression cache size allocation, which
> > needs to have some smarts in Nova, as there's no usable "one size fits
> > all" value for the compression cache size. If we did want to hardcode
> > a compression cache size, you'd have to pick set it as a scaling factor
against
> the guest RAM size.
> > This is going to be very heavy on memory usage, so there needs careful
> > design work to solve the problem of migration compression triggering
> > host OOM scenarios, particularly since we can have multiple concurrent
> > migrations.
> >
>
>
> Use cases for live migration generally fall into two types:
>
> 1. I need to empty the host (host maintenance/reboot)
>
> 2. I generally want to balance load on the cloud
>
> The first case is by far the most common need right now and in that case
the
> node gets progressively more empty as VMs are moved off. So the resources
> available for caching etc. grow as the process goes on.
>I'd rather say that these resources might shrink. You need to turn off one
compute node, stack more VMs on remaining compute nodes and you need to
allocate cache on both sides, source and destination.

>why do we need on destination?
>XBZRLE sends only a delta over network and it works in two phases:
>compressing and decompressing. During compression the original page and
>updated page are XORed together and resulting information is passed over to
>the RLE algorithm - the output is the delta page which is sent over network
>to destination host. During decompression run length decodes each pair of
>symbol-counter and the original page is XORed with the result from the run
>length decoding - the output is the updated page. It means that it needs to
>allocate cache on source and destination node.

>But I think the RAM on the destination is the  original  page . Just 
>decompression 
>with the delta.

>It does not need extra cache.

Ah, got it. Misunderstood conception - destination host is updated in every 
iteration so no need to store a cache there.

Then we need something smart in nova to decide whether xbzrle can be turned on. 
 

> The second case is less likely to be urgent from the operators point of
view,
> so doing things more slowly may not be a problem.
>
> So looking at how much resource is available at the start of a migration
and
> deciding then what to do on a per VM basis is probably not a bad idea.
> Especially if we can differentiate between the two cases.



smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum] Liberty RPMs for RDO

2015-11-30 Thread Alan Pevec

Hi Mathieu,

2015-11-30 10:54 GMT+01:00 Mathieu Velten :
> Hi,
>
> Let me first introduce myself : I am currently working at CERN to help
> evaluate and deploy Magnum.
>
> In this regard Ricardo recently sends an email regarding Puppet
> modules, this one is about RPMs of Magnum for CentOS with RDO.

Nice, looking forward to review it!

> You can find here a repository containing the source and binary RPMs
> for magnum and python-magnumclient.
> http://linuxsoft.cern.ch/internal/repos/magnum7-testing/

This one is 403 ?

> The version 1.0.0.0b2.dev4 is the Magnum Liberty release and the
> 1.1.0.0-5 version is the Mitaka M1 release using Liberty dependencies
> (one client commit regarding keystone auth and one server commit
> regarding oslo.config have been reverted).
>
> Let me know how I can contribute the spec files to somewhere more
> suitable.

Let's discuss this on rdo-list (CCed)

Cheers,
Alan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceph] DevStack plugin for Ceph required for Mitaka-1 and beyond?

Thanks for doing this, sorry I didn't catch the thread earlier, I was
off for holidays.

ironic / ceph / tempest will still work beyond M-1, as whitelisted
things. We do want them not using extras.d by the final Mitaka release.

-Sean

On 11/27/2015 07:29 PM, Sebastien Han wrote:
> The code is here: https://github.com/openstack/devstack-plugin-ceph
> 
>> On 25 Nov 2015, at 13:49, Sebastien Han  wrote:
>>
>> The patch just landed, as soon as I have the repo I’ll move the code and do 
>> some testing.
>>
>>> On 24 Nov 2015, at 16:20, Sebastien Han  wrote:
>>>
>>> Hi Ramana,
>>>
>>> I’ll resurrect the infra patch and put the project under the right 
>>> namespace.
>>> There is no plugin at the moment.
>>> I’ve figured out that this is quite urgent and we need to solve this asap 
>>> since devstack-ceph is used by the gate as well :-/
>>>
>>> I don’t think there is much changes to do on the plugin itself.
>>> Let’s see if we can make all of this happen before Mitaka-1… I highly doubt 
>>> but we’ll see…
>>>
 On 24 Nov 2015, at 15:31, Ramana Raja  wrote:

 Hi,

 I was trying to figure out the state of DevStack plugin
 for Ceph, but couldn't find its source code and ran into
 the following doubt. At Mitaka 1, i.e. next week, wouldn't
 Ceph related Jenkins gates (e.g. Cinder's gate-tempest-dsvm-full-ceph)
 that still use extras.d's hook script instead of a plugin, stop working?
 For reference,
 https://github.com/openstack-dev/devstack/commit/1de9e330de9fd509fcdbe04c4722951b3acf199c
 [Deepak, thanks for reminding me about the deprecation of extra.ds.]

 The patch that seeks to integrate Ceph DevStack plugin with Jenkins
 gates is under review,
 https://review.openstack.org/#/c/188768/
 It's outdated as the devstack-ceph-plugin it seeks to integrate
 seem to be in the now obsolete namespace, 'stackforge/', and hasn't seen
 activity for quite sometime.

 Even if I'm mistaken about all of this can someone please point me to
 the Ceph DevStack plugin's source code? I'm interested to know whether
 the plugin would be identical to the current Ceph hook script,
 extras.d/60-ceph.sh ?

 Thanks,
 Ramana

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> Cheers.
>>> 
>>> Sébastien Han
>>> Senior Cloud Architect
>>>
>>> "Always give 100%. Unless you're giving blood."
>>>
>>> Mail: s...@redhat.com
>>> Address: 11 bis, rue Roquépine - 75008 Paris
>>>
>>
>>
>> Cheers.
>> 
>> Sébastien Han
>> Senior Cloud Architect
>>
>> "Always give 100%. Unless you're giving blood."
>>
>> Mail: s...@redhat.com
>> Address: 11 bis, rue Roquépine - 75008 Paris
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> Cheers.
> 
> Sébastien Han
> Senior Cloud Architect
> 
> "Always give 100%. Unless you're giving blood."
> 
> Mail: s...@redhat.com
> Address: 11 bis, rue Roquépine - 75008 Paris
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How were we going to remove soft delete again?

On 11/24/2015 11:36 AM, Matt Riedemann wrote:
> I know in Vancouver we talked about no more soft delete and in Mitaka
> lxsli decoupled the nova models from the SoftDeleteMixin [1].
> 
> From what I remember, the idea is to not add the deleted column to new
> tables, to not expose soft deleted resources in the REST API in new
> ways, and to eventually drop the deleted column from the models.
> 
> I bring up the REST API because I was tinkering with the idea of
> allowing non-admins to list/show their (soft) deleted instances [2].
> Doing that, however, would expose more of the REST API to deleted
> resources which makes it harder to remove from the data model.
> 
> My question is, how were we thinking we were going to remove the deleted
> column from the data model in a backward compatible way? A new
> microversion in the REST API isn't going to magically work if we drop
> the column in the data model, since anything before that microversion
> should still work - like listing deleted instances for the admin.
> 
> Am I forgetting something? There were a lot of ideas going around the
> room during the session in Vancouver and I'd like to sort out the
> eventual long-term plan so we can document it in the devref about
> policies so that when ideas like [2] come up we can point to the policy
> and say 'no we aren't going to do that and here's why'.
> 
> [1]
> http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
> 
> [2]
> https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances
> 
> 

On the API compat side. There is no contract with the user for how long
deleted items will be shown, or how old the deleted history is.

The contract semantics are:

* by default - don't show me anything that's deleted
* with a flag - ok, you can *also* show me deleted things

The one places soft deleted removal is going to be semantically
meaningful is on GET /servers/details?changes-since=

Because in that case it's used by most people as a journal, and includes
deleted instances. It's our janky event stream for servers (given that
we have no other real event stream). That would need something better
before we make that query not possible.

I also agree, it's not clear to me why removing soft delete off of
things like instances is hugely critical. I think getting rid of it as
default for new things is good. But some places it's pretty intrinsic to
the information people pull out.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ironic] do we really need websockify with numpy speedups?

On 11/26/2015 06:44 AM, Pavlo Shchelokovskyy wrote:
> Hi all,
> 
> I was long puzzled why devstack is installing numpy. Being a fantastic
> package itself, it has the drawback of taking about 4 minutes to compile
> its C extensions when installing on our gates (e.g. [0]). I finally took
> time to research and here is what I've found:
> 
> it is used only by websockify package (installed by AFAIK ironic and
> nova only), and there it is used to speed up the HyBi protocol. Although
> the code itself has a path to work without numpy installed [1], the
> setup.py of websockify declares numpy as a hard dependency [2].
> 
> My question is do we really need those speedups? Do we test any feature
> requiring fast HyBi support on gates? Not installing numpy would shave 4
> minutes off any gate job that is installing Nova or Ironic, which seems
> like a good deal to me.
> 
> If we decide to save this time, I have prepared a pull request for
> websockify that moves numpy requirement to "extras" [3]. As a
> consequence numpy will not be installed by default as dependency, but
> still possible to install with e.g. "pip install websockify[fastHyBi]",
> and package builders can also specify numpy as hard dependency for
> websockify package in package specs.

I went down this same path before. That masking is mandatory in the spec
- https://github.com/kanaka/websockify/pull/163

The right answer here is to get the wheel mirrors working in infra.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

On 11/25/2015 03:22 PM, Shamail wrote:
> Hi,
> 
>> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  wrote:
>>
>> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
>>> Hi everyone,
>>>
>>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs which
>>> discusses how one open-source project is encouraging contributions by new
>>> open-source contributors through a combination of a special tag (which is
>>> associated with work that is needed but can only be completed by someone
>>> who is a first-time contributor) and helpful comments in the review phase
>>> to ensure the contribution(s) eventually get merged.
>>>
>>> While reading the article, I immediately thought about our
>>> low-hanging-fruit bug tag which is used for a very similar purpose in "bug
>>> fixing" section of  the "how to contribute" page[2].  The low-hanging-fruit
>>> tag is used to identify items that are generally suitable for first-time or
>>> beginner contributors but, in reality, anyone can pick them up.
>>>
>>> I wanted to propose a new tag (or even changing the, existing, low-hanging
>>> fruit tag) that would identify items that we are reserving for first-time
>>> OpenStack contributors (e.g. a patch-set for the item submitted by someone
>>> who is not a first time contributor would be rejected)... The same article
>>> that Andrew shared mentions using an "up-for-grabs" tag which also
>>> populates the items at up-for-grabs[3] (a site where people looking to
>>> start working on open-source projects see entry-level items from multiple
>>> projects).  If we move forward with an exclusive tag for first-timers then
>>> it would be nice if we could use the up-for-grabs tag so that OpenStack
>>> also shows up on the list too.  Please let me know if this change should be
>>> proposed elsewhere, the tags are maintained in launchpad and the wiki I
>>> found related to bug tags[4] didn't indicate a procedure for submitting a
>>> change proposal.
>>
>> I like the idea of making bugs suitable for first-timers more
>> discoverable. I'm not sure we need to *reserve* any bugs for any class
>> of contributor. What benefit do you think that provides?
> I would have to defer to additional feedback here... 
> 
> My own perspective from when I was doing my first contribution is that it was 
> hard to find active "low-hanging-fruit" items.  Most were already 
> work-in-progress or assigned.

This was a direct consequence of us dropping the auto-abandoning of old
code reviews in gerrit. When a review is abandoned the bug is flipped
back to New instead of In Progress.

I found quite often people go and gobble up bugs assigning them to
themselves, but don't make real progress on them. Then new contributors
show up, and don't work on any of those issues because our tools say
someone is already on top of it.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][upgrade] Grenade multinode partial upgrade

On 11/25/2015 11:42 AM, Sean M. Collins wrote:
> The first run for the multinode grenade job completed.
> 
> http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/
> 
> I'm still getting my bearings in the grenade log output, but if I
> understand it correctly, after upgrading Neutron, when spawning a new
> instance we do not get successful connectivity. The odd part is we see
> the failure twice. Once when upgrading Nova[1], then once when upgrading
> Cinder[2]. I would have thought Grenade would have exited after just
> failing the first time. It did do a call to worlddump.

We're calling worlddump a bunch of times in grenade during success
operations to try to help track down why connectivity sometimes goes
away when we don't do any actions which we think should affect it. The
Nova operation succeeded, there is a successful ping at the end of it.

Cinder pinged, but also does ssh verification. That failed.

> 
> The odd part is that the first failure is a little strange - it pings
> then sleeps until it successfully gets a packet back. Which it does -
> but then it does a call to worlddump. Odd?
> 
> Anyway, then it goes on to upgrade cinder, then tries to create an
> instance and ssh in, then it fails and grenade exits completely.
> 
> I'll continue digging through the logs to see exactly why we don't get
> connectivity. I see some interesting warnings in the L3 agent log about
> cleaning up a non-existent router[3], which may be the culprit.
> 
> [1]: 
> http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_20_34_06_742
> 
> [2]: 
> http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_21_45_15_133
> 
> [3]: 
> http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/old/screen-q-l3.txt.gz?level=WARNING
> 


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][upgrade] Grenade multinode partial upgrade

On 11/25/2015 02:45 PM, Armando M. wrote:
> 
> 
> On 25 November 2015 at 11:33, Armando M.  > wrote:
> 
> 
> 
> On 25 November 2015 at 10:15, Korzeniewski, Artur
> mailto:artur.korzeniew...@intel.com>>
> wrote:
> 
> Yes, this file is complete fine. The Grenade is running smoke
> test before resource creation.
> The workflow is like this:
> 1. Install old devstack on main and subnode
> 2. Run tempest smoke
> 3. Create resources
> 4. Verify resources
> 5. Shutdown the services
> 6. Verify if shutdown does not affect the resources
> 7. Spin new devstack on main node, subnode stays in old version
> without any touch.
> 8. Upgrade the services - start the new code on main node
> 9. Run tempest smoke tests on upgraded main node - it in theory
> should validate if we are cross-version compatible (N and N+1)
> 10. Verify resources
> 11. Shutdown
> 
> 
> Thanks Artur, now it's clear...I can see the right versions etc.
> Having a good run to compare with helps a lot!
> 
> 
> One more thing:
> 
> http://logs.openstack.org/69/143169/60/experimental/gate-grenade-dsvm-neutron-multinode/7c05ff0/logs/grenade.sh.txt.gz#_2015-11-25_14_31_20_846
> 
> I see no tests running, is it normal?

The final tempest run is handled by devstack-gate, so it's not in the
grenade log, it's in console.html

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][vpnaas] "SKIPPED: Neutron support is required" while running tox

2015-11-30 Thread Paul Michali

There is an effort to move the VPN API tests from neutron to neutron-vpnaas
repo under https://review.openstack.org/#/c/211381.  You may want to help
the submitter getting those tests running as an alternative to trying to
get this working (unless there is some need to run this specific test -
older release or something).

Regards,

Paul Michali (pc_m)


On Mon, Nov 30, 2015 at 4:33 AM Ihar Hrachyshka  wrote:

> bharath  wrote:
>
> > Hi,
> >
> > when running tox "sudo -u stack -H tox -e api
> > neutron.tests.api.test_vpnaas_extensions"
> > test cases are failing with Error " setUpClass
> > (neutron.tests.api.test_vpnaas_extensions.VPNaaSTestJSON) ... SKIPPED:
> > Neutron support is required"
> >
> > I even tried setting tempest_config_dir as below, still i am hitting the
> > issue.
> >
> > [testenv:api]
> > basepython = python2.7
> > passenv = {[testenv]passenv} TEMPEST_CONFIG_DIR
> > setenv = {[testenv]setenv}
> >  OS_TEST_PATH=./neutron/tests/api
> >
> TEMPEST_CONFIG_DIR={env:TEMPEST_CONFIG_DIR:/opt/stack/tempest/etc}
> >  OS_TEST_API_WITH_REST=1
> >
> >
> > Can someone help me out
> >
>
> Does your /opt/stack/tempest/etc have tempest configured to run tests for
> neutron?
>
> Ihar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][library] CI gate for regressions detection in deployment data

2015-11-30 Thread Bogdan Dobrelya

On 20.11.2015 17:41, Bogdan Dobrelya wrote:
>> Hi,
>>
>> let me try to rephrase this a bit and Bogdan will correct me if I'm wrong
>> or missing something.
>>
>> We have a set of top-scope manifests (called Fuel puppet tasks) that we use
>> for OpenStack deployment. We execute those tasks with "puppet apply". Each
>> task supposed to bring target system into some desired state, so puppet
>> compiles a catalog and applies it. So basically, puppet catalog = desired
>> system state.
>>
>> So we can compile* catalogs for all top-scope manifests in master branch
>> and store those compiled* catalogs in fuel-library repo. Then for each
>> proposed patch CI will compare new catalogs with stored ones and print out
>> the difference if any. This will pretty much show what is going to be
>> changed in system configuration by proposed patch.
>>
>> We were discussing such checks before several times, iirc, but we did not
>> have right tools to implement such thing before. Well, now we do :) I think
>> it could be quite useful even in non-voting mode.
>>
>> * By saying compiled catalogs I don't mean actual/real puppet catalogs, I
>> mean sorted lists of all classes/resources with all parameters that we find
>> during puppet-rspec tests in our noop test framework, something like
>> standard puppet-rspec coverage. See example [0] for networks.pp task [1].
>>
>> Regards,
>> Alex
>>
>> [0] http://paste.openstack.org/show/477839/
>> [1] 
>> https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-network/networks.pp
> 
> Thank you, Alex.
> Yes, the composition layer is a top-scope manifests, known as a Fuel
> library modular tasks [0].
> 
> The "deployment data checks", is nothing more than comparing the
> committed vs changed states of fixtures [1] of puppet catalogs for known
> deployment paths under test with rspecs written for each modular task [2].
> 
> And the *current status* is:
> - the script for data layer checks now implemented [3]
> - how-to is being documented here [4]
> - a fix to make catalogs compilation idempotent submitted [5]

The status update:
- the issue [0] is the data regression checks blocker and is only the
Noop tests specific. It has been reworked to not use custom facts [1].
New uuid will be still generated each time in the catalog, but the
augeas ensures it will be processed in idempotent way. Let's make this
change [2] to the upstream puppet-nova as well please.

[0] https://bugs.launchpad.net/fuel/+bug/1517915
[1] https://review.openstack.org/251314
[2] https://review.openstack.org/131710

- pregenerated catalogs for the Noop tests to become the very first
committed state in the data regression process has to be put in the
*separate repo*. Otherwise, the stackalytics would go mad as that would
be a 600k-liner patch to an OpenStack project, which is the Fuel-library
now :)

So, I'm planning to use the separate repo for the templates. Note, we
could as well move the tests/noop/astute.yaml/ there. Thoughts?

> - and there is my WIP branch [6] with the initial committed state of
> deploy data pre-generated. So, you can checkout, make any test changes
> to manifests and run the data check (see the README [4]). It works for
> me, there is no issues with idempotent re-checks of a clean committed
> state or tests failing when unexpected.
> 
> So the plan is to implement this noop tests extention as a non-voting CI
> gate after I make an example workflow update for developers to the
> Fuel wiki. Thoughts?
> 
> [0]
> https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular
> [1]
> https://github.com/openstack/fuel-library/tree/master/tests/noop/astute.yaml
> [2] https://github.com/openstack/fuel-library/tree/master/tests/noop/spec
> [3] https://review.openstack.org/240015
> [4]
> https://github.com/openstack/fuel-library/blob/master/tests/noop/README.rst
> [5] https://review.openstack.org/247989
> [6] https://github.com/bogdando/fuel-library-1/commits/data_checks
> 
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

On 11/24/2015 06:01 AM, Claudiu Belu wrote:
> Hello,
> 
> Thanks Dims for raising the concern and Angus for reaching out. :)
> 
> Most of the time, python development on Windows is not too far off from
> Linux. But the two systems are quite different, which imply different
> modules (fcntl, pwd, grp modules do not exist in Windows) or different
> implementations of some modules (multiprocessing uses Popen instead of
> os.fork, os module is quite different) or some socket options and
> signals are different in Windows.
> 
> 1.a. As I've said earlier, some modules do not exist in Windows. All, or
> at least most standard modules document the fact that they are strictly
> for Linux. [1][2][3]
> b. At the very least, running the unit tests in a Windows environment
> can at least detect simple problems (e.g. imports). Secondly, there is a
> Hyper-V / Windows CI running on some of the OpenStack projects (nova,
> neutron, networking_hyperv, cinder) that can be checked before merging.
> 
> 2. This is a bit more complicated question. Well, for functions, you
> could have separate modules for Linux specific functions and Windows
> specific functions. This has been done before: [4] As for
> object-oriented implementations, I'd suggest having the system-specific
> calls be done in private methods, which will be overriden by Windows /
> Linux subclasses with their specific implementations. We've done
> something like this before, but solutions were pretty much
> straight-forward; it might not be as simple for oslo_privsep, since it
> is very Linux-specific.
> 
> 3. Typically, the OpenStack services on Hyper-V / Windows are run with
> users that have enough privileges to do their job. For example, the
> nova-compute service is run with a user that has Hyper-V Admin
> privileges and is not necessarily in the "Administrator" user group. We
> haven't used rootwrap in our usecases, it is disabled by default, plus,
> oslo.rootwrap imports pwd, which doesn't exist in Windows.

Right, so to me this seems that privsep just needs a NULL mode, and
we're done. If oslo.rootrwap was never supported on windows, I don't
think privsep really needs to be in a real way.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Remove nova-network as a deployment option in Fuel?

2015-11-30 Thread Aleksey Kasatkin

This remains unclear.
Now, for 8.0, the Environment with Nova-Network can be created but cannot
be deployed (and its creation is tested in UI integration tests).
AFAIC, we should either remove the ability of creation of environments with
Nova-Network in 8.0 or return it back into working state.


Aleksey Kasatkin


On Fri, Oct 23, 2015 at 3:42 PM, Sheena Gregson 
wrote:

> As a reminder: there are no individual networking options that can be used
> with both vCenter and KVM/QEMU hypervisors once we deprecate nova-network.
>
>
>
> The code for vCenter as a stand-alone deployment may be there, but the
> code for the component registry (
> https://blueprints.launchpad.net/fuel/+spec/component-registry) is still
> not complete.  The component registry is required for a multi-HV
> environment, because it provides compatibility information for Networking
> and HVs.  In theory, landing this feature will enable us to configure DVS +
> vCenter and Neutron with GRE/VxLAN + KVM/QEMU in the same environment.
>
>
>
> While Andriy Popyvich has made considerable progress on this story, I
> personally feel very strongly against deprecating nova-network until we
> have confirmed that we can support *all current use cases* with the
> available code base.
>
>
>
> Are we willing to lose the multi-HV functionality if something prevents
> the component registry work from landing in its entirety before the next
> release?
>
>
>
> *From:* Sergii Golovatiuk [mailto:sgolovat...@mirantis.com]
> *Sent:* Friday, October 23, 2015 6:30 AM
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [Fuel] Remove nova-network as a deployment
> option in Fuel?
>
>
>
> Hi,
>
>
>
> As far as I know neutron code for VCenter is ready. Guys are still testing
> it. Keep patience... There will be announce soon.
>
>
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Fuel Python Gerrit Dashboard

2015-11-30 Thread Igor Kalnitsky

Hey Fuelers,

As a part of improving review process, I've prepared a Gerrit
Dashboard for Python projects [1]. First and foremost, I did it for
myself (I believe it will help me to get attention to *ready-to-merge*
patches), but I want to share the link with you.

Feel free to use it!

[1] https://goo.gl/aFHdeQ

Thanks,
Igor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

On 11/24/2015 10:09 AM, John Garbutt wrote:
> On 24 November 2015 at 15:00, Balázs Gibizer
>  wrote:
>>> From: Andrew Laski [mailto:and...@lascii.com]
>>> Sent: November 24, 2015 15:35
>>> On 11/24/15 at 10:26am, Balázs Gibizer wrote:
> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> Sent: November 23, 2015 22:33
> On 11/23/2015 2:23 PM, Andrew Laski wrote:
>> On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
 From: Andrew Laski [mailto:and...@lascii.com]
 Sent: November 23, 2015 17:03

 On 11/23/15 at 08:54am, Ryan Rossiter wrote:
>
>
> On 11/23/2015 5:33 AM, John Garbutt wrote:
>> On 20 November 2015 at 09:37, Balázs Gibizer
>>  wrote:
>>> 
>>> 
>
>> There is a bit I am conflicted/worried about, and thats when we
>> start including verbatim, DB objects into the notifications. At
>> least you can now quickly detect if that blob is something
>> compatible with your current parsing code. My preference is
>> really to keep the Notifications as a totally separate object
>> tree, but I am sure there are many cases where that ends up
>> being seemingly stupid duplicate work. I am not expressing this
>> well in text form :(
> Are you saying we don't want to be willy-nilly tossing DB
> objects across the wire? Yeah that was part of the rug-pulling
> of just having the payload contain an object. We're
> automatically tossing everything with the object then, whether
> or not some of that was supposed to be a secret. We could add
> some sort of property to the field like
> dont_put_me_on_the_wire=True (or I guess a
> notification_ready() function that helps an object sanitize
> itself?) that the notifications will look at to know if it puts
> that on the wire-serialized dict, but that's adding a lot more
> complexity and work to a pile that's already growing rapidly.

 I don't want to be tossing db objects across the wire.  But I
 also am not convinced that we should be tossing the current
 objects over the wire either.
 You make the point that there may be things in the object that
 shouldn't be exposed, and I think object version bumps is another
 thing to watch out for.
 So far the only object that has been bumped is Instance but in
 doing so no notifications needed to change.  I think if we just
 put objects into notifications we're coupling the notification
 versions to db or RPC changes unnecessarily.  Some times they'll
 move together but other times, like moving flavor into
 instance_extra, there's no reason to bump notifications.
>>>
>>>
>>> Sanitizing existing versioned objects before putting them to the
>>> wire is not hard to do.
>>> You can see an example of doing it in
>>> https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
>>> cm
>>> L382.
>>> We don't need extra effort to take care of minor version bumps
>>> because that does not break a well written consumer. We do have to
>>> take care of the major version bumps but that is a rare event and
>>> therefore can be handled one by one in a way John suggested, by
>>> keep sending the previous major version for a while too.
>>
>> That review is doing much of what I was suggesting.  There is a
>> separate notification and payload object.  The issue I have is that
>> within the ServiceStatusPayload the raw Service object and version
>> is being dumped, with the filter you point out.  But I don't think
>> that consumers really care about tracking Service object versions
>> and dealing with compatibility there, it would be easier for them
>> to track the ServiceStatusPayload version which can remain
>> relatively stable even if Service is changing to adapt to db/RPC changes.
> Not only do they not really care about tracking the Service object
> versions, they probably also don't care about what's in that filter list.
>
> But I think you're getting on the right track as to where this needs
> to go. We can integrate the filtering into the versioning of the payload.
> But instead of a blacklist, we turn the filter into a white list. If
> the underlying object adds a new field that we don't want/care if
> people know about, the payload version doesn't have to change. But if
> we add something (or if we're changing the existing fields) that we
> want to expose, we then assert that we need to update the version of
> the payload, so the consumer can look at the payload and say "oh, in
> 1.x, now I get ___" and can add the appropriate checks/compat.
> Granted with this you can get into rebase nightmares ([1] still
> haunts me in my sleep),

Re: [openstack-dev] [Fuel] Fuel Python Gerrit Dashboard

2015-11-30 Thread Evgeniy L

Thanks Igor, it's very helpful,

Also if you forget where to get the link, use the documentation [1].
I think something like that may also be useful for library and QA team.

Thanks,

[1] https://wiki.openstack.org/wiki/Fuel#Development_related_links


On Mon, Nov 30, 2015 at 3:29 PM, Igor Kalnitsky 
wrote:

> Hey Fuelers,
>
> As a part of improving review process, I've prepared a Gerrit
> Dashboard for Python projects [1]. First and foremost, I did it for
> myself (I believe it will help me to get attention to *ready-to-merge*
> patches), but I want to share the link with you.
>
> Feel free to use it!
>
> [1] https://goo.gl/aFHdeQ
>
> Thanks,
> Igor
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer][aodh][vitrage] Raising custom alarms in AODH

2015-11-30 Thread AFEK, Ifat (Ifat)

Hi,

A few days ago I sent you this email (see below). Resending in case you didn't 
see it. 
If you could get back to me soon it would be most appreciated, as we are quite 
blocked with our AODH integration right now.

Thanks,
Ifat.

-Original Message-
From: AFEK, Ifat (Ifat) [mailto:ifat.a...@alcatel-lucent.com] 
Sent: Tuesday, November 24, 2015 7:37 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [ceilometer][aodh][vitrage] Raising custom alarms 
in AODH

Hi Gord, Hi Ryota,

(I sent the same mail again in a more readable format)

Thanks for your detailed responses.
Hope you don't mind that I'm sending one reply to both of your emails. I think 
it would be easier to have one thread for this discussion.

Let me explain our use case in more details. 
Here is an example of how we would like to integrate with AODH. Let me know 
what you think about it. 

1. Vitrage gets an alarm from Nagios about high cpu load on one of the hosts 

2. Vitrage evaluator decides (based on its templates) that an "instance might 
be suffering due to high cpu load on the host" alarm should be triggered for 
every instance on this host 

3. Vitrage notifier creates corresponding alarm definitions in AODH 

4. AODH stores these alarms in its database 

5. Vitrage triggers the alarms 

6. AODH updates the alarms states and notifies about it 

7. Horizon user queries AODH for a list of all alarms (we are currently 
checking the status of a blueprint that should implement it[2]). AODH returns a 
list that includes the alarms that were triggered by Vitrage.

8. Horizon user selects one of the alarms that Vitrage generated, and asks to 
see its root cause (we will create a new blueprint for that). Vitrage API 
returns the RCA information for this alarm.

Our current discussion is on steps 3-6 (as far as we understand, and please 
correct me if I'm wrong, nothing blocks the implementation of the blueprint for 
step 7).

Looking at AODH API again, here is what I think we need to do:

1. Define an alarm with an external_trigger_rule or something like that. This 
alarm has no metric data. We just want to be able to trigger it and query its 
state.

2. Use AODH API for triggering this alarm. Will "PUT 
/v2/alarms/(alarm_id)/state" do the job? 

Please see also my comments below.

Thanks,
Ifat.

[2] 
https://blueprints.launchpad.net/horizon/+spec/ceilometer-alarm-management-page 

> -Original Message-
> From: gord chung [mailto:g...@live.ca]
> Sent: Monday, November 23, 2015 9:45 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [ceilometer][aodh][vitrage] Raising 
> custom alarms in AODH
> 
> 
> 
> On 23/11/2015 11:14 AM, AFEK, Ifat (Ifat) wrote:
> > I guess I would like to do both: create a new alarm definition, then 
> > trigger it (call alarm_actions), and possibly later on set its state 
> > back to OK (call ok_action).
> > I understood that currently all alarm triggering is internal in 
> > AODH, according to threshold/events/combination alarm rules. Would 
> > it be possible to add a new kind of rule, that will allow triggering 
> > the alarm externally?
> what type of rule?
> 
> i have https://review.openstack.org/#/c/247211 which would 
> theoretically allow you to push an action into queue which would then 
> trigger appropriate REST call. not sure if it helps you plug into Aodh 
> easier or not?

We need to add an alarm definition with an "external_rule", and then trigger 
it. It is important for us that the alarm definition will be stored in AODH 
database for future queries. As far as I understand, the queue should help only 
with the triggering?

> 
> --
> gord

> -Original Message-
> From: Ryota Mibu [mailto:r-m...@cq.jp.nec.com]
> Sent: Tuesday, November 24, 2015 10:00 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [ceilometer][aodh][vitrage] Raising 
> custom alarms in AODH
> 
> Hi Ifat,
> 
> 
> Thank you for starting discussion how AODH can be integrated with 
> Vitrage that would be a good example of AODH integration with other 
> OpenStack components.
> 
> The key role of creating alarm definition is to set endpoint
> (alarm_actins) which can be receive alarm notification from AODH. How 
> the endpoints can be set in your use case? Those endpoints are 
> configured via virtage API and stored in its DB?

We have a graph database that will include resources and alarms imported from 
few sources of information (including Ceilometer), as well as alarms generated 
by Vitrage. However, we would like our alarms to be stored in AODH as well. If 
I understood you correctly, we will need the endpoints in order to be notified 
on Ceilometer alarms.

> 
> I agree with Gordon, you can use even-alarm with generating "event"
> containing alarming message that can be captured in aodh if vitrage 
> relay the alarm definition to aodh. That is more feasible way rather 
> than creati

Re: [openstack-dev] [nova] How were we going to remove soft delete again?

2015-11-30 Thread Alex Xu

2015-11-30 19:42 GMT+08:00 Sean Dague :

> On 11/24/2015 11:36 AM, Matt Riedemann wrote:
> > I know in Vancouver we talked about no more soft delete and in Mitaka
> > lxsli decoupled the nova models from the SoftDeleteMixin [1].
> >
> > From what I remember, the idea is to not add the deleted column to new
> > tables, to not expose soft deleted resources in the REST API in new
> > ways, and to eventually drop the deleted column from the models.
> >
> > I bring up the REST API because I was tinkering with the idea of
> > allowing non-admins to list/show their (soft) deleted instances [2].
> > Doing that, however, would expose more of the REST API to deleted
> > resources which makes it harder to remove from the data model.
> >
> > My question is, how were we thinking we were going to remove the deleted
> > column from the data model in a backward compatible way? A new
> > microversion in the REST API isn't going to magically work if we drop
> > the column in the data model, since anything before that microversion
> > should still work - like listing deleted instances for the admin.
> >
> > Am I forgetting something? There were a lot of ideas going around the
> > room during the session in Vancouver and I'd like to sort out the
> > eventual long-term plan so we can document it in the devref about
> > policies so that when ideas like [2] come up we can point to the policy
> > and say 'no we aren't going to do that and here's why'.
> >
> > [1]
> >
> http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
> >
> > [2]
> >
> https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances
> >
> >
>
> On the API compat side. There is no contract with the user for how long
> deleted items will be shown, or how old the deleted history is.
>
> The contract semantics are:
>
> * by default - don't show me anything that's deleted
> * with a flag - ok, you can *also* show me deleted things
>
> The one places soft deleted removal is going to be semantically
> meaningful is on GET /servers/details?changes-since=
>
> Because in that case it's used by most people as a journal, and includes
> deleted instances. It's our janky event stream for servers (given that
> we have no other real event stream). That would need something better
> before we make that query not possible.
>
> I also agree, it's not clear to me why removing soft delete off of
> things like instances is hugely critical. I think getting rid of it as
> default for new things is good. But some places it's pretty intrinsic to
> the information people pull out.
>

+1, for user to check out the history, query the deleted instance sounds
like useful.

For showing the deleted instance for non-admin user, looks like it's hard
for operator to make promise to user that how long the deleted instance can
be queried anymore. Currently the db archive just archive all the entries,
whatever it was deleted long time ago, or just in seconds. We probably can
add some filter to db archive call. But anyway this is another thing. Also
want to figure out why query deleted instance is useless first.


>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

2015-11-30 Thread Thierry Carrez

Sean Dague wrote:
> Right, so to me this seems that privsep just needs a NULL mode, and
> we're done. If oslo.rootrwap was never supported on windows, I don't
> think privsep really needs to be in a real way.

+1

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][messaging] configurable ack-then-process (at least/most once) behavior

2015-11-30 Thread Bogdan Dobrelya

Hello.
Please let's make this change [0] happen to the Oslo messaging.
This is reasonable, straightforward and backwards compatible change. And
it is required for OpenStack applications - see [1] - to implement a
sane HA. The only thing left is to cover this change by unit tests.

[0] https://review.openstack.org/229186
[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-October/076217.html

-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

2015-11-30 Thread Andrea Frittoli

Looking at the tool, it seems to me that is servers a combination of
functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test
resources, and have a CLI tool to provision them [0]. We could re-use code
from this tool to achieve that.

Support for distribution specific / cloud specific overrides is also
something that is useful. In clouds where I control the deployment process
I inject extra configs in tempest.conf based on the deployment options. In
clouds where I don't, I maintain a partial tempest.conf with the list of
options which I expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to
the bottom of tempest.conf and it's done. Duplicate configuration options
are not an issue, the last one wins. Still we could support specifying a
number of configuration files to the non-yet implemented "tempest run"
command.

Support for configuration override via CLI is something that I can see it
can be useful during development or troubleshooting, we could support that
as options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use
it today in the gate - i.e. fail fast if the generated configuration does
not match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the
functions implemented by it can fit into tempest - and some are already
there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa 
wrote:

> Hi,
>
> I agree with Ken'ichi's opinion, basically. Tempest users should know
> "what do we test for?" and we shouldn't discover values that we test for
> automatically.
> If users seems that "My current cloud is good. This is what I expect.",
> discovering function could work. But I suppose many of users would use
> tempest-config-generator for a new their cloud. So I feel the tool users
> could be misunderstanding easily.
>
> But I also think that tempest users don't need to know all of the
> configurations.
> So, how about something like introducing "a configuration wizard" for
> tempest configuration?
> This is a just idea, though..
>
>
> Anyway, if you have the motivation to introduce tempest-config, how about
> writing a spec for the feature for a concrete discussion?
> (I think we should have agreements about the target issues, users,
> solutions, etc.)
>
> Best Regards,
> -- Masayuki Igawa
>
>
> 2015年11月29日(日) 22:07 Yair Fried :
>
>> Hi,
>> I agree with Jordan.
>> We don't have to use the tool as part of the gate. It's target audience
>> is people and not CI systems. More specifically - new users.
>> However, we could add a gate (or a few) for the tool that makes sure a
>> proper conf file is generated. It doesn't have to run the tests, just
>> compare the output of the script to the conf generated by devstack.
>>
>> Re Rally - I believe the best place for tempest configuration script is
>> within tempest. That said, if the Tempest community doesn't want this tool,
>> we'll have to settle for the Rally solution.
>>
>> Regards
>> Yair
>>
>> On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <
>> jordan.pitt...@scality.com> wrote:
>>
>>> Hi,
>>> I think this script is valuable to some users: Rally and Red Hat
>>> expressed their needs, they seem clear.
>>>
>>> This tool is far from bullet proof and if used blindly or in case of
>>> bugs, Tempest could be misconfigured. So, we could have this tool inside
>>> the Tempest repository (in the tools/) but not use it at all for the Gate.
>>>
>>> I am not sure I fully understand the resistance for this, if we don"t
>>> use this config generator for the gate, what's the risk ?
>>>
>>> Jordan
>>>
>>> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi >> > wrote:
>>>
 2015-11-27 15:40 GMT+09:00 Daniel Mellado :
 > I still do think that even if there are some issues addressed to the
 > feature, such as skipping tests in the gate, the feature itself it's
 still
 > good -we just won't use it for the gates-
 > Instead it'd be used as a wrapper for a user who would be interested
 on
 > trying it against a real/reals clouds.
 >
 > Ken, do you really think a tempest user should know all tempest
 options?
 > As you pointed out there are quite a few of them and even if they
 should at
 > least know their environment, this script would set a minimum
 acceptable
 > default. Do you think PTL and Pre-PTL concerns that we spoke of would
 still
 > apply to that scenario?

 If Tempest users run part of tests of Tempest, they need to know the
 options which are used with these tests o

Re: [openstack-dev] [devstack] Devstack on wheezy

On 11/26/2015 10:52 AM, Paul Carlton wrote:
> Matt
> 
> Been working on getting devstack running in wheezy.
> 
> Hit one issue with a recent change to devstack
> 
> 202279c4e5a222018e40883ee6500f9d26690b48 202279c
> jenk...@review.openstack.org Fri, 20 Nov 2015 22:02:18 + Merge "Add
> liberasurecode-dev as a swift dependency"
> 2960bfa7f63c26d94c8aed2c3e1a6ff039b530f0 2960bfa tushar.go...@intel.com
> Tue, 17 Nov 2015 14:06:28 -0700 Add liberasurecode-dev as a swift
> dependency
> 
> liberasurecode-dev is not available on wheezy. Since we are meant to
> support wheezy (clearly we don't
> have a CI job to test with it) I will raise a bug for this but for now I
> just checked out a version of devstack
> prior to this change.

Wheezy support is best effort.

liberasurecode-dev is really about to be a hard dependency for Swift, we
need it in our development toolchain. Someone (upstream) needs to
provide that as a backport to wheezy if running swift on wheezy is still
a thing people care about.

> tgt version on wheezy was DOA so I installed a latter version from the
> backports repo.
> Added 'deb http://ftp.de.debian.org/debian wheezy-backports main' to my
> sources.list
> 
> sudo apt-get install tgt=1:1.0.51-1~bpo70+1
> ...
> The following packages have unmet dependencies:
>  tgt : Depends: librdmacm1 (>= 1.0.16) but 1.0.15-1+deb7u1 is to be
> installed
> E: Unable to correct problems, you have held broken packages.
> pcarlton@debian7a:~$ sudo apt-cache policy librdmacm1
> librdmacm1:
>   Installed: (none)
>   Candidate: 1.0.15-1+deb7u1
>   Version table:
>  1.0.16-1~bpo70+1 0
> 100 http://ftp.de.debian.org/debian/ wheezy-backports/main amd64
> Packages
>  1.0.15-1+deb7u1 0
> 500 http://ukdebian.mirror.anlx.net/debian/ wheezy/main amd64
> Packages
> 
> sudo apt-get install librdmacm1=1.0.16-1~bpo70+1
> ...
> Setting up librdmacm1 (1.0.16-1~bpo70+1) ...
> 
> sudo apt-get install tgt=1:1.0.51-1~bpo70+1
> ...
> Setting up tgt (1:1.0.51-1~bpo70+1) ...
> [ ok ] Starting target framework daemon: tgtd.
> pcarlton@debian7a:~$ sudo apt-cache policy tgt
> tgt:
>   Installed: 1:1.0.51-1~bpo70+1
>   Candidate: 1:1.0.51-1~bpo70+1
>   Version table:
>  *** 1:1.0.51-1~bpo70+1 0
> 100 http://ftp.de.debian.org/debian/ wheezy-backports/main amd64
> Packages
> 100 /var/lib/dpkg/status
>  1:1.0.17-1 0
> 500 http://ukdebian.mirror.anlx.net/debian/ wheezy/main amd64
> Packages

The tgt packaging has always been really bad. How are things going with
Cinder making lio the default?

> 
> The final issue seems to be the version of ovs-vsctl.  Wheezy comes with
> 1.4.2.
> I first hit some issues with use of --may-exist option in devstack which
> 1.4.2 doesn't support.
> I tweak devstack code to get round that but it fails to start neutron
> agent with
> 
> 2015-11-26 15:01:07.155 92027 DEBUG neutron.agent.linux.utils [-]
> Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10',
> '--oneline', '--format=json', '--', 'set', 'Bridge', 'br-int',
> 'protocols=[OpenFlow10]'] execute_rootwrap_daemon
> /home/pcarlton/openstack/neutron/neutron/agent/linux/utils.py:100
> 2015-11-26 15:01:07.162 92027 ERROR neutron.agent.ovsdb.impl_vsctl [-]
> Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline',
> '--format=json', '--', 'set', 'Bridge', 'br-int',
> 'protocols=[OpenFlow10]']. Exception: Exit code: 1; Stdin: ; Stdout: ;
> Stderr: ovs-vsctl: Bridge does not contain a column whose name matches
> "protocols"
> 
> 2015-11-26 15:01:07.163 92027 ERROR
> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Exit
> code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: Bridge does not contain a
> column whose name matches "protocols"
>  Agent terminated!
> 
> I'll raise a bug for this too.
> 
> I couldn't see an obvious way to update ovs-vsctl on wheezy.
> 
> I'll ask the neutron team what the minimum version of ovs-ctl we are
> meant to support is.
> Given the US holiday we may not get answers till next week and I've
> spent long enough on this for now!

If this is really a min dep, it should be specified as such.

Wheezy is listed supported because it happened to work. There is no one
active on this front, so if no one steps up shortly I think it's
probably just better to remove it from the supported list in devstack,
so people don't expect it to work.

CI would be handy here. Without it this is going to keep happening.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] consolidate IRC change with #openstack-sdk?

On 11/16/2015 11:09 AM, michael mccune wrote:
> On 11/16/2015 09:42 AM, Sean Dague wrote:
>> On 11/16/2015 09:34 AM, michael mccune wrote:
>>> On 11/13/2015 07:58 AM, Sean Dague wrote:
 The #openstack-api IRC channel is quite quiet most days. As such it's
 not something that people are regularly checking in on, or often forget
 about (I know I've been guilty of that). Plus we don't always have the
 right people in a conversation to make a decision.

 I'd like to propose we drop the channel entirely and make
 #openstack-sdk
 the home of API working group conversations. That's where all the
 openstackclient, openstackclientconfig, and sdk conversations are
 happening. It's where the end consumers of any API WG effort are, so
 are
 incredibly good sounding boards for things we are doing. There is
 already a large overlap between the two channels, so just pushing
 forward on that means more critical mass for conversations around the
 whole space of a more usable API for users.

 This came up at the last API WG meeting, but those are pretty low
 quorum
 events, so this is a thing we should definitely decide over ML instead.

 Please express your feelings here and lets make it happen. :)
>>>
>>> i don't necessarily have an outright objection to this, but i would like
>>> explore the future possibilities of this change.
>>>
>>> i think there is possibility for the api-wg to grow and take on more
>>> responsibility within the openstack community and i'm curious about the
>>> net effect down the road if that expansion were to occur. are there any
>>> side-effects we should be aware of as we merge the two
>>> channels/communities into one namespace?
>>>
>>> i realize there is some overlap of engineering/design that takes place
>>> on these channels, i'm just concerned about the idea of merging the two
>>> and then at some point in the future wanting to separate them. i'm not
>>> proposing these thoughts to quash the idea of joining, i'd just like to
>>> more fully understand the longer term implications of a merge.
>>>
>>> anyone with more experience in this community-oriented side of things
>>> have some light to shed?
>>
>> Typically the only real reason to spin off another channel is that there
>> are now too many conversations happening all at once that are unrelated
>> and people are finding it is impeding their ability to get work done. I
>> can't imagine this happening any time within the next couple of years.
>> An average day in #openstack-api is < 20 messages (from a quick spot
>> check in IRC logs).
>>
>> I feel there is a lot of benefits in having these groups that do related
>> work all doing it in one channel. Even though it's a number of discreet
>> outputs, the "hallway track" of having api producers and consumers doing
>> their business all in front of each other should quite help in ensuring
>> that we all make something better.
> 
> thanks for the clarifications Sean, i appreciate the extra resolution
> and i agree that having both these groups able to comment on similar
> issues in a common place seems ideal.

Ok, I'm going to assume with no real disagreement we're agreed here. I'm
moving the api-wg notifications to #openstack-sdks now -
https://review.openstack.org/#/c/251357/1/gerritbot/channels.yaml,cm

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] consolidate IRC change with #openstack-sdk?

2015-11-30 Thread michael mccune


On 11/30/2015 08:45 AM, Sean Dague wrote:

Ok, I'm going to assume with no real disagreement we're agreed here. I'm
moving the api-wg notifications to #openstack-sdks now -
https://review.openstack.org/#/c/251357/1/gerritbot/channels.yaml,cm


ok, i think we've got a few spots in the documentation that refer to 
openstack-api. we can start the process of adjusting all those to 
openstack-sdks.


mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] IRC Bot issues

2015-11-30 Thread Dugger, Donald D

I can’t even connect to the IRC server at all, can others get it?

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786

From: Joshua Hesketh [mailto:joshua.hesk...@gmail.com]
Sent: Monday, November 30, 2015 2:50 AM
To: OpenStack Development Mailing List (not for usage questions) 
; openstack-infra 

Subject: [openstack-dev] IRC Bot issues

Hi all,
Freenode are currently experiencing a severe DDoS attack that are having an 
effect on our bots. As such the meetbot, irc logging and gerrit watcher are 
interminably available.

We expect the bots to resume their normal function once Freenode has recovered. 
For now, meetings may have to be postponed or minuted by hand.
Cheers,
Josh
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]Move encryptors to os-brick

2015-11-30 Thread Coffman, Joel M.



On 11/25/15, 11:33 AM, "Ben Swartzlander" 
mailto:b...@swartzlander.org>> wrote:

On 11/24/2015 03:27 PM, Nathan Reller wrote:
the cinder admin and the nova admin are ALWAYS the same people

There is interest in hybrid clouds where the Nova and Cinder services
are managed by different providers. The customer would place higher
trust in Nova because you must trust the compute service, and the
customer would place less trust in Cinder. One way to achieve this
would be to have all encryption done by Nova. Cinder would simply see
encrypted data and provide a good cheap storage solution for data.

Consider a company with sensitive data. They can run the compute nodes
themselves and offload Cinder service to some third-party service.
This way they are the only ones who can manage the machines that see
the plaintext.

If you have that level of paranoia, I suggest running LUKS inside the
guest VM and not relying on OpenStack to handle your encryption. Then
you don't have to worry about whether nova is sharing your keys with
cinder because even nova won't have them.
That approach isn't actually more secure — anyone with root access to the 
compute host can dump the VM's memory to extract the encryption keys.

Trying to design a system where we expect nova to do data encryption but
not cinder will not work in the long run. The eventual result will be
that nova will have to take on most of the functionality of cinder and
we'll be back to the nova-volume days.
Could you explain further what you mean by "nova will have to take on most of 
the functionality of cinder"? In the current design, Nova is still passing data 
blocks to Cinder for storage – they're just encrypted instead of plaintext. 
That doesn't seem to subvert the functionality of Cinder or reimplement it.

Also in case it's not obvious, if you use different providers for
compute and storage, your performance is going to be absolutely terrible.
The general idea is probably separation of duties, which contradicts the 
original statement that "the cinder admin and the nova admin are ALWAYS the 
same people." Is there an operational reason that these admins must be the same 
person, or is that just typical?

Joel

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][inspector] CMDB integration

2015-11-30 Thread Pavlo Shchelokovskyy

Hi all,

we are looking at how ironic-inspector could integrate with external CMDB
solutions and be able fetch a minimal set of data needed for discovery
(e.g. IPMI credentials and IPs) from CMDB. This could probably be achieved
with data filters framework that is already in place, but we have one
question:

what are people actually using? There are simple (but not conceivably used
in real life) choices to make a first implementation, like fetching a csv
file from HTTP link. Thus we want to learn if there is an already known and
working solution operators are actually using, either open source or at
least with open API.

We really appreciate if you chime in :) This would help us design this
feature the way that will benefit community the most.

Best regards,
-- 
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Hinds, Luke (Nokia - GB/Bristol)

Me too.  It is possible to get on using the web client though;  
https://webchat.freenode.net/ .

On Mon, 2015-11-30 at 14:00 +, EXT Dugger, Donald D wrote:
I can’t even connect to the IRC server at all, can others get it?

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786

From: Joshua Hesketh [mailto:joshua.hesk...@gmail.com]
Sent: Monday, November 30, 2015 2:50 AM
To: OpenStack Development Mailing List (not for usage questions) 
; openstack-infra 

Subject: [openstack-dev] IRC Bot issues

Hi all,
Freenode are currently experiencing a severe DDoS attack that are having an 
effect on our bots. As such the meetbot, irc logging and gerrit watcher are 
interminably available.

We expect the bots to resume their normal function once Freenode has recovered. 
For now, meetings may have to be postponed or minuted by hand.
Cheers,
Josh

___
OpenStack-Infra mailing list
openstack-in...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]Move encryptors to os-brick

2015-11-30 Thread Duncan Thomas

On 30 November 2015 at 16:04, Coffman, Joel M. 
wrote:

> On 11/25/15, 11:33 AM, "Ben Swartzlander"  wrote:
>
> On 11/24/2015 03:27 PM, Nathan Reller wrote:
>
> Trying to design a system where we expect nova to do data encryption but
> not cinder will not work in the long run. The eventual result will be
> that nova will have to take on most of the functionality of cinder and
> we'll be back to the nova-volume days.
>
> Could you explain further what you mean by "nova will have to take on most
> of the functionality of cinder"? In the current design, Nova is still
> passing data blocks to Cinder for storage – they're just encrypted instead
> of plaintext. That doesn't seem to subvert the functionality of Cinder or
> reimplement it.
>

The functionality of cinder is more than blindly storing blocks - in
particular it has create-from/upload-to image, backup, and retype, all of
which do some degree of manipulation of the data and/or volume encryption
metadata.

We are suffering from somewhat incompatible requirements with encryption
between those who want fully functional cinder and encryption on disk (the
common case I think), and those who have enhanced security requirements.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova]Please see request for input on review

2015-11-30 Thread Paul Carlton


Sean

You may have seen I've updated the spec to implement the 
instance/id/migration/if approach


I'd like to discuss the way forward with you and Alex ahead of tomorrows 
api meeting if possible please?



On 24/11/15 16:13, Paul Carlton wrote:

Sean

Please can you look at cancel live migration nova spec,
I'd like to progress this spec ASAP.

Last Thursday, Jay Commented

@sdague @edleafe: Please see the long conversation about the
RESTful-ness of the API on the pause live migration spec:

https://review.openstack.org/#/c/229040/12/specs/mitaka/approved/pause-vm-during-live-migration.rst 



This proposal keeps things in-line with the current os-instance-actions
API -- even though you are correct that it isn't at all REST-full --
with the expectation that the eventual Tasks API will get rid of all
of the instance actions junk.

Are you ok with this?

Thanks




--
Paul Carlton
Software Engineer
Cloud Services
Hewlett Packard
BUK03:T242
Longdown Avenue
Stoke Gifford
Bristol BS34 8QZ

Mobile:+44 (0)7768 994283
Email:mailto:paul.carlt...@hpe.com
Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12 1HN 
Registered No: 690597 England.
The contents of this message and any attachments to it are confidential and may be 
legally privileged. If you have received this message in error, you should delete it from 
your system immediately and advise the sender. To any recipient of this message within 
HP, unless otherwise stated you should consider this message and attachments as "HP 
CONFIDENTIAL".




smime.p7s
Description: S/MIME Cryptographic Signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread lichen.hangzhou

Can't connect +1 and web client do not work  :(

-chen


At 2015-11-30 22:08:12, "Hinds, Luke (Nokia - GB/Bristol)" 
 wrote:

Me too.  It is possible to get on using the web client though;  
https://webchat.freenode.net/ .


On Mon, 2015-11-30 at 14:00 +, EXT Dugger, Donald D wrote:

I can’t even connect to the IRC server at all, can others get it?

 

--

Don Dugger

"Censeo Toto nos in Kansa esse decisse." - D. Gale

Ph: 303/443-3786

 

From: Joshua Hesketh [mailto:joshua.hesk...@gmail.com]
Sent: Monday, November 30, 2015 2:50 AM
To: OpenStack Development Mailing List (not for usage questions) 
; openstack-infra 

Subject: [openstack-dev] IRC Bot issues

 

Hi all,

Freenode are currently experiencing a severe DDoS attack that are having an 
effect on our bots. As such the meetbot, irc logging and gerrit watcher are 
interminably available.

We expect the bots to resume their normal function once Freenode has recovered. 
For now, meetings may have to be postponed or minuted by hand.

Cheers,

Josh

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-scheduler] Scheduler sub-group meeting - Agenda 11/30

2015-11-30 Thread Dugger, Donald D

Sorry guys, I can't get onto IRC so, if any of you can connect you'll just have 
to muddle through without me :-)  Hopefully freenode will be back up next week.

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786


-Original Message-
From: Dugger, Donald D 
Sent: Monday, November 30, 2015 12:00 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: [nova-scheduler] Scheduler sub-group meeting - Agenda 11/30

Meeting on #openstack-meeting-alt at 1400 UTC (7:00AM MDT)

1) Spec & BPs - 
https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking
2) Bugs - https://bugs.launchpad.net/nova/+bugs?field.tag=scheduler
3) Opens

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-scheduler] Scheduler sub-group meeting - Agenda 11/30

2015-11-30 Thread Sylvain Bauza




Le 30/11/2015 15:17, Dugger, Donald D a écrit :

Sorry guys, I can't get onto IRC so, if any of you can connect you'll just have 
to muddle through without me :-)  Hopefully freenode will be back up next week.



We had a discussion on the channel, and the agreement was to cancel the 
meeting for this week.


Infra folks, all my thoughts are with you. DDoSers, I hate you.

-Sylvain


--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786


-Original Message-
From: Dugger, Donald D
Sent: Monday, November 30, 2015 12:00 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: [nova-scheduler] Scheduler sub-group meeting - Agenda 11/30

Meeting on #openstack-meeting-alt at 1400 UTC (7:00AM MDT)

1) Spec & BPs - 
https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking
2) Bugs - https://bugs.launchpad.net/nova/+bugs?field.tag=scheduler
3) Opens

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Glance] PSA: Today's Artifacts meeting canceled.

2015-11-30 Thread Nikhil Komawar

Hi all,

There was some bot issue this morning and all the meeting commands
weren't working hence.

We canceled the Artifacts meeting after ~20 mins however, there was a
lot of informal chatting on the #openstack-meeting-4 channel during the
wait (no significant decisions taken, and there are plans to further
improve the meeting and other collaborations).

-- 

Thanks,
Nikhil


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread yang, xing

Hi Wang Hao,

Here¹s a Cinder spec in review on replicating a group of volumes:

https://review.openstack.org/#/c/229722/

It is a not an easy problem to solve.  Not that we should rush on this
problem, but we should start thinking about how to solve this as some
backends can only replicate a CG or a pool of volumes.

Thanks,
Xing



On 11/30/15, 4:51 AM, "hao wang"  wrote:

>Hi, Duncan
>
>2015-11-30 15:54 GMT+08:00 Duncan Thomas :
>> Hi WangHao
>>
>> This was quite thoroughly discussed during the early discussions on
>> replication. The general statement was 'not yet'. Getting any kind of
>> workable replication API has proven to be very, very difficult to get
>>right
>> - we won't know for another full cycle whether we've actually gotten it
>> somewhere near right, as operators start to deploy it. Piling more
>>feature
>> in the replication API before a) it has been used by operators and b)
>> storage vendors have implemented what we already has would IMO be a
>>mistake.
>
>I agree with you, in my mind, using replication what we have is first
>thing we should done,
>improve it much better is second thing, and then we will add another
>new features
>one by one stably.
>
>> None of this means that more DR interfaces don't belong in cinder, just
>>that
>> getting them right, getting them universal and getting them useful is
>>quite
>> a hard problem, and not one we should be in a rush to solve.
>>Particularly as
>> DR and replication is still a niche area of cinder, and we still have
>>major
>> issues in our basic functionality.
>
>Yes, this will convince me about DR in Cinder, very clearly, thanks.
>>
>> On 30 November 2015 at 03:45, hao wang  wrote:
>>>
>>> Sean and Michal,
>>>
>>> In fact, there is a reason that I ask this question. Recently I have a
>>> confusion about if cinder should provide the ability of Disaster
>>> Recovery to storage resources, like volume. I mean we have volume
>>> replication v1&v2, but for DR, specially DR between two independent
>>> OpenStack sites(production and DR site), I feel we still need more
>>> features to support it, for example consistency group for replication,
>>> etc. I'm not sure if those features belong in Cinder or some new
>>> project for DR.
>>>
>>> BR
>>> WangHao
>>>
>>> 2015-11-30 3:02 GMT+08:00 Sean McGinnis :
>>> > On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>>> >> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>>> >> > Hi guys,
>>> >> >
>>> >> > I notice nova have a clarification of project scope:
>>> >> > http://docs.openstack.org/developer/nova/project_scope.html
>>> >> >
>>> >> > I want to find cinder's, but failed,  do you know where to find
>>>it?
>>> >> >
>>> >> > It's important to let developers know what feature should be
>>> >> > introduced into cinder and what shouldn't.
>>> >> >
>>> >> > BR
>>> >> > Wang Hao
>>> >>
>>> >> I believe Nova team needed to formalize the scop to have an
>>>explanation
>>> >> for all the "this doesn't belong in Nova" comments on feature
>>>requests.
>>> >> Does Cinder suffer from similar problems? From my perspective it's
>>>not
>>> >> critically needed.
>>> >
>>> > I agree. I haven't seen a need for something like that with Cinder.
>>>Wang
>>> > Hao, is there a reason you feel you need that?
>>> >
>>> >
>>> >
>>> > 
>>>
>>>__
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe:
>>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> 
>>>
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> --
>> --
>> Duncan Thomas
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Derek Higgins


Hi All,

A few months tripleo switch from its devtest based CI to one that 
was based on instack. Before doing this we anticipated disruption in the 
ci jobs and removed them from non tripleo projects.


We'd like to investigate adding it back to heat and ironic as these 
are the two projects where we find our ci provides the most value. But 
we can only do this if the results from the job are treated as voting.


In the past most of the non tripleo projects tended to ignore the 
results from the tripleo job as it wasn't unusual for the job to broken 
for days at a time. The thing is, ignoring the results of the job is the 
reason (the majority of the time) it was broken in the first place.
To decrease the number of breakages we are now no longer running 
master code for everything (for the non tripleo projects we bump the 
versions we use periodically if they are working). I believe with this 
model the CI jobs we run have become a lot more reliable, there are 
still breakages but far less frequently.


What I proposing is we add at least one of our tripleo jobs back to both 
heat and ironic (and other projects associated with them e.g. clients, 
ironicinspector etc..), tripleo will switch to running latest master of 
those repositories and the cores approving on those projects should wait 
for a passing CI jobs before hitting approve. So how do people feel 
about doing this? can we give it a go? A couple of people have already 
expressed an interest in doing this but I'd like to make sure were all 
in agreement before switching it on.


thanks,
Derek.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Automatically generate sample configuration files (oslo config generator)

2015-11-30 Thread Martin Hickey


Hi,

The generation of Neutron configuration files using the oslo config
generator is being introduced in Mitaka as part of blueprint [1] and bug
[2]. This is broken down into two phases: Neutron core and Neutron *aaS
projects.

The first part of the first phase is the generation of sample core
configuration files in patch [3]. The next part is the removal of the
sample configuration files from the core repository in patch [4]. Patch [3]
and [5] make the static sample configuration files in the Neutron
repository redundant.  Developers should no longer update these
configuration files as  they will be removed in patch[4]. Refer to the
Neutron contribution guide for more details.

The final phase of the blueprint is the update of the Neutron *aaS projects
for configuration generation. These commits will be associated to bug [2].

It is advised that Neutron sub-projects do not keep their configuration
files in their respective trees and instead generate them using a similar
approach as Neutron does. Operators should refer to the release notes for
more details.

[1]
https://blueprints.launchpad.net/neutron/+spec/autogen-neutron-conf-file
[2] https://bugs.launchpad.net/neutron/+bug/1199963
[3] https://review.openstack.org/#/c/204206/
[4] https://review.openstack.org/#/c/251348/
[5] https://review.openstack.org/#/c/204722/

Regards,
Martin


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Performance] Filling performance team working items list and taking part in their resolving

2015-11-30 Thread Matt Riedemann




On 11/27/2015 3:54 AM, Dina Belova wrote:

Hey OpenStack devs and operators!

Folks, I would like to share list of working items Performance Team is
currently having in the backlog -
https://etherpad.openstack.org/p/perf-zoom-zoom [Work Items to grab]
section. I'm really encouraging you to fill it with concrete pieces of
work you think will be useful and take part in the
development/investigation by assigning some of them to yourself and
working on them :)

Cheers,
Dina


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



One of the work items was to enable the nova metadata service in a large 
ops job. There are two voting large ops jobs in nova, one runs with 
nova-network and the other runs with neutron. Besides those differences, 
I'm not sure if there is any other difference in the jobs. So I guess 
we'd just need to pick which one runs the nova-api-metadata service 
rather than using config drive. The only other job I know of that runs 
that is the postgres job and that runs nova-network, so I'd say we turn 
on n-api-meta in the neutron large ops job.


Are there any issues with that?

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Clark, Jay

Can't connect either. Dead in the water

Regards,
Jay Clark
Sr. OpenStack Deployment Engineer
E: jason.t.cl...@hpe.com
H: 919.341.4670
M: 919.345.1127
IRC (freenode): jasondotstar


From: lichen.hangzhou [lichen.hangz...@gmail.com]
Sent: Monday, November 30, 2015 9:17 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: openstack-in...@lists.openstack.org
Subject: Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

Can't connect +1 and web client do not work  :(

-chen

At 2015-11-30 22:08:12, "Hinds, Luke (Nokia - GB/Bristol)" 
 wrote:
Me too.  It is possible to get on using the web client though;  
https://webchat.freenode.net/ .

On Mon, 2015-11-30 at 14:00 +, EXT Dugger, Donald D wrote:
I can’t even connect to the IRC server at all, can others get it?

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786

From: Joshua Hesketh 
[mailto:joshua.hesk...@gmail.com]
Sent: Monday, November 30, 2015 2:50 AM
To: OpenStack Development Mailing List (not for usage questions) 
mailto:openstack-dev@lists.openstack.org>>; 
openstack-infra 
mailto:openstack-in...@lists.openstack.org>>
Subject: [openstack-dev] IRC Bot issues

Hi all,
Freenode are currently experiencing a severe DDoS attack that are having an 
effect on our bots. As such the meetbot, irc logging and gerrit watcher are 
interminably available.

We expect the bots to resume their normal function once Freenode has recovered. 
For now, meetings may have to be postponed or minuted by hand.
Cheers,
Josh

___
OpenStack-Infra mailing list
openstack-in...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder]Do we have project scope for cinder?

On 11/29/2015 02:02 PM, Sean McGinnis wrote:
> On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>>> Hi guys,
>>>
>>> I notice nova have a clarification of project scope:
>>> http://docs.openstack.org/developer/nova/project_scope.html
>>>
>>> I want to find cinder's, but failed,  do you know where to find it?
>>>
>>> It's important to let developers know what feature should be
>>> introduced into cinder and what shouldn't.
>>>
>>> BR
>>> Wang Hao
>>
>> I believe Nova team needed to formalize the scop to have an explanation
>> for all the "this doesn't belong in Nova" comments on feature requests.
>> Does Cinder suffer from similar problems? From my perspective it's not
>> critically needed.
> 
> I agree. I haven't seen a need for something like that with Cinder. Wang
> Hao, is there a reason you feel you need that?
> 

For reference here is the Cinder mission statement:
http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml#n273

All projects listed in the governance repository reference/projects.yaml
have a mission statement, I do encourage folks thinking about starting a
project to look at the mission statements here first as there may
already be an effort ongoing with which you can align your work.

Thanks Wang Hao,
Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon] Supporting Django 1.9

2015-11-30 Thread Thomas Goirand

On 11/29/2015 08:59 PM, Rob Cresswell (rcresswe) wrote:
> https://blueprints.launchpad.net/horizon/+spec/drop-dj17
> 
> 
> This is where changes are currently being tracked. I don’t quite
> understand why these would be back ported; they would break Liberty with
> 1.7. Perhaps we can discuss on IRC tomorrow?
> 
> Rob

Before Mitaka is out, Sid will hold Horizon from Liberty. I don't care
having breakage for Django 1.7 (Sid has 1.8 already), but what I need is
having support for Django 1.9. That's what I want to backport.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

2015-11-30 Thread sean roberts

How about:
First timers assign a bug to a mentor and the mentor takes responsibility
for the first timer learning from the bug to completion.

Per project, a few people volunteer themselves as mentors. As easy as
responding to [project][mentor] emails.

On Monday, November 30, 2015, Sean Dague  wrote:

> On 11/25/2015 03:22 PM, Shamail wrote:
> > Hi,
> >
> >> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  > wrote:
> >>
> >> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
> >>> Hi everyone,
> >>>
> >>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs which
> >>> discusses how one open-source project is encouraging contributions by
> new
> >>> open-source contributors through a combination of a special tag (which
> is
> >>> associated with work that is needed but can only be completed by
> someone
> >>> who is a first-time contributor) and helpful comments in the review
> phase
> >>> to ensure the contribution(s) eventually get merged.
> >>>
> >>> While reading the article, I immediately thought about our
> >>> low-hanging-fruit bug tag which is used for a very similar purpose in
> "bug
> >>> fixing" section of  the "how to contribute" page[2].  The
> low-hanging-fruit
> >>> tag is used to identify items that are generally suitable for
> first-time or
> >>> beginner contributors but, in reality, anyone can pick them up.
> >>>
> >>> I wanted to propose a new tag (or even changing the, existing,
> low-hanging
> >>> fruit tag) that would identify items that we are reserving for
> first-time
> >>> OpenStack contributors (e.g. a patch-set for the item submitted by
> someone
> >>> who is not a first time contributor would be rejected)... The same
> article
> >>> that Andrew shared mentions using an "up-for-grabs" tag which also
> >>> populates the items at up-for-grabs[3] (a site where people looking to
> >>> start working on open-source projects see entry-level items from
> multiple
> >>> projects).  If we move forward with an exclusive tag for first-timers
> then
> >>> it would be nice if we could use the up-for-grabs tag so that OpenStack
> >>> also shows up on the list too.  Please let me know if this change
> should be
> >>> proposed elsewhere, the tags are maintained in launchpad and the wiki I
> >>> found related to bug tags[4] didn't indicate a procedure for
> submitting a
> >>> change proposal.
> >>
> >> I like the idea of making bugs suitable for first-timers more
> >> discoverable. I'm not sure we need to *reserve* any bugs for any class
> >> of contributor. What benefit do you think that provides?
> > I would have to defer to additional feedback here...
> >
> > My own perspective from when I was doing my first contribution is that
> it was hard to find active "low-hanging-fruit" items.  Most were already
> work-in-progress or assigned.
>
> This was a direct consequence of us dropping the auto-abandoning of old
> code reviews in gerrit. When a review is abandoned the bug is flipped
> back to New instead of In Progress.
>
> I found quite often people go and gobble up bugs assigning them to
> themselves, but don't make real progress on them. Then new contributors
> show up, and don't work on any of those issues because our tools say
> someone is already on top of it.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
~sean
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release] using reno for libraries

2015-11-30 Thread Doug Hellmann

Excerpts from Dmitry Tantsur's message of 2015-11-30 10:06:25 +0100:
> On 11/28/2015 02:48 PM, Doug Hellmann wrote:
> > Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:
> >> Liaisons,
> >>
> >> We're making good progress on adding reno to service projects as
> >> we head to the Mitaka-1 milestone. Thank you!
> >>
> >> We also need to add reno to all of the other deliverables with
> >> changes that might affect deployers. That means clients and other
> >> libraries, SDKs, etc. with configuration options or where releases
> >> can change deployment behavior in some way. Now that most teams
> >> have been through this conversion once, it should be easy to replicate
> >> for the other repositories in a similar way.
> >>
> >> Libraries have 2 audiences for release notes: developers consuming
> >> the library and deployers pushing out new versions of the libraries.
> >> To separate the notes for the two audiences, and avoid doing manually
> >> something that we have been doing automatically, we can use reno
> >> just for deployer release notes (changes in support for options,
> >> drivers, etc.). That means the library repositories that need reno
> >> should have it configured just like for the service projects, with
> >> the separate jobs and a publishing location different from their
> >> existing developer documentation. The developer docs can continue
> >> to include notes for the developer audience.
> >
> > I've had a couple of questions about this split for release notes. The
> > intent is for developer-focused notes to continue to come from commit
> > messages and in-tree documentation, while using reno for new and
> > additional deployer-focused communication. Most commits to libraries
> > won't need reno release notes.
> 
> This looks like unnecessary overcomplication. Why not use such a 
> convenient tool for both kinds of release notes instead of having us 
> invent and maintain one more place to put release notes, now for 

In the past we have had rudimentary release notes and changelogs
for developers to read based on the git commit messages. Since
deployers and developers care about different things, we don't want
to make either group sift through the notes meant for the other.
So, we publish notes in different ways.

The thing that is new here is publishing release notes for changes
in libraries that deployers need to know about. While the Oslo code
was in the incubator, and being copied into applications, it was
possible to detect deployer-focused changes like new or deprecated
configuration options in the application and put the notes there.
Using shared libraries means those changes can happen without
application developers being aware of them, so the library maintainers
need to be publishing notes. Using reno for those notes is consistent
with the way they are handled in the applications, so we're extending
one tool to more repositories.

> developers? It's already not so easy to explain reno to newcomers, this 
> idea makes it even harder...

Can you tell me more about the difficulty you've had? I would like to
improve the documentation for reno and for how we use it.

Doug

> 
> >
> > Doug
> >
> >>
> >> After we start using reno for libraries, the release announcement
> >> email tool will be updated to use those same notes to build the
> >> message in addition to looking at the git change log. This will be
> >> a big step toward unifying the release process for services and
> >> libraries, and will allow us to make progress on completing the
> >> automation work we have planned for this cycle.
> >>
> >> It's not necessary to add reno to the liberty branch for library
> >> projects, since we tend to backport far fewer changes to libraries.
> >> If you maintain a library that does see a lot of backports, by all
> >> means go ahead and add reno, but it's not a requirement. If you do
> >> set up multiple branches, make sure you have one page that uses the
> >> release-notes directive without specifing a branch, as in the
> >> oslo.config example, to build notes for the "current" branch to get
> >> releases from master and to serve as a test for rendering notes
> >> added to stable branches.
> >>
> >> Thanks,
> >> Doug
> >>
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

2015-11-30 Thread Doug Hellmann

Excerpts from sean roberts's message of 2015-11-30 07:57:54 -0800:
> How about:
> First timers assign a bug to a mentor and the mentor takes responsibility
> for the first timer learning from the bug to completion.

That would mean the learning process is different from what we want the
regular process to be.

If the problem is identifying "In Progress" bugs that are actually not
being worked on, then let's figure out a way to make that easier.
sdague's point about the auto-abandon process may help. We could query
gerrit for "stale" reviews that would have met the old abandon
requirements and that refer to bugs, for example. Using that
information, someone could follow-up with the patch owner to see if it
is actually abandoned, before changing the bug status or encouraging the
owner to abandon the patch.

> 
> Per project, a few people volunteer themselves as mentors. As easy as
> responding to [project][mentor] emails.
> 
> On Monday, November 30, 2015, Sean Dague  wrote:
> 
> > On 11/25/2015 03:22 PM, Shamail wrote:
> > > Hi,
> > >
> > >> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  > > wrote:
> > >>
> > >> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
> > >>> Hi everyone,
> > >>>
> > >>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs which
> > >>> discusses how one open-source project is encouraging contributions by
> > new
> > >>> open-source contributors through a combination of a special tag (which
> > is
> > >>> associated with work that is needed but can only be completed by
> > someone
> > >>> who is a first-time contributor) and helpful comments in the review
> > phase
> > >>> to ensure the contribution(s) eventually get merged.
> > >>>
> > >>> While reading the article, I immediately thought about our
> > >>> low-hanging-fruit bug tag which is used for a very similar purpose in
> > "bug
> > >>> fixing" section of  the "how to contribute" page[2].  The
> > low-hanging-fruit
> > >>> tag is used to identify items that are generally suitable for
> > first-time or
> > >>> beginner contributors but, in reality, anyone can pick them up.
> > >>>
> > >>> I wanted to propose a new tag (or even changing the, existing,
> > low-hanging
> > >>> fruit tag) that would identify items that we are reserving for
> > first-time
> > >>> OpenStack contributors (e.g. a patch-set for the item submitted by
> > someone
> > >>> who is not a first time contributor would be rejected)... The same
> > article
> > >>> that Andrew shared mentions using an "up-for-grabs" tag which also
> > >>> populates the items at up-for-grabs[3] (a site where people looking to
> > >>> start working on open-source projects see entry-level items from
> > multiple
> > >>> projects).  If we move forward with an exclusive tag for first-timers
> > then
> > >>> it would be nice if we could use the up-for-grabs tag so that OpenStack
> > >>> also shows up on the list too.  Please let me know if this change
> > should be
> > >>> proposed elsewhere, the tags are maintained in launchpad and the wiki I
> > >>> found related to bug tags[4] didn't indicate a procedure for
> > submitting a
> > >>> change proposal.
> > >>
> > >> I like the idea of making bugs suitable for first-timers more
> > >> discoverable. I'm not sure we need to *reserve* any bugs for any class
> > >> of contributor. What benefit do you think that provides?
> > > I would have to defer to additional feedback here...
> > >
> > > My own perspective from when I was doing my first contribution is that
> > it was hard to find active "low-hanging-fruit" items.  Most were already
> > work-in-progress or assigned.
> >
> > This was a direct consequence of us dropping the auto-abandoning of old
> > code reviews in gerrit. When a review is abandoned the bug is flipped
> > back to New instead of In Progress.
> >
> > I found quite often people go and gobble up bugs assigning them to
> > themselves, but don't make real progress on them. Then new contributors
> > show up, and don't work on any of those issues because our tools say
> > someone is already on top of it.
> >
> > -Sean
> >
> > --
> > Sean Dague
> > http://dague.net
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Jeremy Stanley

On 2015-11-30 20:49:46 +1100 (+1100), Joshua Hesketh wrote:
> Freenode are currently experiencing a severe DDoS attack that are
> having an effect on our bots. As such the meetbot, irc logging and
> gerrit watcher are interminably available.
> 
> We expect the bots to resume their normal function once Freenode
> has recovered. For now, meetings may have to be postponed or
> minuted by hand.

I'm not sure if the attacks on Freenode have subsided yet, but I was
finally able to get the "openstack" meetbot reconnected around 15:45
UTC and it's seemed stable for half hour since. That's not to say I
have much faith at the moment it'll stick around, but it's at least
an improvement.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Gururaj Grandhi

Hi,



 This is to announce that  we have  setup  a  Third Party CI environment
for Proliant iLO Drivers. The results will be posted  under "HP Proliant CI
check" section in Non-voting mode.   We will be  running the basic deploy
tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We will
first  pursue to make the results consistent and over a period of time we
will try to promote it to voting mode.



   For more information check the Wiki:
https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci ,
for any issues please contact ilo_driv...@groups.ext.hpe.com





Thanks & Regards,

Gururaja Grandhi

R&D Project Manager

HPE Proliant  Ironic  Project
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:
> Hi,
> 
> 
> 
>  This is to announce that  we have  setup  a  Third Party CI environment
> for Proliant iLO Drivers. The results will be posted  under "HP Proliant CI
> check" section in Non-voting mode.   We will be  running the basic deploy
> tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We will
> first  pursue to make the results consistent and over a period of time we
> will try to promote it to voting mode.
> 
> 
> 
>For more information check the Wiki:
> https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci ,
> for any issues please contact ilo_driv...@groups.ext.hpe.com
> 
> 
> 
> 
> 
> Thanks & Regards,
> 
> Gururaja Grandhi
> 
> R&D Project Manager
> 
> HPE Proliant  Ironic  Project
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

Please do not post announcements to the mailing list about the existence
of your third party ci system.

Ensure your third party ci system is listed here:
https://wiki.openstack.org/wiki/ThirdPartySystems (there are
instructions on the bottom of the page) as well as fill out a template
on your system so that folks can find your third party ci system the
same as all other third party ci systems.

Ensure you are familiar with the requirements for third party systems
listed here:
http://docs.openstack.org/infra/system-config/third_party.html#requirements

Thank you,
Anita.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] notification subteam meeting

2015-11-30 Thread Balázs Gibizer

Hi, 

The next meeting of the nova notification subteam will happen 2015-12-01 
Tuesday 20:00 UTC [1] on #openstack-meeting-alt on freenode 

Agenda:
- Status of the outstanding specs and code reviews
- Mid-cycle
- AOB

See you there.

Cheers,
Gibi

 [1] https://www.timeanddate.com/worldclock/fixedtime.html?iso=20151201T20 
 [2] https://wiki.openstack.org/wiki/Meetings/NovaNotification


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread Walter A. Boring IV

As a side note to the DR discussion here, there was a session in Tokyo 
that talked about a new

DR project called Smaug.   You can see their mission statement here:
https://launchpad.net/smaug

https://github.com/openstack/smaug

There is another service in the making called DRagon:
https://www.youtube.com/watch?v=upCzuFnswtw
http://www.slideshare.net/AlonMarx/dragon-and-cinder-v-brownbag-54639869

Yes that's 2 DR like service starting in OpenStack that are related to 
dragons.


Walt



Sean and Michal,

In fact, there is a reason that I ask this question. Recently I have a
confusion about if cinder should provide the ability of Disaster
Recovery to storage resources, like volume. I mean we have volume
replication v1&v2, but for DR, specially DR between two independent
OpenStack sites(production and DR site), I feel we still need more
features to support it, for example consistency group for replication,
etc. I'm not sure if those features belong in Cinder or some new
project for DR.

BR
WangHao

2015-11-30 3:02 GMT+08:00 Sean McGinnis :

On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:

On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:

Hi guys,

I notice nova have a clarification of project scope:
http://docs.openstack.org/developer/nova/project_scope.html

I want to find cinder's, but failed,  do you know where to find it?

It's important to let developers know what feature should be
introduced into cinder and what shouldn't.

BR
Wang Hao

I believe Nova team needed to formalize the scop to have an explanation
for all the "this doesn't belong in Nova" comments on feature requests.
Does Cinder suffer from similar problems? From my perspective it's not
critically needed.

I agree. I haven't seen a need for something like that with Cinder. Wang
Hao, is there a reason you feel you need that?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic] Do we need to have a mid-cycle?

On 11/23/2015 01:00 PM, Jim Rollenhagen wrote:
> On Mon, Nov 16, 2015 at 06:05:54AM -0800, Jim Rollenhagen wrote:
>>
>> Another idea I floated last week was to do a virtual midcycle of sorts.
>> Treat it like a normal midcycle in that everyone tells their management
>> "I'm out for 3-4 days for the midcycle", but they don't travel anywhere.
>> We come up with an agenda, see if there's any planning/syncing work to
>> do, or if it's all just hacking on code/reviews.
>>
>> Then we can set up some hangouts (or similar) to get people in the same
>> "room" working on things. Time zones will get weird, but we tend to
>> split into smaller groups at the midcycle anyway; this is just more
>> timezone-aligned. We can also find windows where time zones overlap when
>> we want to go across those boundaries. Disclaimer: people may need to
>> work some weird hours to do this well.
>>
>> I think this might get a little bit bumpy, but if it goes relatively
>> well we can try to improve on it for the future. Worst case, it's a
>> total failure and is roughly equivalent to the "no midcycle" option.
> 
> Nobody has objected, so we're going to roll with this. See y'all there. :)
> 
> // jim
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

Listing your virtual sprint on the virtual sprints wikipage is helpful
to those folks who might not work on ironic daily to consider helping:
https://wiki.openstack.org/wiki/VirtualSprints

Thanks,
Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release] using reno for libraries

On 11/30/2015 05:14 PM, Doug Hellmann wrote:

Excerpts from Dmitry Tantsur's message of 2015-11-30 10:06:25 +0100:

On 11/28/2015 02:48 PM, Doug Hellmann wrote:

Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:

Liaisons,

We're making good progress on adding reno to service projects as
we head to the Mitaka-1 milestone. Thank you!

We also need to add reno to all of the other deliverables with
changes that might affect deployers. That means clients and other
libraries, SDKs, etc. with configuration options or where releases
can change deployment behavior in some way. Now that most teams
have been through this conversion once, it should be easy to replicate
for the other repositories in a similar way.

Libraries have 2 audiences for release notes: developers consuming
the library and deployers pushing out new versions of the libraries.
To separate the notes for the two audiences, and avoid doing manually
something that we have been doing automatically, we can use reno
just for deployer release notes (changes in support for options,
drivers, etc.). That means the library repositories that need reno
should have it configured just like for the service projects, with
the separate jobs and a publishing location different from their
existing developer documentation. The developer docs can continue
to include notes for the developer audience.

I've had a couple of questions about this split for release notes. The
intent is for developer-focused notes to continue to come from commit
messages and in-tree documentation, while using reno for new and
additional deployer-focused communication. Most commits to libraries
won't need reno release notes.

This looks like unnecessary overcomplication. Why not use such a
convenient tool for both kinds of release notes instead of having us
invent and maintain one more place to put release notes, now for

In the past we have had rudimentary release notes and changelogs
for developers to read based on the git commit messages. Since
deployers and developers care about different things, we don't want
to make either group sift through the notes meant for the other.
So, we publish notes in different ways.

Hmm, so maybe for small libraries with few changes it's still fine to
publish them together, what do you think?

The thing that is new here is publishing release notes for changes
in libraries that deployers need to know about. While the Oslo code
was in the incubator, and being copied into applications, it was
possible to detect deployer-focused changes like new or deprecated
configuration options in the application and put the notes there.
Using shared libraries means those changes can happen without
application developers being aware of them, so the library maintainers
need to be publishing notes. Using reno for those notes is consistent
with the way they are handled in the applications, so we're extending
one tool to more repositories.

developers? It's already not so easy to explain reno to newcomers, this
idea makes it even harder...

Can you tell me more about the difficulty you've had? I would like to
improve the documentation for reno and for how we use it.

Usually people are stuck at the "how do I do this at all" stage :) we've
even added it to the ironic developer FAQ. As to me, the official reno
documentation is nice enough (but see below), maybe people are not aware
of it.

Another "issue" (at least for our newcomers) with reno docs is that
http://docs.openstack.org/developer/reno/usage.html#generating-a-report
mentions the "reno report" command which is not something we all
actually use, we use these "tox -ereleasenotes" command. What is worse,
this command (I guess it's by design) does not catch release note files
that are just created locally. It took me time to figure out that I have
to commit release notes before "tox -ereleasenotes" would show them in
the rendered HTML.

Finally, people are confused by how our release note jobs handle
branches. E.g. ironic-inspector release notes [1] currently seem to show
release notes from stable/liberty (judging by the version), so no
current items [2] are shown.

[1] http://docs.openstack.org/releasenotes/ironic-inspector/unreleased.html
[2] for example
http://docs-draft.openstack.org/18/250418/2/gate/gate-ironic-inspector-releasenotes/f0b9363//releasenotes/build/html/unreleased.html

Doug

After we start using reno for libraries, the release announcement
email tool will be updated to use those same notes to build the
message in addition to looking at the git change log. This will be
a big step toward unifying the release process for services and
libraries, and will allow us to make progress on completing the
automation work we have planned for this cycle.

It's not necessary to add reno to the liberty branch for library
projects, since we tend to backport far fewer changes to libraries.
If you maintain a library that does see a lot of backports, by all
means go

Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job


On 11/30/2015 04:19 PM, Derek Higgins wrote:

Hi All,

 A few months tripleo switch from its devtest based CI to one that
was based on instack. Before doing this we anticipated disruption in the
ci jobs and removed them from non tripleo projects.

 We'd like to investigate adding it back to heat and ironic as these
are the two projects where we find our ci provides the most value. But
we can only do this if the results from the job are treated as voting.

 In the past most of the non tripleo projects tended to ignore the
results from the tripleo job as it wasn't unusual for the job to broken
for days at a time. The thing is, ignoring the results of the job is the
reason (the majority of the time) it was broken in the first place.
 To decrease the number of breakages we are now no longer running
master code for everything (for the non tripleo projects we bump the
versions we use periodically if they are working). I believe with this
model the CI jobs we run have become a lot more reliable, there are
still breakages but far less frequently.

What I proposing is we add at least one of our tripleo jobs back to both
heat and ironic (and other projects associated with them e.g. clients,
ironicinspector etc..), tripleo will switch to running latest master of
those repositories and the cores approving on those projects should wait
for a passing CI jobs before hitting approve. So how do people feel
about doing this? can we give it a go? A couple of people have already
expressed an interest in doing this but I'd like to make sure were all
in agreement before switching it on.


I'm one of these "people", so definitely +1 here.

By the way, is it possible to NOT run tripleo-ci on changes touching 
only tests and docs? We do the same for our devstack jobs, it saves some 
infra resources.




thanks,
Derek.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral] Team meeting minutes - 11/30/2015

2015-11-30 Thread Anastasia Kuznetsova

Thanks everyone for joining us!

Meeting minutes:
http://eavesdrop.openstack.org/meetings/mistral/2015/mistral.2015-11-30-16.01.html
Meeting log:
http://eavesdrop.openstack.org/meetings/mistral/2015/mistral.2015-11-30-16.01.log.html

-- 
Best regards,
Anastasia Kuznetsova
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Derek Higgins




On 30/11/15 17:03, Dmitry Tantsur wrote:

On 11/30/2015 04:19 PM, Derek Higgins wrote:

Hi All,

 A few months tripleo switch from its devtest based CI to one that
was based on instack. Before doing this we anticipated disruption in the
ci jobs and removed them from non tripleo projects.

 We'd like to investigate adding it back to heat and ironic as these
are the two projects where we find our ci provides the most value. But
we can only do this if the results from the job are treated as voting.

 In the past most of the non tripleo projects tended to ignore the
results from the tripleo job as it wasn't unusual for the job to broken
for days at a time. The thing is, ignoring the results of the job is the
reason (the majority of the time) it was broken in the first place.
 To decrease the number of breakages we are now no longer running
master code for everything (for the non tripleo projects we bump the
versions we use periodically if they are working). I believe with this
model the CI jobs we run have become a lot more reliable, there are
still breakages but far less frequently.

What I proposing is we add at least one of our tripleo jobs back to both
heat and ironic (and other projects associated with them e.g. clients,
ironicinspector etc..), tripleo will switch to running latest master of
those repositories and the cores approving on those projects should wait
for a passing CI jobs before hitting approve. So how do people feel
about doing this? can we give it a go? A couple of people have already
expressed an interest in doing this but I'd like to make sure were all
in agreement before switching it on.


I'm one of these "people", so definitely +1 here.

By the way, is it possible to NOT run tripleo-ci on changes touching
only tests and docs? We do the same for our devstack jobs, it saves some
infra resources.
We don't do it currently, but I'm sure we could and it sounds like a 
good idea to me.






thanks,
Derek.

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers


On 11/30/2015 05:34 PM, Anita Kuno wrote:

On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:

Hi,



  This is to announce that  we have  setup  a  Third Party CI environment
for Proliant iLO Drivers. The results will be posted  under "HP Proliant CI
check" section in Non-voting mode.   We will be  running the basic deploy
tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We will
first  pursue to make the results consistent and over a period of time we
will try to promote it to voting mode.



For more information check the Wiki:
https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci ,
for any issues please contact ilo_driv...@groups.ext.hpe.com





Thanks & Regards,

Gururaja Grandhi

R&D Project Manager

HPE Proliant  Ironic  Project



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Please do not post announcements to the mailing list about the existence
of your third party ci system.


Could you please explain why? As a developer I appreciated this post.



Ensure your third party ci system is listed here:
https://wiki.openstack.org/wiki/ThirdPartySystems (there are
instructions on the bottom of the page) as well as fill out a template
on your system so that folks can find your third party ci system the
same as all other third party ci systems.


Wiki is not an announcement FWIW.



Ensure you are familiar with the requirements for third party systems
listed here:
http://docs.openstack.org/infra/system-config/third_party.html#requirements

Thank you,
Anita.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Lucas Alvares Gomes

I'm +1 for re-enabling it in Ironic. I think ironic plays a big role
in the tripleo puzzle so we should be aware of breakages in the
tripleo project.

On Mon, Nov 30, 2015 at 3:19 PM, Derek Higgins  wrote:
> Hi All,
>
> A few months tripleo switch from its devtest based CI to one that was
> based on instack. Before doing this we anticipated disruption in the ci jobs
> and removed them from non tripleo projects.
>
> We'd like to investigate adding it back to heat and ironic as these are
> the two projects where we find our ci provides the most value. But we can
> only do this if the results from the job are treated as voting.
>
> In the past most of the non tripleo projects tended to ignore the
> results from the tripleo job as it wasn't unusual for the job to broken for
> days at a time. The thing is, ignoring the results of the job is the reason
> (the majority of the time) it was broken in the first place.
> To decrease the number of breakages we are now no longer running master
> code for everything (for the non tripleo projects we bump the versions we
> use periodically if they are working). I believe with this model the CI jobs
> we run have become a lot more reliable, there are still breakages but far
> less frequently.
>
> What I proposing is we add at least one of our tripleo jobs back to both
> heat and ironic (and other projects associated with them e.g. clients,
> ironicinspector etc..), tripleo will switch to running latest master of
> those repositories and the cores approving on those projects should wait for
> a passing CI jobs before hitting approve. So how do people feel about doing
> this? can we give it a go? A couple of people have already expressed an
> interest in doing this but I'd like to make sure were all in agreement
> before switching it on.
>
> thanks,
> Derek.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-11-30 Thread Steve Martinelli

In the Mitaka release, the keystone team will be removing functionality
that was marked for deprecation in Kilo, and marking certain functions as
deprecated in Mitaka (that may be removed in at least 2 cycles).

removing deprecated functionality
=

This is not a full list, but these are by and large the most contentious
topics.

* Eventlet support: This was marked as deprecated back in Kilo and is
currently scheduled to be removed in Mitaka in favor of running keystone in
a WSGI server. This is currently how we test keystone in the gate, and
based on the feedback we received at the summit, a lot of folks have moved
to running keystone under Apache since we’ve announced this change.
OpenStack's CI is configured to mainly test using this deployment model.
See [0] for when we started to issue warnings.

* Using LDAP to store assignment data: Like eventlet support, this feature
was also deprecated in Kilo and scheduled to be removed in Mitaka. To store
assignment data (role assignments) we suggest using an SQL based backend
rather than LDAP. See [1] for when we started to issue warnings.

* Using LDAP to store project and domain data: The same as above, see [2]
for when we started to issue warnings.

* for a complete list:
https://blueprints.launchpad.net/keystone/+spec/removed-as-of-mitaka

functions deprecated as of mitaka
=

The following will adhere to the TC’s new standard on deprecating
functionality [3].

* LDAP write support for identity: We suggest simply not writing to LDAP
for users and groups, this effectively makes create, delete and update of
LDAP users and groups a no-op. It will be removed in the O release.

* PKI tokens: We suggest using UUID or fernet tokens instead. The PKI token
format has had issues with security and causes problems with both horizon
and swift when the token contains an excessively large service catalog. It
will be removed in the O release.

* v2.0 of our API: Lastly, the keystone team recommends using v3 of our
Identity API. We have had the intention of deprecating v2.0 for a while
(since Juno actually), and have finally decided to formally deprecate v2.0.
OpenStack’s CI runs successful v3 only jobs, there is complete feature
parity with v2.0, and we feel the CLI exposed via openstackclient is mature
enough to say with certainty that we can deprecate v2.0. It will be around
for at least FOUR releases, with the authentication routes
(POST /auth/tokens) potentially sticking around for longer.

* for a complete list:
https://blueprints.launchpad.net/keystone/+spec/deprecated-as-of-mitaka

If you have ANY concern about the following, please speak up now and let us
know!

Thanks!

Steve Martinelli
OpenStack Keystone Project Team Lead

[0]
https://github.com/openstack/keystone/blob/b475040636ccc954949e6372a60dd86845644611/keystone/server/eventlet.py#L77-L80
[1]
https://github.com/openstack/keystone/blob/28a30f53a6c0d4e84d60795e08f137e8194abbe9/keystone/assignment/backends/ldap.py#L34
[2]
https://github.com/openstack/keystone/blob/28a30f53a6c0d4e84d60795e08f137e8194abbe9/keystone/resource/backends/ldap.py#L36-L39

[3]
http://governance.openstack.org/reference/tags/assert_follows-standard-deprecation.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

On 11/30/2015 12:17 PM, Dmitry Tantsur wrote:
> On 11/30/2015 05:34 PM, Anita Kuno wrote:
>> On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:
>>> Hi,
>>>
>>>
>>>
>>>   This is to announce that  we have  setup  a  Third Party CI
>>> environment
>>> for Proliant iLO Drivers. The results will be posted  under "HP
>>> Proliant CI
>>> check" section in Non-voting mode.   We will be  running the basic
>>> deploy
>>> tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We
>>> will
>>> first  pursue to make the results consistent and over a period of
>>> time we
>>> will try to promote it to voting mode.
>>>
>>>
>>>
>>> For more information check the Wiki:
>>> https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci
>>> ,
>>> for any issues please contact ilo_driv...@groups.ext.hpe.com
>>>
>>>
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> Gururaja Grandhi
>>>
>>> R&D Project Manager
>>>
>>> HPE Proliant  Ironic  Project
>>>
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> Please do not post announcements to the mailing list about the existence
>> of your third party ci system.
> 
> Could you please explain why? As a developer I appreciated this post.
> 
>>
>> Ensure your third party ci system is listed here:
>> https://wiki.openstack.org/wiki/ThirdPartySystems (there are
>> instructions on the bottom of the page) as well as fill out a template
>> on your system so that folks can find your third party ci system the
>> same as all other third party ci systems.
> 
> Wiki is not an announcement FWIW.

If Ironic wants to hear about announced drivers they have agreed to do
so as part of their weekly irc meeting:
2015-11-30T17:19:55   I think it is reasonable for each
driver team, if they want to announce it in the meeting, to do so on the
whiteboard section for their driver. we'll all see that in the weekly
meeting
2015-11-30T17:20:08   but it will avoid spamming the whole
openstack list

http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-30.log

Thank you,
Anita.

> 
>>
>> Ensure you are familiar with the requirements for third party systems
>> listed here:
>> http://docs.openstack.org/infra/system-config/third_party.html#requirements
>>
>>
>> Thank you,
>> Anita.
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers


On 11/30/2015 06:24 PM, Anita Kuno wrote:

On 11/30/2015 12:17 PM, Dmitry Tantsur wrote:

On 11/30/2015 05:34 PM, Anita Kuno wrote:

On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:

Hi,



   This is to announce that  we have  setup  a  Third Party CI
environment
for Proliant iLO Drivers. The results will be posted  under "HP
Proliant CI
check" section in Non-voting mode.   We will be  running the basic
deploy
tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We
will
first  pursue to make the results consistent and over a period of
time we
will try to promote it to voting mode.



 For more information check the Wiki:
https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci
,
for any issues please contact ilo_driv...@groups.ext.hpe.com





Thanks & Regards,

Gururaja Grandhi

R&D Project Manager

HPE Proliant  Ironic  Project



__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Please do not post announcements to the mailing list about the existence
of your third party ci system.


Could you please explain why? As a developer I appreciated this post.



Ensure your third party ci system is listed here:
https://wiki.openstack.org/wiki/ThirdPartySystems (there are
instructions on the bottom of the page) as well as fill out a template
on your system so that folks can find your third party ci system the
same as all other third party ci systems.


Wiki is not an announcement FWIW.


If Ironic wants to hear about announced drivers they have agreed to do
so as part of their weekly irc meeting:
2015-11-30T17:19:55   I think it is reasonable for each
driver team, if they want to announce it in the meeting, to do so on the
whiteboard section for their driver. we'll all see that in the weekly
meeting
2015-11-30T17:20:08   but it will avoid spamming the whole
openstack list

http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-30.log


I was there and I already said that I'm not buying into "spamming the 
list" argument. There are much less important things that I see here 
right now, even though I do actively use filters to only see potentially 
relevant things. We've been actively (and not very successfully) 
encouraging people to use ML instead of IRC conversations (or even 
private messages and video chats), and this thread does not seem in line 
with it.




Thank you,
Anita.





Ensure you are familiar with the requirements for third party systems
listed here:
http://docs.openstack.org/infra/system-config/third_party.html#requirements


Thank you,
Anita.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Call for review focus

2015-11-30 Thread Carl Baldwin

On Tue, Nov 24, 2015 at 4:47 AM, Rossella Sblendido  wrote:
>> I looked for the address scopes blueprint [1] which is targeted for
>> Mitaka-1 [2] and there are 6 (or 5, one is in the gate) patches on the
>> bp/address-scopes topic [3].  It isn't obvious to me yet why it didn't
>> get picked up on the dashboard.  I've only started to look in to this
>> and may not have much time right now.  I wondered if you could easily
>> tell why it didn't get picked up.
>>
>> Isn't it missing bp/ ? From the URL I can only see topic:address-scope,
>> which isn't the right one.
>
>
> Yes Armando is right. I fixed that. Another reason is that I am filtering
> out patches that are WIP or that failed Jenkins tests. This can be changed
> anyway. This is what I get now (after fixing the missing 'bp/') [1]

I tend to think that eliminating patches that are failing tests may
not be a good thing to do in general.  I think it makes the dashboard
too dynamic in the face of all of the unreliable tests that we see
come and go.  It would go dark when we see those check queue bombs
that fail all of the patches.  This is just my opinion and I could
probably adjust the end result to suit my tastes.

I do filter patch sets with failing tests in some of my own personal queries.

Carl

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Call for review focus

2015-11-30 Thread Carl Baldwin

++

On Wed, Nov 25, 2015 at 3:05 PM, Assaf Muller  wrote:
> On Mon, Nov 23, 2015 at 7:02 AM, Rossella Sblendido  
> wrote:
>>
>>
>> On 11/20/2015 03:54 AM, Armando M. wrote:
>>>
>>>
>>>
>>> On 19 November 2015 at 18:26, Assaf Muller >> > wrote:
>>>
>>> On Wed, Nov 18, 2015 at 9:14 PM, Armando M. >> > wrote:
>>> > Hi Neutrites,
>>> >
>>> > We are nearly two weeks away from the end of Mitaka 1.
>>> >
>>> > I am writing this email to invite you to be mindful to what you
>>> review,
>>> > especially in the next couple of weeks. Whenever you have the time
>>> to review
>>> > code, please consider giving priority to the following:
>>> >
>>> > Patches that target blueprints targeted for Mitaka;
>>> > Patches that target bugs that are either critical or high;
>>> > Patches that target rfe-approved 'bugs';
>>> > Patches that target specs that have followed the most current
>>> submission
>>> > process;
>>>
>>> Is it possible to create Gerrit dashboards for patches that answer
>>> these
>>> criteria, and then persist the links in Neutron's dashboards devref
>>> page?
>>> http://docs.openstack.org/developer/neutron/dashboards/index.html
>>> That'd be super useful.
>>>
>>>
>>> We should look into that, but to be perfectly honest I am not sure how
>>> easy it would be, since we'd need to cross-reference content that lives
>>> into gerrit as well as launchpad. Would that even be possible?
>>
>>
>> To cross-reference we can use the bug ID or the blueprint name.
>>
>> I created a script that queries launchpad to get:
>> 1) Bug number of the bugs tagged with approved-rfe
>> 2) Bug number of the critical/high bugs
>> 3) list of blueprints targeted for the current milestone (mitaka-1)
>>
>> With this info the script builds a .dash file that can be used by
>> gerrit-dash-creator [2] to produce a dashboard url .
>>
>> The script prints also the queries that can be used in gerrit UI directly,
>> e.g.:
>> Critical/High Bugs
>> (topic:bug/1399249 OR topic:bug/1399280 OR topic:bug/1443421 OR
>> topic:bug/1453350 OR topic:bug/1462154 OR topic:bug/1478100 OR
>> topic:bug/1490051 OR topic:bug/1491131 OR topic:bug/1498790 OR
>> topic:bug/1505575 OR topic:bug/1505843 OR topic:bug/1513678 OR
>> topic:bug/1513765 OR topic:bug/1514810)
>>
>>
>> This is the dashboard I get right now [3]
>>
>> I tried in many ways to get Gerrit to filter patches if the commit message
>> contains a bug ID. Something like:
>>
>> (message:"#1399249" OR message:"#1399280" OR message:"#1443421" OR
>> message:"#1453350" OR message:"#1462154" OR message:"#1478100" OR
>> message:"#1490051" OR message:"#1491131" OR message:"#1498790" OR
>> message:"#1505575" OR message:"#1505843" OR message:"#1513678" OR
>> message:"#1513765" OR message:"#1514810")
>>
>> but it doesn't work well, the result of the filter contains patches that
>> have nothing to do with the bugs queried.
>> That's why I had to filter using the topic.
>>
>> CAVEAT: To make the dashboard work, bug fixes must use the topic "bug/ID"
>> and patches implementing a blueprint the topic "bp/name". If a patch is not
>> following this convention it won't be showed in the dashboard, since the
>> topic is used as filter. Most of us use this convention already anyway so I
>> hope it's not too much of a burden.
>>
>> Feedback is appreciated :)
>
> Rossella this is exactly what I wanted :) Let's iterate on the patch
> and merge it.
> We could then consider running the script automatically on a daily
> basis and publishing the
> resulting URL in a nice bookmarkable place.
>
>>
>> [1] https://review.openstack.org/248645
>> [2] https://github.com/openstack/gerrit-dash-creator
>> [3] https://goo.gl/sglSbp
>>
>>>
>>> Btw, I was looking at the current blueprint assignments [1] for Mitaka:
>>> there are some blueprints that still need assignee, approver and
>>> drafter; we should close the gap. If there are volunteers, please reach
>>> out to me.
>>>
>>> Thanks,
>>> Armando
>>>
>>> [1] https://blueprints.launchpad.net/neutron/mitaka/+assignments
>>>
>>>
>>> >
>>> > Everything else should come later, no matter how easy or interesting
>>> it is
>>> > to review; remember that as a community we have the collective duty
>>> to work
>>> > towards a common (set of) target(s), as being planned in
>>> collaboration with
>>> > the Neutron Drivers team and the larger core team.
>>> >
>>> > I would invite submitters to ensure that the Launchpad resources
>>> > (blueprints, and bug report) capture the most updated view in terms
>>> of
>>> > patches etc. Work with your approver to help him/her be focussed
>>> where it
>>> > matters most.
>>> >
>>> > Finally, we had plenty of discussions at the design summit, and some
>>> of
>>> > those discussions will have to be followed up with actions (aka code
>>> in
>>> > OpenStack lingo). Even though, we no long

Re: [openstack-dev] [all][glance] Add Sabari Kumar Murugesan

2015-11-30 Thread Ian Cordasco

On 11/23/15, 14:20, "Flavio Percoco"  wrote:

>Greetings,
>
>I'd like to propose adding Sabari Kumar Murugesan to the glance-core
>team. Sabari has been contributing for quite a bit to the project with
>great reviews and he's also been providing great feedback in matters
>related to the design of the service, libraries and other areas of the
>team.
>
>I believe he'd be a great addition to the glance-core team as he has
>demonstrated a good knowledge of the code, service and project's
>priorities.
>
>If Sabari accepts to join and there are no objections from other
>members of the community, I'll proceed to add Sabari to the team in a
>week from now.
>
>Thanks,
>Flavio
>
>-- 
>@flaper87
>Flavio Percoco

I'm no longer a core, but I'm +1 on this.

Congrats Sabari!

--
Ian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

On 11/30/2015 12:33 PM, Dmitry Tantsur wrote:
> On 11/30/2015 06:24 PM, Anita Kuno wrote:
>> On 11/30/2015 12:17 PM, Dmitry Tantsur wrote:
>>> On 11/30/2015 05:34 PM, Anita Kuno wrote:
 On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:
> Hi,
>
>
>
>This is to announce that  we have  setup  a  Third Party CI
> environment
> for Proliant iLO Drivers. The results will be posted  under "HP
> Proliant CI
> check" section in Non-voting mode.   We will be  running the basic
> deploy
> tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We
> will
> first  pursue to make the results consistent and over a period of
> time we
> will try to promote it to voting mode.
>
>
>
>  For more information check the Wiki:
> https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci
>
> ,
> for any issues please contact ilo_driv...@groups.ext.hpe.com
>
>
>
>
>
> Thanks & Regards,
>
> Gururaja Grandhi
>
> R&D Project Manager
>
> HPE Proliant  Ironic  Project
>
>
>
> __
>
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

 Please do not post announcements to the mailing list about the
 existence
 of your third party ci system.
>>>
>>> Could you please explain why? As a developer I appreciated this post.
>>>

 Ensure your third party ci system is listed here:
 https://wiki.openstack.org/wiki/ThirdPartySystems (there are
 instructions on the bottom of the page) as well as fill out a template
 on your system so that folks can find your third party ci system the
 same as all other third party ci systems.
>>>
>>> Wiki is not an announcement FWIW.
>>
>> If Ironic wants to hear about announced drivers they have agreed to do
>> so as part of their weekly irc meeting:
>> 2015-11-30T17:19:55   I think it is reasonable for each
>> driver team, if they want to announce it in the meeting, to do so on the
>> whiteboard section for their driver. we'll all see that in the weekly
>> meeting
>> 2015-11-30T17:20:08   but it will avoid spamming the whole
>> openstack list
>>
>> http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-30.log
>>
> 
> I was there and I already said that I'm not buying into "spamming the
> list" argument. There are much less important things that I see here
> right now, even though I do actively use filters to only see potentially
> relevant things. We've been actively (and not very successfully)
> encouraging people to use ML instead of IRC conversations (or even
> private messages and video chats), and this thread does not seem in line
> with it.

Please discuss this with the leadership of your project.

All announcements about the existence of a third party ci will be
redirected to the third party systems wikipage.

Thank you,
Anita.

> 
>>
>> Thank you,
>> Anita.
>>
>>>

 Ensure you are familiar with the requirements for third party systems
 listed here:
 http://docs.openstack.org/infra/system-config/third_party.html#requirements



 Thank you,
 Anita.
>>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Ruby Loo

On 30 November 2015 at 10:19, Derek Higgins  wrote:

> Hi All,
>
> A few months tripleo switch from its devtest based CI to one that was
> based on instack. Before doing this we anticipated disruption in the ci
> jobs and removed them from non tripleo projects.
>
> We'd like to investigate adding it back to heat and ironic as these
> are the two projects where we find our ci provides the most value. But we
> can only do this if the results from the job are treated as voting.
>

What does this mean? That the tripleo job could vote and do a -1 and block
ironic's gate?


>
> In the past most of the non tripleo projects tended to ignore the
> results from the tripleo job as it wasn't unusual for the job to broken for
> days at a time. The thing is, ignoring the results of the job is the reason
> (the majority of the time) it was broken in the first place.
> To decrease the number of breakages we are now no longer running
> master code for everything (for the non tripleo projects we bump the
> versions we use periodically if they are working). I believe with this
> model the CI jobs we run have become a lot more reliable, there are still
> breakages but far less frequently.
>
> What I proposing is we add at least one of our tripleo jobs back to both
> heat and ironic (and other projects associated with them e.g. clients,
> ironicinspector etc..), tripleo will switch to running latest master of
> those repositories and the cores approving on those projects should wait
> for a passing CI jobs before hitting approve. So how do people feel about
> doing this? can we give it a go? A couple of people have already expressed
> an interest in doing this but I'd like to make sure were all in agreement
> before switching it on.
>
> This seems to indicate that the tripleo jobs are non-voting, or at least
won't block the gate -- so I'm fine with adding tripleo jobs to ironic. But
if you want cores to wait/make sure they pass, then shouldn't they be
voting? (Guess I'm a bit confused.)

--ruby
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][all] The lock files saga (and where we can go from here)


Hi all,

I just wanted to bring up an issue, possible solution and get feedback 
on it from folks because it seems to be an on-going problem that shows 
up not when an application is initially deployed but as on-going 
operation and running of that application proceeds (ie after running for 
a period of time).


The jist of the problem is the following:

A <> has a need to ensure that no 
application on the same machine can manipulate a given resource on that 
same machine, so it uses the lock file pattern (acquire a *local* lock 
file for that resource, manipulate that resource, release that lock 
file) to do actions on that resource in a safe manner (note this does 
not ensure safety outside of that machine, lock files are *not* 
distributed locks).


The api that we expose from oslo is typically accessed via the following:

  oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None, 
external=False, lock_path=None, semaphores=None, delay=0.01)


or via its underlying library (that I extracted from oslo.concurrency 
and have improved to add more usefulness) @ 
http://fasteners.readthedocs.org/


The issue though for <> is that each of 
these projects now typically has a large amount of lock files that exist 
or have existed and no easy way to determine when those lock files can 
be deleted (afaik no? periodic task exists in said projects to clean up 
lock files, or to delete them when they are no longer in use...) so what 
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387 
appear and there is no a simple solution to clean lock files up (since 
oslo.concurrency is really not the right layer to know when a lock can 
or can not be deleted, only the application knows that...)


So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in > cleaning up these files (are they at all?)


Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single* 
lock file per project and use offsets into it as the way to lock. For 
example nova could/would create a 1MB (or larger/smaller) *empty* file 
for locks, that would allow for 1,048,576 locks to be used at the same 
time, which honestly should be way more than enough, and then there 
would not need to be any lock cleanup at all... Is there any reason this 
wasn't initially done back way when this lock file code was created? 
(https://github.com/harlowja/fasteners/pull/10 adds this functionality 
to the underlying library if people want to look it over)


In general would like to hear peoples thoughts/ideas/complaints/other,

-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova]New Quota Subteam on Nova

2015-11-30 Thread melanie witt

On Nov 26, 2015, at 9:36, John Garbutt  wrote:

> A suggestion in the past, that I like, is creating a nova functional
> test that stress tests the quota code.
> 
> Hopefully that will be able to help reproduce the error.
> That should help prove if any proposed fix actually works.

+1, I think it's wise to get some data on the current state of quotas before 
choosing a redesign. IIRC, Joe Gordon described a test scenario he used to use 
to reproduce quota bugs locally, in one of the launchpad bugs. If we could 
automate something like that, we could use it to demonstrate how quotas 
currently behave during parallel requests and try things like disabling 
reservations. I also like the idea of being able to verify the effects of 
proposed fixes.

-melanie (irc: melwitt)







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release] using reno for libraries

2015-11-30 Thread Doug Hellmann

Excerpts from Dmitry Tantsur's message of 2015-11-30 17:58:45 +0100:
> On 11/30/2015 05:14 PM, Doug Hellmann wrote:
> > Excerpts from Dmitry Tantsur's message of 2015-11-30 10:06:25 +0100:
> >> On 11/28/2015 02:48 PM, Doug Hellmann wrote:
> >>> Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:
>  Liaisons,
> 
>  We're making good progress on adding reno to service projects as
>  we head to the Mitaka-1 milestone. Thank you!
> 
>  We also need to add reno to all of the other deliverables with
>  changes that might affect deployers. That means clients and other
>  libraries, SDKs, etc. with configuration options or where releases
>  can change deployment behavior in some way. Now that most teams
>  have been through this conversion once, it should be easy to replicate
>  for the other repositories in a similar way.
> 
>  Libraries have 2 audiences for release notes: developers consuming
>  the library and deployers pushing out new versions of the libraries.
>  To separate the notes for the two audiences, and avoid doing manually
>  something that we have been doing automatically, we can use reno
>  just for deployer release notes (changes in support for options,
>  drivers, etc.). That means the library repositories that need reno
>  should have it configured just like for the service projects, with
>  the separate jobs and a publishing location different from their
>  existing developer documentation. The developer docs can continue
>  to include notes for the developer audience.
> >>>
> >>> I've had a couple of questions about this split for release notes. The
> >>> intent is for developer-focused notes to continue to come from commit
> >>> messages and in-tree documentation, while using reno for new and
> >>> additional deployer-focused communication. Most commits to libraries
> >>> won't need reno release notes.
> >>
> >> This looks like unnecessary overcomplication. Why not use such a
> >> convenient tool for both kinds of release notes instead of having us
> >> invent and maintain one more place to put release notes, now for
> >
> > In the past we have had rudimentary release notes and changelogs
> > for developers to read based on the git commit messages. Since
> > deployers and developers care about different things, we don't want
> > to make either group sift through the notes meant for the other.
> > So, we publish notes in different ways.
> 
> Hmm, so maybe for small libraries with few changes it's still fine to 
> publish them together, what do you think?

I'm not sure why you would want to do that. Publishing the ChangeLog
contents in the developer documentation is (or can be) completely
automatic. It should only be possible to add reno notes for
deployer-facing changes, and those notes will need to be written
in a way the deployer can understand, which is not necessarily a
requirement for a commit message.

> > The thing that is new here is publishing release notes for changes
> > in libraries that deployers need to know about. While the Oslo code
> > was in the incubator, and being copied into applications, it was
> > possible to detect deployer-focused changes like new or deprecated
> > configuration options in the application and put the notes there.
> > Using shared libraries means those changes can happen without
> > application developers being aware of them, so the library maintainers
> > need to be publishing notes. Using reno for those notes is consistent
> > with the way they are handled in the applications, so we're extending
> > one tool to more repositories.
> >
> >> developers? It's already not so easy to explain reno to newcomers, this
> >> idea makes it even harder...
> >
> > Can you tell me more about the difficulty you've had? I would like to
> > improve the documentation for reno and for how we use it.
> 
> Usually people are stuck at the "how do I do this at all" stage :) we've 
> even added it to the ironic developer FAQ. As to me, the official reno 
> documentation is nice enough (but see below), maybe people are not aware 
> of it.
> 
> Another "issue" (at least for our newcomers) with reno docs is that 
> http://docs.openstack.org/developer/reno/usage.html#generating-a-report 
> mentions the "reno report" command which is not something we all 
> actually use, we use these "tox -ereleasenotes" command. What is worse, 
> this command (I guess it's by design) does not catch release note files 
> that are just created locally. It took me time to figure out that I have 
> to commit release notes before "tox -ereleasenotes" would show them in 
> the rendered HTML.

The reno documentation is written for any user, not just OpenStack
developers. Those instructions should work if reno is installed,
even though we've wrapped reno in tox to make it simpler to run in
the gate. We can add some information about using tox to build
locally to the project team guide.

I'll look into

Re: [openstack-dev] [Nova] Nova mid cycle details

2015-11-30 Thread Murray, Paul (HP Cloud)

The rates are listed on the hotel information page on that site. All include 
tax and breakfast.

Paul

From: Michael Still [mailto:mi...@stillhq.com]
Sent: 28 November 2015 04:39
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] Nova mid cycle details

Hey,

I filled in the first part of that page, but when it got to hotels I got 
confused. The web site doesn't seem to mention the night rate for the HP price. 
Do you know what that is?

Thanks,
Michael

On Sat, Nov 28, 2015 at 3:06 AM, Murray, Paul (HP Cloud) 
mailto:pmur...@hpe.com>> wrote:
The Nova Mitaka mid cycle meetup is in Bristol, UK at the Hewlett Packard 
Enterprise office.

The mid cycle wiki page is here:
https://wiki.openstack.org/wiki/Sprints/NovaMitakaSprint

Note that there is a web site for signing up for the event and booking hotel 
rooms at a reduced event rate here:
https://starcite.smarteventscloud.com/hpe/NovaMidCycleMeeting

If you want to book a room at the event rate you do need to register on that 
site.

There is also an Eventbrite event that was created before the above web site 
was available. Do not worry if you have registered using Eventbrite, we will 
recognize those registrations as well. But if you do want to book a room you 
will need to register again on the above site.

Paul

Paul Murray
Nova Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Infra] Meeting Tuesday December 1st at 19:00 UTC

2015-11-30 Thread Elizabeth K. Joseph

Hi everyone,

The OpenStack Infrastructure (Infra) team is having our next weekly
meeting on Tuesday December 1st, at 19:00 UTC in #openstack-meeting

Meeting agenda available here:
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting

Anyone is welcome to to add agenda items and everyone interested in
the project infrastructure and process surrounding automated testing
and deployment is encouraged to attend.

In case you missed it or would like a refresher, the meeting minutes
and full logs from our last meeting are available:

Minutes: 
http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-11-24-19.01.html
Minutes (text):
http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-11-24-19.01.txt
Log: 
http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-11-24-19.01.log.html

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Performance] Filling performance team working items list and taking part in their resolving

2015-11-30 Thread Dina Belova

Matt,

I guess it's good solution. Thanks for doing that!

Cheers,
Dina

On Mon, Nov 30, 2015 at 6:30 PM, Matt Riedemann 
wrote:

>
>
> On 11/27/2015 3:54 AM, Dina Belova wrote:
>
>> Hey OpenStack devs and operators!
>>
>> Folks, I would like to share list of working items Performance Team is
>> currently having in the backlog -
>> https://etherpad.openstack.org/p/perf-zoom-zoom [Work Items to grab]
>> section. I'm really encouraging you to fill it with concrete pieces of
>> work you think will be useful and take part in the
>> development/investigation by assigning some of them to yourself and
>> working on them :)
>>
>> Cheers,
>> Dina
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> One of the work items was to enable the nova metadata service in a large
> ops job. There are two voting large ops jobs in nova, one runs with
> nova-network and the other runs with neutron. Besides those differences,
> I'm not sure if there is any other difference in the jobs. So I guess we'd
> just need to pick which one runs the nova-api-metadata service rather than
> using config drive. The only other job I know of that runs that is the
> postgres job and that runs nova-network, so I'd say we turn on n-api-meta
> in the neutron large ops job.
>
> Are there any issues with that?
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

Best regards,

Dina Belova

Software Engineer

Mirantis Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Sahara] Debugging via PyCharm (pydevd)

2015-11-30 Thread Chad Roberts

I submitted a bug (https://bugs.launchpad.net/sahara/+bug/1521266) and have
asked in-channel, but I'll also post here just in case it rings a bell for
someone.

Recently, Sahara was changed to use oslo.service to launch our wsgi api
server.  Prior to that change, I was able to successfully run and debug the
sahara-api process using PyCharm.

After the change to use oslo.service as the launcher, I am still able to
run sahara-api using the same config that has worked for as long as I can
remember, but when I run in debug mode, the api appears to start up
normally, but any request I send never receives a response and any
breakpoints I've tried never seem to get hit.

If I backout the changes to use oslo.service, I am able to debug
successfully again.

Any chance that sort of thing sounds familiar to anyone?

Thanks,
Chad
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone][all] keystone-spec proposal freeze date and using roll call instead of +2/+2/+A for specs

2015-11-30 Thread Steve Martinelli



As a reminder: keystone spec freeze date is the end of mitaka-1 (this
friday), any spec being proposed for mitaka will have to go through an
exception process next week.

For the next keystone meeting (tomorrow) I'd like to propose we use a roll
call / vote mechanism on each open specs that is proposed against the
mitaka release. If a majority of the cores (50% or more) agree that a spec
aligns with project plans for mitaka, then we should get it merged by
friday (clean up any wording and proposed APIs) so the author can start
working on it on monday.

I added [all] because I wanted to know how other projects approach this
subject. I think a roll call feature for -specs would be great, similar to
how there is one for governance changes. Do others just use +2/+2/+A? In
keystone we've often let specs sit for a little to give other cores a
chance to chime in. Thoughts?

Thanks,

Steve Martinelli
OpenStack Keystone Project Team Lead
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-30 Thread Andrew Laski


On 11/30/15 at 07:32am, Sean Dague wrote:

On 11/24/2015 10:09 AM, John Garbutt wrote:

On 24 November 2015 at 15:00, Balázs Gibizer
 wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 24, 2015 15:35
On 11/24/15 at 10:26am, Balázs Gibizer wrote:

From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
Sent: November 23, 2015 22:33
On 11/23/2015 2:23 PM, Andrew Laski wrote:

On 11/23/15 at 04:43pm, Balázs Gibizer wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 23, 2015 17:03

On 11/23/15 at 08:54am, Ryan Rossiter wrote:



On 11/23/2015 5:33 AM, John Garbutt wrote:

On 20 November 2015 at 09:37, Balázs Gibizer
 wrote:






There is a bit I am conflicted/worried about, and thats when we
start including verbatim, DB objects into the notifications. At
least you can now quickly detect if that blob is something
compatible with your current parsing code. My preference is
really to keep the Notifications as a totally separate object
tree, but I am sure there are many cases where that ends up
being seemingly stupid duplicate work. I am not expressing this
well in text form :(

Are you saying we don't want to be willy-nilly tossing DB
objects across the wire? Yeah that was part of the rug-pulling
of just having the payload contain an object. We're
automatically tossing everything with the object then, whether
or not some of that was supposed to be a secret. We could add
some sort of property to the field like
dont_put_me_on_the_wire=True (or I guess a
notification_ready() function that helps an object sanitize
itself?) that the notifications will look at to know if it puts
that on the wire-serialized dict, but that's adding a lot more
complexity and work to a pile that's already growing rapidly.


I don't want to be tossing db objects across the wire.  But I
also am not convinced that we should be tossing the current
objects over the wire either.
You make the point that there may be things in the object that
shouldn't be exposed, and I think object version bumps is another
thing to watch out for.
So far the only object that has been bumped is Instance but in
doing so no notifications needed to change.  I think if we just
put objects into notifications we're coupling the notification
versions to db or RPC changes unnecessarily.  Some times they'll
move together but other times, like moving flavor into
instance_extra, there's no reason to bump notifications.



Sanitizing existing versioned objects before putting them to the
wire is not hard to do.
You can see an example of doing it in
https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
cm
L382.
We don't need extra effort to take care of minor version bumps
because that does not break a well written consumer. We do have to
take care of the major version bumps but that is a rare event and
therefore can be handled one by one in a way John suggested, by
keep sending the previous major version for a while too.


That review is doing much of what I was suggesting.  There is a
separate notification and payload object.  The issue I have is that
within the ServiceStatusPayload the raw Service object and version
is being dumped, with the filter you point out.  But I don't think
that consumers really care about tracking Service object versions
and dealing with compatibility there, it would be easier for them
to track the ServiceStatusPayload version which can remain
relatively stable even if Service is changing to adapt to db/RPC changes.

Not only do they not really care about tracking the Service object
versions, they probably also don't care about what's in that filter list.

But I think you're getting on the right track as to where this needs
to go. We can integrate the filtering into the versioning of the payload.
But instead of a blacklist, we turn the filter into a white list. If
the underlying object adds a new field that we don't want/care if
people know about, the payload version doesn't have to change. But if
we add something (or if we're changing the existing fields) that we
want to expose, we then assert that we need to update the version of
the payload, so the consumer can look at the payload and say "oh, in
1.x, now I get ___" and can add the appropriate checks/compat.
Granted with this you can get into rebase nightmares ([1] still
haunts me in my sleep), but I don't see us frantically changing the
exposed fields all too often. This way gives us some form of
pseudo-pinning of the subobject. Heck, in this method, we could even
pass the whitelist on the wire right? That way we tell the consumer

explicitly what's available to them (kinda like a fake schema).


I think see your point, and it seems like a good way forward. Let's
turn the black list to a white list. Now I'm thinking about creating a
new Field type something like WhiteListedObjectField which get a type
name (as the ObjectField) but also get a white_list that describes which

fields needs to be used from the original type.

Then this new field

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Ben Nemec

On 11/30/2015 12:42 PM, Joshua Harlow wrote:
> Hi all,
> 
> I just wanted to bring up an issue, possible solution and get feedback 
> on it from folks because it seems to be an on-going problem that shows 
> up not when an application is initially deployed but as on-going 
> operation and running of that application proceeds (ie after running for 
> a period of time).
> 
> The jist of the problem is the following:
> 
> A <> has a need to ensure that no 
> application on the same machine can manipulate a given resource on that 
> same machine, so it uses the lock file pattern (acquire a *local* lock 
> file for that resource, manipulate that resource, release that lock 
> file) to do actions on that resource in a safe manner (note this does 
> not ensure safety outside of that machine, lock files are *not* 
> distributed locks).
> 
> The api that we expose from oslo is typically accessed via the following:
> 
>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None, 
> external=False, lock_path=None, semaphores=None, delay=0.01)
> 
> or via its underlying library (that I extracted from oslo.concurrency 
> and have improved to add more usefulness) @ 
> http://fasteners.readthedocs.org/
> 
> The issue though for <> is that each of 
> these projects now typically has a large amount of lock files that exist 
> or have existed and no easy way to determine when those lock files can 
> be deleted (afaik no? periodic task exists in said projects to clean up 
> lock files, or to delete them when they are no longer in use...) so what 
> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387 
> appear and there is no a simple solution to clean lock files up (since 
> oslo.concurrency is really not the right layer to know when a lock can 
> or can not be deleted, only the application knows that...)
> 
> So then we get a few creative solutions like the following:
> 
> - https://review.openstack.org/#/c/241663/
> - https://review.openstack.org/#/c/239678/
> - (and others?)
> 
> So I wanted to ask the question, how are people involved in < favorite openstack project>> cleaning up these files (are they at all?)
> 
> Another idea that I have been proposing also is to use offset locks.
> 
> This would allow for not creating X lock files, but create a *single* 
> lock file per project and use offsets into it as the way to lock. For 
> example nova could/would create a 1MB (or larger/smaller) *empty* file 
> for locks, that would allow for 1,048,576 locks to be used at the same 
> time, which honestly should be way more than enough, and then there 
> would not need to be any lock cleanup at all... Is there any reason this 
> wasn't initially done back way when this lock file code was created? 
> (https://github.com/harlowja/fasteners/pull/10 adds this functionality 
> to the underlying library if people want to look it over)

I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
 Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.

Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.

I know we've had this discussion in the past, but I don't think anyone
has ever told me that having lock files hang around was a functional
problem for them.  It seems to be largely cosmetic complaints about not
cleaning up the old files (which, as you noted, Oslo can't really solve
because we have no idea when consumers are finished with locks) and
given the amount of trouble we've had with interprocess locking in the
past I've never felt that a cosmetic issue was sufficient reason to
reopen that can of worms.  I'll just note again that every time we've
started messing with this stuff we run into a bunch of sticky problems
and edge cases, so it would take a pretty c

Re: [openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Steve Baker


On 30/11/15 23:21, Steven Hardy wrote:

On Mon, Nov 30, 2015 at 10:03:29AM +0100, Lennart Regebro wrote:

I'm tasked to implement a command that shows error messages when a
deployment has failed. I have a vague memory of having seen scripts
that do something like this, if that exists, can somebody point me in
teh right direction?

I wrote a super simple script and put it in a blog post a while back:

http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html

All it does is find the failed SoftwareDeployment resources, then do heat
deployment-show on the resource, so you can see the stderr associated with
the failure.

Having tripleoclient do that by default would be useful.


Any opinions on what that should do, specifically? Traverse failed
resources to find error messages, I assume. Anything else?

Yeah, but I think for this to be useful, we need to go a bit deeper than
just showing the resource error - there are a number of typical failure
modes, and I end up repeating the same steps to debug every time.

1. SoftwareDeployment failed (mentioned above).  Every time, you need to
see the name of the SoftwareDeployment which failed, figure out if it
failed on one or all of the servers, then look at the stderr for clues.

2. A server failed to build (OS::Nova::Server resource is FAILED), here we
need to check both nova and ironic, looking first to see if ironic has the
node(s) in the wrong state for scheduling (e.g nova gave us a no valid
host error), and then if they are OK in ironic, do nova show on the failed
host to see the reason nova gives us for it failing to go ACTIVE.

3. A stack timeout happened.  IIRC when this happens, we currently fail
with an obscure keystone related backtrace due to the token expiring.  We
should instead catch this error and show the heat stack status_reason,
which should say clearly the stack timed out.

If we could just make these three cases really clear and easy to debug, I
think things would be much better (IME the above are a high proportion of
all failures), but I'm sure folks can come up with other ideas to add to
the list.

I'm actually drafting a spec which includes a command which does this. I 
hope to submit it soon, but here is the current state of that command's 
description:


Diagnosing resources in a FAILED state
--

One command will be implemented:
- openstack overcloud failed list

This will print a yaml tree showing the hierarchy of nested stacks until it
gets to the actual failed resource, then it will show information 
regarding the

failure. For most resource types this information will be the status_reason,
but for software-deployment resources the deploy_stdout, deploy_stderr and
deploy_status code will be printed.

In addition to this stand-alone command, this output will also be 
printed when

an ``openstack overcloud deploy`` or ``openstack overcloud update`` command
results in a stack in a FAILED state.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

2015-11-30 Thread sean roberts

Being successful at your first patch for most people means that their first
effort is different than an a regular patch.

Identifying abandoned work more quickly is good. It doesn't help the first
timer.

Tagging low hanging fruit for first timers I like. I'm recommending we add
a mentor as part of the idea, so the project mentor is responsible for the
work and the first timer learning.

On Monday, November 30, 2015, Doug Hellmann  wrote:

> Excerpts from sean roberts's message of 2015-11-30 07:57:54 -0800:
> > How about:
> > First timers assign a bug to a mentor and the mentor takes responsibility
> > for the first timer learning from the bug to completion.
>
> That would mean the learning process is different from what we want the
> regular process to be.
>
> If the problem is identifying "In Progress" bugs that are actually not
> being worked on, then let's figure out a way to make that easier.
> sdague's point about the auto-abandon process may help. We could query
> gerrit for "stale" reviews that would have met the old abandon
> requirements and that refer to bugs, for example. Using that
> information, someone could follow-up with the patch owner to see if it
> is actually abandoned, before changing the bug status or encouraging the
> owner to abandon the patch.
>
> >
> > Per project, a few people volunteer themselves as mentors. As easy as
> > responding to [project][mentor] emails.
> >
> > On Monday, November 30, 2015, Sean Dague >
> wrote:
> >
> > > On 11/25/2015 03:22 PM, Shamail wrote:
> > > > Hi,
> > > >
> > > >> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  
> > > > wrote:
> > > >>
> > > >> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
> > > >>> Hi everyone,
> > > >>>
> > > >>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs
> which
> > > >>> discusses how one open-source project is encouraging contributions
> by
> > > new
> > > >>> open-source contributors through a combination of a special tag
> (which
> > > is
> > > >>> associated with work that is needed but can only be completed by
> > > someone
> > > >>> who is a first-time contributor) and helpful comments in the review
> > > phase
> > > >>> to ensure the contribution(s) eventually get merged.
> > > >>>
> > > >>> While reading the article, I immediately thought about our
> > > >>> low-hanging-fruit bug tag which is used for a very similar purpose
> in
> > > "bug
> > > >>> fixing" section of  the "how to contribute" page[2].  The
> > > low-hanging-fruit
> > > >>> tag is used to identify items that are generally suitable for
> > > first-time or
> > > >>> beginner contributors but, in reality, anyone can pick them up.
> > > >>>
> > > >>> I wanted to propose a new tag (or even changing the, existing,
> > > low-hanging
> > > >>> fruit tag) that would identify items that we are reserving for
> > > first-time
> > > >>> OpenStack contributors (e.g. a patch-set for the item submitted by
> > > someone
> > > >>> who is not a first time contributor would be rejected)... The same
> > > article
> > > >>> that Andrew shared mentions using an "up-for-grabs" tag which also
> > > >>> populates the items at up-for-grabs[3] (a site where people
> looking to
> > > >>> start working on open-source projects see entry-level items from
> > > multiple
> > > >>> projects).  If we move forward with an exclusive tag for
> first-timers
> > > then
> > > >>> it would be nice if we could use the up-for-grabs tag so that
> OpenStack
> > > >>> also shows up on the list too.  Please let me know if this change
> > > should be
> > > >>> proposed elsewhere, the tags are maintained in launchpad and the
> wiki I
> > > >>> found related to bug tags[4] didn't indicate a procedure for
> > > submitting a
> > > >>> change proposal.
> > > >>
> > > >> I like the idea of making bugs suitable for first-timers more
> > > >> discoverable. I'm not sure we need to *reserve* any bugs for any
> class
> > > >> of contributor. What benefit do you think that provides?
> > > > I would have to defer to additional feedback here...
> > > >
> > > > My own perspective from when I was doing my first contribution is
> that
> > > it was hard to find active "low-hanging-fruit" items.  Most were
> already
> > > work-in-progress or assigned.
> > >
> > > This was a direct consequence of us dropping the auto-abandoning of old
> > > code reviews in gerrit. When a review is abandoned the bug is flipped
> > > back to New instead of In Progress.
> > >
> > > I found quite often people go and gobble up bugs assigning them to
> > > themselves, but don't make real progress on them. Then new contributors
> > > show up, and don't work on any of those issues because our tools say
> > > someone is already on top of it.
> > >
> > > -Sean
> > >
> > > --
> > > Sean Dague
> > > http://dague.net
> > >
> > >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> openstack-d

Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Paul Michali

Check out https://freenode.net/irc_servers.shtml which lists the servers. I
was using irc.freenode.net. Switched to weber.freenode.net and able to
connect.

(now everyone will hop on that one and I'll have to pick another :)



On Mon, Nov 30, 2015 at 2:46 PM Clark, Jay  wrote:

> Can't connect either. Dead in the water
>
> Regards,
> Jay Clark
> Sr. OpenStack Deployment Engineer
> E: jason.t.cl...@hpe.com
> H: 919.341.4670
> M: 919.345.1127
> IRC (freenode): jasondotstar
>
> 
> From: lichen.hangzhou [lichen.hangz...@gmail.com]
> Sent: Monday, November 30, 2015 9:17 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: openstack-in...@lists.openstack.org
> Subject: Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues
>
> Can't connect +1 and web client do not work  :(
>
> -chen
>
> At 2015-11-30 22:08:12, "Hinds, Luke (Nokia - GB/Bristol)" <
> luke.hi...@nokia.com> wrote:
> Me too.  It is possible to get on using the web client though;
> https://webchat.freenode.net/ .
>
> On Mon, 2015-11-30 at 14:00 +, EXT Dugger, Donald D wrote:
> I can’t even connect to the IRC server at all, can others get it?
>
> --
> Don Dugger
> "Censeo Toto nos in Kansa esse decisse." - D. Gale
> Ph: 303/443-3786
>
> From: Joshua Hesketh [mailto:joshua.hesk...@gmail.com joshua.hesk...@gmail.com>]
> Sent: Monday, November 30, 2015 2:50 AM
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>;
> openstack-infra  openstack-in...@lists.openstack.org>>
> Subject: [openstack-dev] IRC Bot issues
>
> Hi all,
> Freenode are currently experiencing a severe DDoS attack that are having
> an effect on our bots. As such the meetbot, irc logging and gerrit watcher
> are interminably available.
>
> We expect the bots to resume their normal function once Freenode has
> recovered. For now, meetings may have to be postponed or minuted by hand.
> Cheers,
> Josh
>
> ___
> OpenStack-Infra mailing list
> openstack-in...@lists.openstack.org openstack-in...@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)


Ben Nemec wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)


I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
  Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.


Ya, I wonder how many folks are actually doing this, because the exposed 
API of @synchronized doesn't seem to tell u what file to even delete in 
the first place :-/ perhaps we should make that more accessible so that 
people/consumers of that code could know what to delete...




Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.


Except as we move to never shutting an app down (always online and live 
upgrades and all that jazz), it will have to run more than just on boot, 
but point taken.




I know we've had this discussion in the past, but I don't think anyone
has ever told me that having lock files hang around was a functional
problem for them.  It seems to be largely cosmetic complaints about not
cleaning up the old files (which, as you noted, Oslo can't really solve
because we have no idea when consumers are finished with locks) and
given the amount

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Robert Collins

On 1 December 2015 at 08:37, Ben Nemec  wrote:
> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
>> Hi all,
>>
>> I just wanted to bring up an issue, possible solution and get feedback
>> on it from folks because it seems to be an on-going problem that shows
>> up not when an application is initially deployed but as on-going
>> operation and running of that application proceeds (ie after running for
>> a period of time).
>>
>> The jist of the problem is the following:
>>
>> A <> has a need to ensure that no
>> application on the same machine can manipulate a given resource on that
>> same machine, so it uses the lock file pattern (acquire a *local* lock
>> file for that resource, manipulate that resource, release that lock
>> file) to do actions on that resource in a safe manner (note this does
>> not ensure safety outside of that machine, lock files are *not*
>> distributed locks).
>>
>> The api that we expose from oslo is typically accessed via the following:
>>
>>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
>> external=False, lock_path=None, semaphores=None, delay=0.01)
>>
>> or via its underlying library (that I extracted from oslo.concurrency
>> and have improved to add more usefulness) @
>> http://fasteners.readthedocs.org/
>>
>> The issue though for <> is that each of
>> these projects now typically has a large amount of lock files that exist
>> or have existed and no easy way to determine when those lock files can
>> be deleted (afaik no? periodic task exists in said projects to clean up
>> lock files, or to delete them when they are no longer in use...) so what
>> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
>> appear and there is no a simple solution to clean lock files up (since
>> oslo.concurrency is really not the right layer to know when a lock can
>> or can not be deleted, only the application knows that...)
>>
>> So then we get a few creative solutions like the following:
>>
>> - https://review.openstack.org/#/c/241663/
>> - https://review.openstack.org/#/c/239678/
>> - (and others?)
>>
>> So I wanted to ask the question, how are people involved in <> favorite openstack project>> cleaning up these files (are they at all?)
>>
>> Another idea that I have been proposing also is to use offset locks.
>>
>> This would allow for not creating X lock files, but create a *single*
>> lock file per project and use offsets into it as the way to lock. For
>> example nova could/would create a 1MB (or larger/smaller) *empty* file
>> for locks, that would allow for 1,048,576 locks to be used at the same
>> time, which honestly should be way more than enough, and then there
>> would not need to be any lock cleanup at all... Is there any reason this
>> wasn't initially done back way when this lock file code was created?
>> (https://github.com/harlowja/fasteners/pull/10 adds this functionality
>> to the underlying library if people want to look it over)
>
> I think the main reason was that even with a million locks available,
> you'd have to find a way to hash the lock names to offsets in the file,
> and a million isn't a very large collision space for that.  Having two
> differently named locks that hashed to the same offset would lead to
> incredibly confusing bugs.
>
> We could switch to requiring the projects to provide the offsets instead
> of hashing a string value, but that's just pushing the collision problem
> off onto every project that uses us.
>
> So that's the problem as I understand it, but where does that leave us
> for solutions?  First, there's
> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
> which allows consumers to delete lock files when they're done with them.
>  Of course, in that case the onus is on the caller to make sure the lock
> couldn't possibly be in use anymore.
>
> Second, is this actually a problem?  Modern filesystems have absurdly
> large limits on the number of files in a directory, so it's highly
> unlikely we would ever exhaust that, and we're creating all zero byte
> files so there shouldn't be a significant space impact either.  In the
> past I believe our recommendation has been to simply create a cleanup
> job that runs on boot, before any of the OpenStack services start, that
> deletes all of the lock files.  At that point you know it's safe to
> delete them, and it prevents your lock file directory from growing forever.

Not that high - ext3 (still the default for nova ephemeral
partitions!) has a limit of 64k in one directory.

That said, I don't disagree - my thinkis is that we should advise
putting such files on a tmpfs.

-Rob

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [ceph] Puppet Ceph CI

2015-11-30 Thread David Moreau Simard

Hey Adam,

A bit late here, sorry.
Ceph works fine with OpenStack Kilo but at the time we developed the
integration tests for puppet-ceph with Kilo, there were some issues
specific to our test implementation and we chose to settle with Juno
at the time.

On the topic of CI, I can no longer sponsor the third party CI
(through my former employer, iWeb) as I am with Red Hat now.
I see this as an opportunity to drop the custom system tests with
vagrant and instead improve the acceptance tests.

What do you think ?


David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]


On Mon, Nov 23, 2015 at 6:45 PM, Adam Lawson  wrote:
> I'm confused, what is the context here? We use Ceph with OpenStack Kilo
> without issue.
>
> On Nov 23, 2015 2:28 PM, "David Moreau Simard"  wrote:
>>
>> Last I remember, David Gurtner tried to use Kilo instead of Juno but
>> he bumped into some problems and we settled for Juno at the time [1].
>> At this point we should already be testing against both Liberty and
>> Infernalis, we're overdue for an upgrade in that regard.
>>
>> But, yes, +1 to split acceptance tests:
>> 1) Ceph
>> 2) Ceph + Openstack
>>
>> Actually learning what failed is indeed challenging sometimes, I don't
>> have enough experience with the acceptance testing to suggest anything
>> better.
>> We have the flexibility of creating different logfiles, maybe we can
>> find a way to split out the relevant bits into another file.
>>
>> [1]: https://review.openstack.org/#/c/153783/
>>
>> David Moreau Simard
>> Senior Software Engineer | Openstack RDO
>>
>> dmsimard = [irc, github, twitter]
>>
>>
>> On Mon, Nov 23, 2015 at 2:45 PM, Andrew Woodward  wrote:
>> > I think I have a good lead on the recent failures in openstack / swift /
>> > radosgw integration component that we have since disabled. It looks like
>> > there is a oslo.config version upgrade conflict in the Juno repo we
>> > where
>> > using for CentOS. I think moving to Kilo will help sort this out, but at
>> > the
>> > same time I think it would be prudent to separate the Ceph v.s.
>> > OpenStack
>> > integration into separate jobs so that we have a better idea of which is
>> > a
>> > problem. If there is census for this, I'd need some direction / help, as
>> > well as set them up as non-voting for now.
>> >
>> > Looking into this I also found that the only place that we do
>> > integration
>> > any of the cephx logic was in the same test so we will need to create a
>> > component for it in the ceph integration as well as use it in the
>> > OpenStack
>> > side.
>> >
>> > Lastly un-winding the integration failure seemed overly complex. Is
>> > there a
>> > way that we can correlate the test status inside the job at a high level
>> > besides the entire job passed / failed without breaking them into
>> > separate
>> > jobs?
>> > --
>> >
>> > --
>> >
>> > Andrew Woodward
>> >
>> > Mirantis
>> >
>> > Fuel Community Ambassador
>> >
>> > Ceph Community
>> >
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic] weekly subteam status report

2015-11-30 Thread Ruby Loo

Hi,


We are elated to present this week's subteam report for Ironic. As usual,
this is pulled directly from the Ironic whiteboard[0] and formatted.


Bugs (dtantsur)

===

(diff with Nov 23)

- Open: 174 (+3). 11 new, 60 in progress, 0 critical, 16 high and 11
incomplete

- Nova bugs with Ironic tag: 24. 0 new, 0 critical, 0 high

- Inspector bugs: 13 (-1). 0 new, 0 critical, 6 high

- bug number is slowly growing since 28.09.2015 (135 open bugs)

- bugs to be aware of before the release this week:

- https://bugs.launchpad.net/ironic/+bug/1507738 - gives headache to people
with CentOS 7, might be considered a regression (status: fix on review)

- https://bugs.launchpad.net/ironic/+bug/1512544 - "grenade jobs are
failing" does not sound promising (status: in progress, no patch attached)

- https://bugs.launchpad.net/ironic/+bug/1408067 - do we still experience
these?



Network isolation (Neutron/Ironic work) (jroll)

===

- nova spec has landed; 2/3 patches are updated and passing unit tests

- ironic reviews still ongoing...



Live upgrades (lucasagomes, lintan)

===

- Submit a patch to add test to enforce object version bump correctly

- https://review.openstack.org/#/c/249624/ MERGED



Boot interface refactor (jroll)

===

- done!

- let's remove this one for next meeting \o/



Parallel tasks with futurist (dtantsur)

===

- still refactoring manager.py a bit:
https://review.openstack.org/#/c/249938/ is WIP



Node filter API and claims endpoint (jroll, devananda)

==

- spec still in review - https://review.openstack.org/#/c/204641



Multiple compute hosts (jroll, devananda)

=

- dependant on node filter API


ironic-lib adoption (rloo)

==

- ironic-lib 0.4.0 has been released and updated in requirements, patch is
almost ready (needs reno) but holding off til after ironic release:
https://review.openstack.org/#/c/184443/



Nova Liaisons (jlvillal & mrda)

===

- No meetiing/updates



Testing/Quality (jlvillal/lekha/krtaylor)

=

- No meeting/updates



Inspector (dtansur)

===

- Fully switched to Reno for both inspector projects

- e.g
http://docs.openstack.org/releasenotes/python-ironic-inspector-client/unreleased.html

- Documentation is now using Sphinx, will appear on
docs.openstack.org/developer/ironic-inspector this week

- python-ironic-inspector-client to be released this week, ironic-inspector
does not seem to have enough changes

- ironic-inspector-specs addition is proposed to the TC



Bifrost (TheJulia)

==

- Gate job is working again, presently looking at refactoring non-voting
test job.



webclient (krotscheck / betherly)

=

- Panel currently being wrapped in plugin. Actions on the nodes are
working. Details page to do then start port



Drivers:



DRAC (ifarkas/lucas)



- patches refactoring the driver to use python-dracclient instead of
pywsman are on gerrit


iLO (wanyen)



- 3rd party CI:
http://lists.openstack.org/pipermail/openstack-dev/2015-November/080806.html


iRMC (naohirot)

---

https://review.openstack.org//#/q/owner:+naohirot+status:+open,n,z

- Status: Reactive (solicited for core team's review)

- iRMC out of band inspection (bp/ironic-node-properties-discovery)

- Status: Active

- Enhance Power Interface for Soft Reboot and NMI
(bp/enhance-power-interface-for-soft-reboot-and-nmi)

- Add 'abort' support for Soft Power Off and Inject NMI
(bp/task-control-functions-for-long-running-tasks)

- iRMC OOB rescue mode support (bp/irmc-oob-rescue-mode-support)

- Status: Done, thanks!

- New boot driver interface for iRMC drivers (bp/new-boot-interface)


.


Until next week,

--ruby


[0] https://etherpad.openstack.org/p/IronicWhiteBoard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Clint Byrum

Excerpts from Joshua Harlow's message of 2015-11-30 10:42:53 -0800:
> Hi all,
> 
> I just wanted to bring up an issue, possible solution and get feedback 
> on it from folks because it seems to be an on-going problem that shows 
> up not when an application is initially deployed but as on-going 
> operation and running of that application proceeds (ie after running for 
> a period of time).
> 
> The jist of the problem is the following:
> 
> A <> has a need to ensure that no 
> application on the same machine can manipulate a given resource on that 
> same machine, so it uses the lock file pattern (acquire a *local* lock 
> file for that resource, manipulate that resource, release that lock 
> file) to do actions on that resource in a safe manner (note this does 
> not ensure safety outside of that machine, lock files are *not* 
> distributed locks).
> 
> The api that we expose from oslo is typically accessed via the following:
> 
>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None, 
> external=False, lock_path=None, semaphores=None, delay=0.01)
> 
> or via its underlying library (that I extracted from oslo.concurrency 
> and have improved to add more usefulness) @ 
> http://fasteners.readthedocs.org/
> 
> The issue though for <> is that each of 
> these projects now typically has a large amount of lock files that exist 
> or have existed and no easy way to determine when those lock files can 
> be deleted (afaik no? periodic task exists in said projects to clean up 
> lock files, or to delete them when they are no longer in use...) so what 
> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387 
> appear and there is no a simple solution to clean lock files up (since 
> oslo.concurrency is really not the right layer to know when a lock can 
> or can not be deleted, only the application knows that...)
> 
> So then we get a few creative solutions like the following:
> 
> - https://review.openstack.org/#/c/241663/
> - https://review.openstack.org/#/c/239678/
> - (and others?)
> 
> So I wanted to ask the question, how are people involved in < favorite openstack project>> cleaning up these files (are they at all?)
> 
> Another idea that I have been proposing also is to use offset locks.
> 
> This would allow for not creating X lock files, but create a *single* 
> lock file per project and use offsets into it as the way to lock. For 
> example nova could/would create a 1MB (or larger/smaller) *empty* file 
> for locks, that would allow for 1,048,576 locks to be used at the same 
> time, which honestly should be way more than enough, and then there 
> would not need to be any lock cleanup at all... Is there any reason this 
> wasn't initially done back way when this lock file code was created? 
> (https://github.com/harlowja/fasteners/pull/10 adds this functionality 
> to the underlying library if people want to look it over)

This is really complicated, and basically just makes the directory of
lock files _look_ clean. But it still leaves each offset stale, and has
to be cleaned anyway.

Fasteners already has process locks that use fcntl/flock.

These locks provide enough to allow you to infer things about.  the owner
of the lock file. If there's no process still holding the exclusive lock
when you try to lock it, then YOU own it, and thus control the resource.

A cron job which tries to flock anything older than ${REASONABLE_TIME}
and deletes them seems fine. Whatever process was trying to interact
with the resource is gone at that point.

Now, anything that needs to safely manage a resource beyond without a
live process will need to keep track of its own state and be idempotent
anyway. IMO this isn't something lock files alone solve well. I believe
you're familiar with a library named taskflow that is supposed to help
write code that does this better ;). Even without taskflow, if you are
trying to do something exclusive without a single process that stays
alive, you need to do _something_ to keep track of state and restart
or revert that flow. That is a state management problem, not a locking
problem.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

On 11/30/2015 03:01 PM, Robert Collins wrote:
> On 1 December 2015 at 08:37, Ben Nemec  wrote:
>> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
>>> Hi all,
>>>
>>> I just wanted to bring up an issue, possible solution and get feedback
>>> on it from folks because it seems to be an on-going problem that shows
>>> up not when an application is initially deployed but as on-going
>>> operation and running of that application proceeds (ie after running for
>>> a period of time).
>>>
>>> The jist of the problem is the following:
>>>
>>> A <> has a need to ensure that no
>>> application on the same machine can manipulate a given resource on that
>>> same machine, so it uses the lock file pattern (acquire a *local* lock
>>> file for that resource, manipulate that resource, release that lock
>>> file) to do actions on that resource in a safe manner (note this does
>>> not ensure safety outside of that machine, lock files are *not*
>>> distributed locks).
>>>
>>> The api that we expose from oslo is typically accessed via the following:
>>>
>>>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
>>> external=False, lock_path=None, semaphores=None, delay=0.01)
>>>
>>> or via its underlying library (that I extracted from oslo.concurrency
>>> and have improved to add more usefulness) @
>>> http://fasteners.readthedocs.org/
>>>
>>> The issue though for <> is that each of
>>> these projects now typically has a large amount of lock files that exist
>>> or have existed and no easy way to determine when those lock files can
>>> be deleted (afaik no? periodic task exists in said projects to clean up
>>> lock files, or to delete them when they are no longer in use...) so what
>>> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
>>> appear and there is no a simple solution to clean lock files up (since
>>> oslo.concurrency is really not the right layer to know when a lock can
>>> or can not be deleted, only the application knows that...)
>>>
>>> So then we get a few creative solutions like the following:
>>>
>>> - https://review.openstack.org/#/c/241663/
>>> - https://review.openstack.org/#/c/239678/
>>> - (and others?)
>>>
>>> So I wanted to ask the question, how are people involved in <>> favorite openstack project>> cleaning up these files (are they at all?)
>>>
>>> Another idea that I have been proposing also is to use offset locks.
>>>
>>> This would allow for not creating X lock files, but create a *single*
>>> lock file per project and use offsets into it as the way to lock. For
>>> example nova could/would create a 1MB (or larger/smaller) *empty* file
>>> for locks, that would allow for 1,048,576 locks to be used at the same
>>> time, which honestly should be way more than enough, and then there
>>> would not need to be any lock cleanup at all... Is there any reason this
>>> wasn't initially done back way when this lock file code was created?
>>> (https://github.com/harlowja/fasteners/pull/10 adds this functionality
>>> to the underlying library if people want to look it over)
>>
>> I think the main reason was that even with a million locks available,
>> you'd have to find a way to hash the lock names to offsets in the file,
>> and a million isn't a very large collision space for that.  Having two
>> differently named locks that hashed to the same offset would lead to
>> incredibly confusing bugs.
>>
>> We could switch to requiring the projects to provide the offsets instead
>> of hashing a string value, but that's just pushing the collision problem
>> off onto every project that uses us.
>>
>> So that's the problem as I understand it, but where does that leave us
>> for solutions?  First, there's
>> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
>> which allows consumers to delete lock files when they're done with them.
>>  Of course, in that case the onus is on the caller to make sure the lock
>> couldn't possibly be in use anymore.
>>
>> Second, is this actually a problem?  Modern filesystems have absurdly
>> large limits on the number of files in a directory, so it's highly
>> unlikely we would ever exhaust that, and we're creating all zero byte
>> files so there shouldn't be a significant space impact either.  In the
>> past I believe our recommendation has been to simply create a cleanup
>> job that runs on boot, before any of the OpenStack services start, that
>> deletes all of the lock files.  At that point you know it's safe to
>> delete them, and it prevents your lock file directory from growing forever.
> 
> Not that high - ext3 (still the default for nova ephemeral
> partitions!) has a limit of 64k in one directory.
> 
> That said, I don't disagree - my thinkis is that we should advise
> putting such files on a tmpfs.

So, I think the issue really is that the named external locks were
originally thought to be handling some pretty sensitive critical
sections. Both cinder / nova have less than 20 such named locks.

Cinder uses

Re: [openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node

2015-11-30 Thread Fox, Kevin M

Is that because of trying to shoehorn docker containers into rpm's though? I've 
never seen anyone else try and use them that way. Maybe they belong in a docker 
repo like the hub or something openstack.org hosted instead?

Thanks,
Kevin

From: Igor Kalnitsky [ikalnit...@mirantis.com]
Sent: Friday, November 27, 2015 1:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Fuel] Getting rid of Docker containers on the 
Fuel master node

Hey Vladimir,

Thanks for your effort on doing this job. Unfortunately we have not so
much time left and FF is coming, so I'm afraid it's become unreal to
make it before FF. Especially if it takes 2-3 days to fix system
tests.


Andrew,

I had the same opinion some time ago, but it was changed because
nobody puts effort to fix our Docker experience. Moreover Docker is
still buggy and we have a plenty of issues such as stale mount points
for instance. Besides, I don't like our upgrade procedure -

1. Install fuel-docker-images.rpm
2. Load images from installed tarball to Docker
3. Re-create containers from new images

Where (2) and (3) are manual steps and breaks idea of "yum update"
delivery approach.

Thanks,
Igor

On Wed, Nov 25, 2015 at 9:43 PM, Andrew Woodward  wrote:
> 
> IMO, removing the docker containers is a mistake v.s. fixing them and using
> them properly. They provide an isolation that is necessary (and that we
> mangle) to make services portable and scaleable. We really should sit down
> and document how we really want all of the services to interact before we
> rip the containers out.
>
> I agree, the way we use containers now still is quite wrong, and brings us
> some negative value, but I'm not sold on stripping them out now just because
> they no longer bring the same upgrades value as before.
> 
>
> My opinion aside, we are rushing into this far to late in the feature cycle.
> Prior to moving forward with this, we need a good QA plan, the spec is quite
> light on that and must receive review and approval from QA. This needs to
> include an actual testing plan.
>
> From the implementation side, we are pushing up against the FF deadline. We
> need to document what our time objectives are for this and when we will no
> longer consider this for 8.0.
>
> Lastly, for those that are +1 on the thread here, please review and comment
> on the spec, It's received almost no attention for something with such a
> large impact.
>
> On Tue, Nov 24, 2015 at 4:58 PM Vladimir Kozhukalov
>  wrote:
>>
>> The status is as follows:
>>
>> 1) Fuel-main [1] and fuel-library [2] patches can deploy the master node
>> w/o docker containers
>> 2) I've not built experimental ISO yet (have been testing and debugging
>> manually)
>> 3) There are still some flaws (need better formatting, etc.)
>> 4) Plan for tomorrow is to build experimental ISO and to begin fixing
>> system tests and fix the spec.
>>
>> [1] https://review.openstack.org/#/c/248649
>> [2] https://review.openstack.org/#/c/248650
>>
>> Vladimir Kozhukalov
>>
>> On Mon, Nov 23, 2015 at 7:51 PM, Vladimir Kozhukalov
>>  wrote:
>>>
>>> Colleagues,
>>>
>>> I've started working on the change. Here are two patches (fuel-main [1]
>>> and fuel-library [2]). They are not ready to review (still does not work and
>>> under active development). Changes are not going to be huge. Here is a spec
>>> [3]. Will keep the status up to date in this ML thread.
>>>
>>>
>>> [1] https://review.openstack.org/#/c/248649
>>> [2] https://review.openstack.org/#/c/248650
>>> [3] https://review.openstack.org/#/c/248814
>>>
>>>
>>> Vladimir Kozhukalov
>>>
>>> On Mon, Nov 23, 2015 at 3:35 PM, Aleksandr Maretskiy
>>>  wrote:



 On Mon, Nov 23, 2015 at 2:27 PM, Bogdan Dobrelya
  wrote:
>
> On 23.11.2015 12:47, Aleksandr Maretskiy wrote:
> > Hi all,
> >
> > as you know, Rally runs inside docker on Fuel master node, so docker
> > removal (good improvement) is a problem for Rally users.
> >
> > To solve this, I'm planning to make native Rally installation on Fuel
> > master node that is running on CentOS 7,
> > and then make a step-by-step instruction how to make this
> > installation.
> >
> > So I hope docker removal will not make issues for Rally users.
>
> I believe the most backwards compatible scenario is to keep the docker
> installed while removing the fuel-* docker things back to the host OS.
> So nothing would prevent user from pulling and running whichever docker
> containers he wants to put on the Fuel master node. Makes sense?
>

 Sounds good


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

>>>
>>
>> _

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)


Joshua Harlow wrote:

Ben Nemec wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<> has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the
following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<> is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<> cleaning up these files (are they at all?)


From some simple greps using:

$ echo "Removal usage in" $(basename `pwd`); grep -R 
remove_external_lock_file *


Removal usage in cinder


Removal usage in nova
nova/virt/libvirt/imagecache.py: 
lockutils.remove_external_lock_file(lock_file,


Removal usage in glance


Removal usage in neutron


So me thinks people aren't cleaning any of these up :-/



Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)


I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that. Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions? First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151

which allows consumers to delete lock files when they're done with them.
Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.


Ya, I wonder how many folks are actually doing this, because the exposed
API of @synchronized doesn't seem to tell u what file to even delete in
the first place :-/ perhaps we should make that more accessible so that
people/consumers of that code could know what to delete...



Second, is this actually a problem? Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either. In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files. At that point you know it's safe to
delete them, and it prevents your lock file directory from growing
forever.


Except as we move to never shutting an app down (always online and live
upgrades and all that jazz), it will have to run more than just on boot,
but point taken.



I know

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)


Clint Byrum wrote:

Excerpts from Joshua Harlow's message of 2015-11-30 10:42:53 -0800:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)


This is really complicated, and basically just makes the directory of
lock files _look_ clean. But it still leaves each offset stale, and has
to be cleaned anyway.


What do u mean here (out of curiosity), each offset stale? The file 
would basically never change size after startup (pick a large enough 
number, 10 million, 1 trillion billion...) and use it appropriately from 
there on out...




Fasteners already has process locks that use fcntl/flock.

These locks provide enough to allow you to infer things about.  the owner
of the lock file. If there's no process still holding the exclusive lock
when you try to lock it, then YOU own it, and thus control the resource.


Well not really, python doesn't expose the ability to introspect who has 
the handle afaik, I  tried to look into that and it looks like fnctl 
(the C api) might have a way to get it, but u can't really introspect 
that, without as u stated, acquiring the lock yourself... I can try to 
recall more of this investigation when I was trying to add a @owner_pid 
property onto fasteners interprocess lock class but from my simple 
memory the exposed API isn't there in python.




A cron job which tries to flock anything older than ${REASONABLE_TIME}
and deletes them seems fine. Whatever process was trying to interact
with the resource is gone at that point.


Yes, or a periodic thread in the application that can do this in a safe 
manner (using its ability to know exactly what its own apps internals 
are doing...)




Now, anything that needs to safely manage a resource beyond without a
live process will need to keep track of its own state and be idempotent
anyway. IMO this isn't something lock files alone solve well. I believe
you're familiar with a library named taskflow that is supposed to help
write code that does this better ;). Even without taskflow, if you are
trying to do something exclusive without a single process that stays
alive, you need to do _something_ to keep track of state and restart
or revert that flow. That is a state management problem, not a locking
problem.



Agreed. ;)


__
OpenStack Development Mailing List (not for usage questions)
Unsubs

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)