date:20131119

Sweet!

Feel free to ask lots of questions and jump on both irc channels 
(#openstack-state-management and #openstack-cinder) if u need any help that can 
be better solved in real time chat.

Thanks for helping getting this ball rolling :-)

Sent from my really tiny device...

> On Nov 19, 2013, at 8:46 AM, "Walter A. Boring IV"  
> wrote:
> 
> Awesome guys,
>  Thanks for picking this up.   I'm looking forward to the reviews :)
> 
> Walt
>>> On 19.11.2013 10:38, Kekane, Abhishek wrote:
>>> Hi All,
>>> 
>>> Greetings!!!
>> Hi there!
>> 
>> And thanks for your interest in cinder and taskflow!
>> 
>>> We are in process of implementing the TaskFlow 0.1 in Cinder for "copy
>>> volume to image" and "delete volume".
>>> 
>>> I have added two blueprints for the same.
>>> 1. 
>>> https://blueprints.launchpad.net/cinder/+spec/copy-volume-to-image-task-flow
>>> 2. https://blueprints.launchpad.net/cinder/+spec/delete-volume-task-flow
>>> 
>>> I would like to know if any other developers/teams are working or
>>> planning to work on any cinder api apart from above two api's.
>>> 
>>> Your help is appreciated.
>> Anastasia Karpinska works on updating existing flows to use released
>> TaskFlow 0.1.1 instead of internal copy:
>> 
>> https://review.openstack.org/53922
>> 
>> It was blocked because taskflow was not in openstack/requirements, but
>> now we're there, and Anastasia promised to finish the work and submit
>> updated changeset for review in couple of days.
>> 
>> There are also two changesets that convert cinder APIs to use TaskFlow:
>> - https://review.openstack.org/53480 for create_backup by Victor
>>   Rodionov
>> - https://review.openstack.org/55134 for create_snapshot by Stanislav
>>   Kudriashev
>> 
>> As far as I know both Stanislav and Victor suspended their work unitil
>> Anastasia's change lands.
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Search Project - summit follow up

2013-11-19 Thread Dmitri Zimin(e) | StackStorm

Thanks Jarda for the intro;

Hi Stackers,

The project Search is a service providing fast full-text search for
resources across OpenStack services.

The idea was introduced and discussed at the Summit. We confirmed the
need for Search service: 1) from core Horizon team and UX 2) from
integrators building custom consoles 3) some suggested CLI use-cases.
We established it’s likely be a new service. I spoke with Julien @
Ceilometer about leveraging some overlapping functionality and he
suggested an integration point to address it.

Now let’s discuss it with the broader community and have you guys
guide it through the proper next steps.

DETAILS:
* Wiki - basic project description, and 2 min video showing Search in action.
  https://wiki.openstack.org/wiki/SearchProject
* Etherpad - captures Summit discussions:
https://etherpad.openstack.org/p/search
* Github - the rough prototype:
   * Search backend: https://github.com/StackStorm/search
   * Horizon plugin: https://github.com/StackStorm/search-horizon

QUESTIONS:

1) Is it relevant to OpenStack vision and directions?
--> yes, based on Summit discussions

2) What is a good home for it? A a new standalone program or a part of
existing program?
--> So far: Standalone program (input from David Lyle, Matthias Runge,
Doug Helmann)

3) Who else wants to be involved? Pigs? Chickens?

4) What do we do next?
--> Create an Launchpad?
--> Form the team?
--> Give it a name?
--> Begin flashing out the design?

Thanks to David Lyle, Mattihas Runge, Nicolas Simonds, Julien Danjou,
Noy Itzikowitz, Ryan Lane and Doug Helmann for your help and active
input.

Cheers, Dmitri.

On Tue, Nov 19, 2013 at 7:56 AM, Jaromir Coufal  wrote:
> Hey OpenStackers,
>
> I hope you all enjoyed Hong Kong Summit and for those who couldn't attend, I
> hope we will see each other next time!
>
> For all of you, I'd like to follow up on conversation from Summit about
> Search project [0], which Dmitri introduced and I believe it has great
> potential for enhancing OpenStack's user experience, especially in Horizon,
> but not just in there.
>
> I'd like to thank to Dmitri and all folks who were working on this project
> for sharing and their will for contributing.
>
> It would be great, if we can follow up on this topic, start discussion
> around the project here on mailing list (since not everybody could attend
> the summit) and gather some feedback. There are lot of questions like where
> to accommodate the code, what are next steps, etc.
>
> I am cc'ing Dmitri, so he can bring more light into this discussion and I
> hope we can move forward.
>
> Thanks
> -- Jarda
>
> [0] https://wiki.openstack.org/wiki/SearchProject

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Edgar Magana

Isaku,

Do you have in mind any implementation, any BP?
We could actually work on this together, all plugins will get the benefits
of a better implementation.

Thanks,

Edgar

On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:

>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>Robert Kukura  wrote:
>
>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>> > Developers,
>> > 
>> > This topic has been discussed before but I do not remember if we have
>>a
>> > good solution or not.
>> 
>> The ML2 plugin addresses this by calling each MechanismDriver twice. The
>> create_network_precommit() method is called as part of the DB
>> transaction, and the create_network_postcommit() method is called after
>> the transaction has been committed. Interactions with devices or
>> controllers are done in the postcommit methods. If the postcommit method
>> raises an exception, the plugin deletes that partially-created resource
>> and returns the exception to the client. You might consider a similar
>> approach in your plugin.
>
>Splitting works into two phase, pre/post, is good approach.
>But there still remains race window.
>Once the transaction is committed, the result is visible to outside.
>So the concurrent request to same resource will be racy.
>There is a window after pre_xxx_yyy before post_xxx_yyy() where
>other requests can be handled.
>
>The state machine needs to be enhanced, I think. (plugins need
>modification)
>For example, adding more states like pending_{create, delete, update}.
>Also we would like to consider serializing between operation of ports
>and subnets. or between operation of subnets and network depending on
>performance requirement.
>(Or carefully audit complex status change. i.e.
>changing port during subnet/network update/deletion.)
>
>I think it would be useful to establish reference locking policy
>for ML2 plugin for SDN controllers.
>Thoughts or comments? If this is considered useful and acceptable,
>I'm willing to help.
>
>thanks,
>Isaku Yamahata
>
>> -Bob
>> 
>> > Basically, if concurrent API calls are sent to Neutron, all of them
>>are
>> > sent to the plug-in level where two actions have to be made:
>> > 
>> > 1. DB transaction ? No just for data persistence but also to collect
>>the
>> > information needed for the next action
>> > 2. Plug-in back-end implementation ? In our case is a call to the
>>python
>> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>> > 
>> > For instance:
>> > 
>> > def create_port(self, context, port):
>> > with context.session.begin(subtransactions=True):
>> > # Plugin DB - Port Create and Return port
>> > port_db = super(NeutronPluginPLUMgridV2,
>> > self).create_port(context,
>> >   
>> port)
>> > device_id = port_db["device_id"]
>> > if port_db["device_owner"] == "network:router_gateway":
>> > router_db = self._get_router(context, device_id)
>> > else:
>> > router_db = None
>> > try:
>> > LOG.debug(_("PLUMgrid Library: create_port() called"))
>> > # Back-end implementation
>> > self._plumlib.create_port(port_db, router_db)
>> > except Exception:
>> > 
>> > 
>> > The way we have implemented at the plugin-level in Havana (even in
>> > Grizzly) is that both action are wrapped in the same "transaction"
>>which
>> > automatically rolls back any operation done to its original state
>> > protecting mostly the DB of having any inconsistency state or left
>>over
>> > data if the back-end part fails.=.
>> > The problem that we are experiencing is when concurrent calls to the
>> > same API are sent, the number of operation at the plug-in back-end are
>> > long enough to make the next concurrent API call to get stuck at the
>>DB
>> > transaction level, which creates a hung state for the Neutron Server
>>to
>> > the point that all concurrent API calls will fail.
>> > 
>> > This can be fixed if we include some "locking" system such as calling:
>> > 
>> > from neutron.common import utile
>> > 
>> > 
>> > @utils.synchronized('any-name', external=True)
>> > def create_port(self, context, port):
>> > 
>> > 
>> > Obviously, this will create a serialization of all concurrent calls
>> > which will ends up in having a really bad performance. Does anyone
>>has a
>> > better solution?
>> > 
>> > Thanks,
>> > 
>> > Edgar
>> > 
>> > 
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > 
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>-- 
>Isaku Yamahata 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listin

Re: [openstack-dev] [nova][heat][[keystone] RFC: introducing "request identification"

2013-11-19 Thread Adam Young

On 11/19/2013 08:21 AM, Dolph Mathews wrote:

Related BP:

  Create a unified request identifier
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id


And we have discussed  workplans as well, which would be data to be 
carried along with the request id


https://blueprints.launchpad.net/keystone/+spec/delegation-workplans
https://etherpad.openstack.org/keystone_workplans



On Tue, Nov 19, 2013 at 5:04 AM, haruka tanizawa > wrote:


Hi stackers!!

I'd like to ask for your opinions about my idea of identifying
request.

Challenges
==

We have no way to know the final result of an API request.
Indeed we can continuously get the status of allocated resources,
but this is just resource status, not request status.

It doesn't matter so much for manual operations.
But it does for automated clients like heat.
We need request-oriented-status and it should be disclosed to clients.

Literally, we need to address two items for it.
 1. how to identify request from clients
 2. how to disclose status of request to clients

Note that this email includes only 1 for initiating the discussion.
Also, bp:instance-tasks-api[0] should include both two items above.

Proposal: introducing "request identification"
=

I'd like to introduce "request identification", which is disclosed
to clients.
There are two characteristics:

 - "request identification" is unique ID for each request
   so that we can identify tell a specific request from others.
 - "request identification" is available for clients
   so that we can enable any after-request-operations like check,
retry[1] or cancel[2].
   (of course we need to add more logic for check/retry/cancel,
but I'm pretty sure that indentifying request is the starting
point for these features)

In my understandings, main objections should be 'who should
generate and maintain such IDs?'.

How to implement "request identification"
=

There are several options at least. (See also recent discussion at
openstack-dev[3])

1. Enable user to provide his/her own "request identification"
within API request.
   This should be the simplest way. But providing id from outside
looks hard to be accepted.
   For example, Idempotency for OpenStack API[4].
   Or instance-tasks-api enable to user to put his/her own
"request identification".

2. Use correlation_id in oslo as "request identification".
   correlation_id looks similar concept of "request indentification",
   but correlation_id in nova was already rejected[5].

3. Enable keystone to generate "request identification" (we can
call it 'request-token', for example).
   Before sending actual API request to nova-api, client sends a
request to keystone to get 'request-token'.


-2

   Then client sends an actual API request with 'request-token'.
   Nova-api will consult it to keystone whether it was really
generated.
   Sounds like a auth-token generated by keystone, differences are:
 [lifecycle] auth-token is used for multiple API requests
before it expires,
'request-token' is used for only single API request.
 [reusing] if the same 'request-token' was specified twice or
more times,
nova-api simply returns 20x (works like client token in
AWS[6]).
Keystone needs to maintain 'request-tokens' until they expire.
   For backward compatibility, actual API request without
'request-token' should work as before.
   We can consider several options for uniqueness of 'request-token':
 uuid, any string with uniqueness per tennant, etc.

IMO, since adding new implementation to Keystone is a little bit
hard work,
so implement of 1 is reasonable for me, just idea.

Any comments will be appreciated.

Sincerely, Haruka Tanizawa

[0] https://blueprints.launchpad.net/nova/+spec/instance-tasks-api
[1] https://wiki.openstack.org/wiki/Support-retry-with-idempotency
[2] https://blueprints.launchpad.net/nova/+spec/cancel-swap-volume
[3]
http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg09023.html
[4]
https://blueprints.launchpad.net/nova/+spec/idempotentcy-client-token
[5] https://review.openstack.org/#/c/29480/
[6]

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--

-Dolph


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openst

Re: [openstack-dev] [Ironic][Ceilometer] get IPMI data for ceilometer

2013-11-19 Thread Doug Hellmann

On Mon, Nov 18, 2013 at 8:58 PM, Lu, Lianhao  wrote:

>
> Doug Hellmann wrote on 2013-11-19:
> >
> > On Mon, Nov 18, 2013 at 12:25 PM, Devananda van der Veen <
> devananda@gmail.com> wrote:
> >
> >
> >   Hi Lianhao Lu,
> >
> >   I briefly summarized my recollection of that session in this
> blueprint:
> >
> >   https://blueprints.launchpad.net/ironic/+spec/add-ceilometer-agent
> >
> >
> >   I've responded to your questions inline as well.
> >
> >
> >   On Sun, Nov 17, 2013 at 10:24 PM, Lu, Lianhao <
> lianhao...@intel.com> wrote:
> >
> >
> >   Hi stackers,
> >
> >   During the summit session Expose hardware sensor (IPMI)
> data
> >
> https://etherpad.openstack.org/p/icehouse-summit-ceilometer-hardware-sensors,
> it was proposed to deploy a ceilometer agent next to
> > the ironic conductor to the get the ipmi data. Here I'd like to ask some
> questions to figure out what's the current missing pieces in ironic
> > and ceilometer for that proposal.
> >
> >   1. Just double check, ironic won't provide API to get IPMI
> data, right?
> >
> >
> >
> >   Correct. This was generally felt to be unnecessary.
> >
> >
> >   2. If deploying a ceilometer agent next to the ironic
> conductor, how does the agent talk to the conductor? Through rpc?
> >
> >
> >
> >   My understanding is that ironic-conductor will emit messages to
> the ceilimeter agent, and the communication is one-way. These could
> > be triggered by a periodic task, or by some other event within Ironic,
> such as a change in the power state of a node.
> >
> >
> > Cool, so this eliminates the need for a separate agent. The ceilometer
> work can be done in the collector, to receive the new messages.
> >
> Does this means we lose the ability to specify the different polling
> interval for different monitoring data, like we have in ceilometer pipeline?
>

It means that would need to be implemented in the ironic code that is doing
the polling. 

Doug


>
> >
> >   3. Does the current ironic conductor have rpc_method to
> support getting generic ipmi data, i.e. let the rpc_method caller
> > specifying arbitrary netfn/command to get any type of ipmi data?
> >
> >
> >
> >   No, and as far as I understand, it doesn't need one.
> >
> >
> > It would only need that if we were going to poll for the data, but if
> ironic is emitting notifications then we don't have to do that.
> >
> >   4. I believe the ironic conductor uses some kind of
> node_id to associate the bmc with its credentials, right? If so, how can the
> > ceilometer agent get those node_ids to ask the ironic conductor to poll
> the ipmi data? And how can the ceilometer agent extract
> > meaningful information from that node_id to set those fields in the
> ceilometer Sample(e.g. recource_id, project_id, user_id, etc.) to identify
> > which physical node the ipmi data is coming from?
> >
> >   This question perhaps requires a longer answer.
> >
> >   Ironic references physical machines (nodes) internally with an
> integer node_id and externally with a standard uuid. When a Nova
> > instance is created, it will be associated to a node, that node will
> have a reference to the nova instance_uuid which is exposed in our API,
> > and can be passed to Ceilometer's agent. I believe that nova
> instance_uuid will enable ceilometer to detect the project, user, etc.
> >
> >
> > If ironic has those values (at least the project), and can include them
> in the notification payload, that will make processing the incoming
> > notifications significantly less expensive.
> >
> >
> >   Should Ironic emit messages regarding nodes which are not
> provisioned? Physical machines that don't have a tenant instance on them
> > are not associated to any project, user, tenant, quota, etc, so I
> suspect that we shouldn't notify about them. It would be like tracking the
> > unused disks in a SAN.
> >
> >
> > I don't think it would be useful, but if someone else does then it seems
> OK to include them.
>
> I think it'd better for Ironic to emit those data in case some users want
> to collect them, at least Ironic should have a configuration setting to
> emit those kind of data.
>
> -Lianhao
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-security] Neutron security groups for L2 networks in Havana

2013-11-19 Thread Aaron Rosen

Hi Kanthi,

The issue is that the nova-api says that by default every instance needs to
be in a default security group that blocks all ingress traffic from outside
and allows all ingress from instances that belong to the same security
group. If an instance does not have an ip address we are unable to enforce
the instance->instance communication of members that are part of the same
security group (or at least the iptables implementation in place doesn't).

There is an extension in neutron that implements port_security that allows
one to disable this so that one can do L2 networking as you want to. Though
unfortunately, this does not work correctly at this time with nova as
currently it's calling into neutron to create the security group and attach
it to the instance anways. See: https://bugs.launchpad.net/nova/+bug/1175464 .
I'm hoping to resolve this issue in the next few weeks.

Best,

Aaron


On Fri, Nov 8, 2013 at 1:45 AM, Kanthi P  wrote:

> Hi Aaron,
>
> Thanks for the reply !
> Yes security groups are mapped to L3/L4 headers, these security rules are
> being converted to iptables.
>
> If we remove the subnet check, we will be able to launch instances without
> ipaddress, they just have the mac address.
> Users can configure their own ipaddress after they are booted.
>
> If we use neutron security groups, it provides a default group (on port
> basis) which blocks all ingress and allows all egress traffic.
>
> Later users can configure security groups based on the ip address what
> they provided to the vnics.
>
> I mean to say, ports will have subnet but just that this subnet is not
> known to openstack during the instance boot time.
>
>
>
>
> On Fri, Nov 8, 2013 at 9:42 AM, Aaron Rosen  wrote:
>
>>
>>
>>
>> On Thu, Nov 7, 2013 at 12:23 PM, Kanthi P wrote:
>>
>>> Hi,
>>>
>>> I am trying to boot a VM which has a network without subnet in Havana,
>>> but it throws an exception saying that subnet is mandatory if quantum
>>> security groups are enabled.
>>>
>>> Here are the commands I used.
>>>
>>> neutron net-create testNet
>>> neutron net-show testNet
>>> +---+--+
>>> | Field | Value|
>>> +---+--+
>>> | admin_state_up| True |
>>> | id| 47208beb-2801-4729-bc7b-6532717232fc |
>>> | name  | testNet  |
>>> | provider:network_type | local|
>>> | provider:physical_network |  |
>>> | provider:segmentation_id  |  |
>>> | router:external   | False|
>>> | shared| False|
>>> | status| ACTIVE   |
>>> | subnets   |  |
>>> | tenant_id | b5b591dcda2645cd9d15a4fe3eb1b50d |
>>> +---+--+
>>>
>>> nova boot --flavor 2 --image 30c1984c-8a4f-4e3f-8c9a-7de0b321caa2 --nic
>>> net-id=47208beb-2801-4729-bc7b-6532717232fc testServer1
>>>
>>> Here is the piece of code which generated this exception
>>>
>>> nova/network/neutronv2/api.py
>>>
>>> if (security_groups and not (
>>> network['subnets']
>>> and network.get('port_security_enabled', True))):
>>>
>>> raise exception.SecurityGroupCannotBeApplied()
>>>
>>>
>>> Can someone please explain why do we need this check?
>>>
>>
>> Hi Kanthi,
>>
>> We need this check because because in order to enforce security groups
>> the port needs to have an ip_address (i.e: network needs a subnet) since
>> Security groups only map to L3/4 headers. Today, nova automatically adds a
>> default security group to all instances when booted. Hopefully we can punt
>> this task off to neutron in this release by moving the port-creation up on
>> the stack to nova-api instead of nova-compute though this isn't the case
>> right now.
>>
>> Aaron
>>
>>>
>>> To my understanding subnet is used for two purposes in terms of security
>>> groups
>>> 1. To allow dhcp traffic if dhcp is enabled on the subnet, example given
>>> below
>>> -A quantum-openvswi-i7bf776d2-b -s 192.168.1.3/32 -p udp -m udp
>>> --sport 67 --dport 68 -j RETURN
>>> 2. To avoid ip spoof
>>> -A quantum-openvswi-o7bf776d2-b ! -s 192.168.1.2/32 -j DROP
>>>
>>> Can we remove this so that we can have guests which has nic with just
>>> MAC address, guest can configure its own ipaddress. Later if needed they
>>> can enable their own security rules via quantum api.
>>>
>>> ___
>>> Openstack-security mailing list
>>> openstack-secur...@lists.openstack.org
>>> http://lists.op

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Jay Pipes

On 11/19/2013 11:33 AM, Paula Ta-Shma wrote:



Hi Paula,

Where can we see the source code for the prototype and any API specs
that you may have?

Best,
-jay


Hi Jay,

Thanks for your interest. Our prototype intercepts Swift REST requests
using proxy server middleware and uses Solr as the indexing/search back
end.


OK, sounds interesting.


Our search API is similar to the one proposed by HP.


My apologies, I'm apparently coming into this quite late :) Would you 
mind sharing a link to the HP proposal? I wasn't at the summit 
unfortunately and am playing a bit of catch up.


> Rather than propose

yet another API (we already have the HP proposal and the SoftLayer API out
there), we would be glad to participate in the discussion on search API,
functionality, and which parts should be open source and which vendor
specific. We haven't put our code out in the open yet but may be able to
contribute to a reference implementation.


The FLOSSian in me encourages you and your team to share early and 
release often. Open code breeds innovation! :)



In general we would propose that the API be general and allow different
vendors to implement subsets, for example by publishing which 
are implemented as in the HP proposal. So the SoftLayer API would be one
example subset.


K, sounds good. If there's already a good API proposal, I totally agree 
with you on not creating another one and instead working to improve 
existing ones if necessary.



Some things to consider as optional additions to the API (could be included
as optional  as in the HP proposal):
- allowing to search for accounts/containers/objects having certain
attribute names, where the attribute names could possibly have wildcards
- allowing user defined types with their own sort orders to be
indexed/searched
- allowing multiple search criteria where each one has a different scope
i.e. account, container etc., for example, search for objects in containers
created in 2013 whose color is red


++

Best,
-jay


regards
Paula



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

2013-11-19 Thread yunhong jiang

On Tue, 2013-11-19 at 12:52 +, Daniel P. Berrange wrote:
> On Wed, Nov 13, 2013 at 02:46:06PM +0200, Tuomas Paappanen wrote:
> > Hi all,
> > 
> > I would like to hear your thoughts about core pinning in Openstack.
> > Currently nova(with qemu-kvm) supports usage of cpu set of PCPUs
> > what can be used by instances. I didn't find blueprint, but I think
> > this feature is for isolate cpus used by host from cpus used by
> > instances(VCPUs).
> > 
> > But, from performance point of view it is better to exclusively
> > dedicate PCPUs for VCPUs and emulator. In some cases you may want to
> > guarantee that only one instance(and its VCPUs) is using certain
> > PCPUs.  By using core pinning you can optimize instance performance
> > based on e.g. cache sharing, NUMA topology, interrupt handling, pci
> > pass through(SR-IOV) in multi socket hosts etc.
> > 
> > We have already implemented feature like this(PoC with limitations)
> > to Nova Grizzly version and would like to hear your opinion about
> > it.
> > 
> > The current implementation consists of three main parts:
> > - Definition of pcpu-vcpu maps for instances and instance spawning
> > - (optional) Compute resource and capability advertising including
> > free pcpus and NUMA topology.
> > - (optional) Scheduling based on free cpus and NUMA topology.
> > 
> > The implementation is quite simple:
> > 
> > (additional/optional parts)
> > Nova-computes are advertising free pcpus and NUMA topology in same
> > manner than host capabilities. Instances are scheduled based on this
> > information.
> > 
> > (core pinning)
> > admin can set PCPUs for VCPUs and for emulator process, or select
> > NUMA cell for instance vcpus, by adding key:value pairs to flavor's
> > extra specs.
> > 
> > EXAMPLE:
> > instance has 4 vcpus
> > :
> > vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
> > emulator:5 --> emulator pinned to pcpu5
> > or
> > numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.
> > 
> > In nova-compute, core pinning information is read from extra specs
> > and added to domain xml same way as cpu quota values(cputune).
> > 
> > 
> >   
> >   
> >   
> >   
> >   
> > 
> > 
> > What do you think? Implementation alternatives? Is this worth of
> > blueprint? All related comments are welcome!
> 
> I think there are several use cases mixed up in your descriptions
> here which should likely be considered independantly
> 
>  - pCPU/vCPU pinning
> 
>I don't really think this is a good idea as a general purpose
>feature in its own right. It tends to lead to fairly inefficient
>use of CPU resources when you consider that a large % of guests
>will be mostly idle most of the time. It has a fairly high
>administrative burden to maintain explicit pinning too. This
>feels like a data center virt use case rather than cloud use
>case really.
> 
>  - Dedicated CPU reservation
> 
>The ability of an end user to request that their VM (or their
>group of VMs) gets assigned a dedicated host CPU set to run on.
>This is obviously something that would have to be controlled
>at a flavour level, and in a commercial deployment would carry
>a hefty pricing premium.
> 
>I don't think you want to expose explicit pCPU/vCPU placement
>for this though. Just request the high level concept and allow
>the virt host to decide actual placement
> 
>  - Host NUMA placement.
> 
>By not taking NUMA into account currently the libvirt driver
>at least is badly wasting resources. Having too much cross-numa
>node memory access by guests just kills scalability. The virt
>driver should really automaticall figure out cpu & memory pinning
>within the scope of a NUMA node automatically. No admin config
>should be required for this.
> 
>  - Guest NUMA topology
> 
>If the flavour memory size / cpu count exceeds the size of a
>single NUMA node, then the flavour should likely have a way to
>express that the guest should see multiple NUMA nodes. The
>virt host would then set guest NUMA topology to match the way
>it places vCPUs & memory on host NUMA nodes. Again you don't
>want explicit pcpu/vcpu mapping done by the admin for this.
> 
> 
> 
> Regards,
> Daniel

Quite clear splitting and +1 for P/V pin option.

--jyh



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

On 11/18/2013 06:47 PM, Joshua Harlow wrote:

An idea related to this, what would need to be done to make the DB have
the exact state that a compute node is going through (and therefore the
scheduler would not make unreliable/racey decisions, even when there are
multiple schedulers). It's not like we are dealing with a system which
can not know the exact state (as long as the compute nodes are connected
to the network, and a network partition does not occur).


How would you synchronize the various schedulers with each other? 
Suppose you have multiple scheduler nodes all trying to boot multiple 
instances each.


Even if each at the start of the process each scheduler has a perfect 
view of the system, each scheduler would need to have a view of what 
every other scheduler is doing in order to not make racy decisions.


I see a few options:

1) Push scheduling down into the database itself.  Implement scheduler 
filters as SQL queries or stored procedures.


2) Get rid of the DB for scheduling.  It looks like people are working 
on this: https://blueprints.launchpad.net/nova/+spec/no-db-scheduler


3) Do multi-stage scheduling.  Do a "tentative" schedule, then try and 
update the DB to reserve all the necessary resources.  If that fails, 
someone got there ahead of you so try again with the new data.


Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reg : Security groups implementation using openflows in quantum ovs plugin

2013-11-19 Thread Amir Sadoughi

Yes, my work has been on ML2 with neutron-openvswitch-agent.  I’m interested to 
see what Jun Park has. I might have something ready before he is available 
again, but would like to collaborate regardless.

Amir

On Nov 19, 2013, at 3:31 AM, Kanthi P 
mailto:pavuluri.kan...@gmail.com>> wrote:

Hi All,

Thanks for the response!
Amir,Mike: Is your implementation being done according to ML2 plugin

Regards,
Kanthi

On Tue, Nov 19, 2013 at 1:43 AM, Mike Wilson 
mailto:geekinu...@gmail.com>> wrote:
Hi Kanthi,

Just to reiterate what Kyle said, we do have an internal implementation using 
flows that looks very similar to security groups. Jun Park was the guy that 
wrote this and is looking to get it upstreamed. I think he'll be back in the 
office late next week. I'll point him to this thread when he's back.

-Mike

On Mon, Nov 18, 2013 at 3:39 PM, Kyle Mestery (kmestery) 
mailto:kmest...@cisco.com>> wrote:
On Nov 18, 2013, at 4:26 PM, Kanthi P 
mailto:pavuluri.kan...@gmail.com>> wrote:
> Hi All,
>
> We are planning to implement quantum security groups using openflows for ovs 
> plugin instead of iptables which is the case now.
>
> Doing so we can avoid the extra linux bridge which is connected between the 
> vnet device and the ovs bridge, which is given as a work around since ovs 
> bridge is not compatible with iptables.
>
> We are planning to create a blueprint and work on it. Could you please share 
> your views on this
>
Hi Kanthi:

Overall, this idea is interesting and removing those extra bridges would 
certainly be nice. Some people at Bluehost gave a talk at the Summit [1] in 
which they explained they have done something similar, you may want to reach 
out to them since they have code for this internally already.

The OVS plugin is in feature freeze during Icehouse, and will be deprecated in 
favor of ML2 [2] at the end of Icehouse. I would advise you to retarget your 
work at ML2 when running with the OVS agent instead. The Neutron team will not 
accept new features into the OVS plugin anymore.

Thanks,
Kyle

[1] 
http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/towards-truly-open-and-commoditized-software-defined-networks-in-openstack
[2] https://wiki.openstack.org/wiki/Neutron/ML2

> Thanks,
> Kanthi
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Caitlin Bestler

On 11/18/2013 11:35 AM, Mike Spreitzer wrote:

There were some concerns expressed at the summit about scheduler
scalability in Nova, and a little recollection of Boris' proposal to
keep the needed state in memory.  I also heard one guy say that he
thinks Nova does not really need a general SQL database, that a NOSQL
database with a bit of denormalization and/or client-maintained
secondary indices could suffice.  Has that sort of thing been considered
before?  What is the community's level of interest in exploring that?

Thanks,
Mike



How the data is stored is not the central question. The real issue is 
how is the data normalized and distributed.


Data that is designed to be distributed deals with temporary 
inconsistencies and only worries about eventual consistency.

Once you have that you can store the data in Objects, or in
a distributed database.

If you define your data so that you need global synchronization then
you will always be fighting scaling issues.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] - Vendor specific erros

2013-11-19 Thread Salvatore Orlando

Thanks Avishay.

I think the status description error was introduced with this aim.
Whether vendor-specific error descriptions can make sense to a tenant,
that's a good question.
Personally, I feel like as a tenant that information would not be a lot
useful to me, as I would not be able to do any debug or maintenance on the
appliance where the error was generated; on the other hand, as a deployer I
might find that message very useful, but probably I would look for it in
the logs rather than in API responses; furthermore, as a deployer I might
find more convenient to not provide tenants any detail about the peculiar
driver being used.

On this note however, this is just my personal opinion. I'm sure there are
plenty of valid use cases for giving tenants vendor-specific error messages.

Salvatore


On 19 November 2013 13:00, Avishay Balderman  wrote:

>  Hi Salvatore
>
> I think you are mixing between the state machine (ACTIVE,PENDEING_XYZ,etc)
>  and the status description
>
> All I want to do is to write a vendor specific error message when the
> state is ERROR.
>
> I DO NOT want to touch the state machine.
>
>
>
> See: https://bugs.launchpad.net/neutron/+bug/1248423
>
>
>
> Thanks
>
>
>
> Avishay
>
>
>
> *From:* Salvatore Orlando [mailto:sorla...@nicira.com]
> *Sent:* Thursday, November 14, 2013 1:15 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [Neutron] - Vendor specific erros
>
>
>
> In general, an error state make sense.
>
> I think you might want to send more details about how this state plugs
> into the load balancer state machine, but I reckon it is a generally
> non-recoverable state which could be reached by any other state; in that
> case it would be a generic enough case which might be supported by all
> drivers.
>
>
>
> It is good to point out that driver-specific state transitions however, in
> my opinion, are to avoid; application using the Neutron API will become
> non-portable, or at least users of the Neutron API would need to be aware
> that an entity might have a different state machine from driver to driver,
> which I reckon would be bad enough for a developer to decide to switch over
> to Cloudstack or AWS APIs!
>
>
>
> Salvatore
>
>
>
> PS: On the last point I am obviously joking, but not so much.
>
>
>
>
>
> On 12 November 2013 08:00, Avishay Balderman  wrote:
>
>
>
> Hi
>
> Some of the DB entities in the LBaaS domain inherit from
> HasStatusDescription
>
> With this we can set the entity status (ACTIVE, PENDING_CREATE,etc) and a
> description for the status.
>
> There are flows in the Radware LBaaS driver that the  driver needs to set
> the entity status to ERROR and it is able to set the description of the
> error –  the description is Radware specific.
>
> My question is:  Does it make sense to do that?
>
> After all the tenant is aware to the fact that he works against Radware
> load balancer -  the tenant selected Radware as the lbaas provider in the
> UI.
>
> Any reason not to do that?
>
>
>
> This is a generic issue/question and does not relate  to a specific plugin
> or driver.
>
>
>
> Thanks
>
>
>
> Avishay
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread James Bottomley

On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
> Hey all
> 
> Not having been at the summit (maybe the next one), could somebody
> give a really short explanation as to why it needs to be a separate
> service?
> It sounds like it should fit within the Nova area. It is, after all,
> just another hypervisor type, or so it seems.

I can take a stab at this:  Firstly, a container is *not* a hypervisor.
Hypervisor based virtualisation is done at the hardware level (so with
hypervisors you boot a second kernel on top of the virtual hardware),
container based virtualisation is done at the OS (kernel) level (so all
containers share the same kernel ... and sometimes even huge chunks of
the OS). With recent advances in the Linux Kernel, we can make a
container behave like a hypervisor (full OS/IaaS virtualisation), but
quite a bit of the utility of containers is that they can do much more
than hypervisors, so they shouldn't be constrained by hypervisor APIs
(which are effectively virtual hardware APIs).

It is possible to extend the Nova APIs to control containers more fully,
but there was resistance do doing this on the grounds that it's
expanding the scope of Nova, hence the new project.

James

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration

Excerpts from Zane Bitter's message of 2013-11-15 12:41:53 -0800:
> Good news, everyone! I have created the missing whiteboard diagram that 
> we all needed at the design summit:
> 
> https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram
> 
> I've documented 5 possibilities. (1) is the current implementation, 
> which we agree we want to get away from. I strongly favour (2) for the 
> reasons listed. I don't think (3) has many friends. (4) seems to be 
> popular despite the obvious availability problem and doubts that it is 
> even feasible. Finally, I can save us all some time by simply stating 
> that I will -2 on sight any attempt to implement (5).
> 
> When we're discussing this, please mention explicitly the number of the 
> model you are talking about at any given time.
> 
> If you have a suggestion for a different model, make your own diagram! 
> jk, you can sketch it or something for me and I'll see if I can add it.

Thanks for putting this together Zane. I just now got around to looking
closely.

Option 2 is good. I'd love for option 1 to be made automatic by making
the client smarter, but parsing templates in the client will require
some deep thought before we decide it is a good idea.

I'd like to consider a 2a, which just has the same Heat engines the user
is talking to being used to do the orchestration in whatever region
they are in. I think that is actually the intention of the diagram,
but it looks like there is a "special" one that talks to the engines
that actually do the work.

2 may morph into 3 actually, if users don't like the nested stack
requirement for 2, we can do the work to basically make the engine create
a nested stack per region. So that makes 2 a stronger choice for first
implementation.

4 has an unstated pro, which is that attack surface is reduced. This
makes more sense when you consider the TripleO case where you may want
the undercloud (hardware cloud) to orchestrate things existing in the
overcloud (vm cloud) but you don't want the overcloud administrators to
be able to control your entire stack.

Given CAP theorem, option 5, the global orchestrator, would be doable
with not much change as long as partition tolerance were the bit we gave
up. We would just have to have a cross-region RPC bus and database. Of
course, since regions are most likely to be partitioned, that is not
really a good choice. Trading partition tolerance for consistency lands
us in the complexity black hole. Trading out availability makes it no
better than option 4.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Paula Ta-Shma

> My apologies, I'm apparently coming into this quite late :) Would you
> mind sharing a link to the HP proposal? I wasn't at the summit
> unfortunately and am playing a bit of catch up.

Hi Jay,

Take a look at https://wiki.openstack.org/wiki/MetadataSearch
There are links there to the REST API proposed by HP and the slides
presented at the summit.

regards
Paula



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] How to determine patch set load for a given project

2013-11-19 Thread Ilya Shakhat

Matt,

As an option you may estimate the load using Stackalytics data on number of
commits -
http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=sqlalchemy-migrateNumber
of commits is certainly less than number of patches, but for project
sqlalchemy-migrate the multiplier 2 will give a good estimation.

Thanks,
Ilya


2013/11/19 Matt Riedemann 

> We have a team working on getting CI setup for DB2 10.5 in
> sqlalchemy-migrate and they were asking me if there was a way to calculate
> the patch load through that project.
>
> I asked around in the infra IRC channel and Jeremy Stanley pointed out
> that there might be something available in http://graphite.openstack.org/by 
> looking for the project's test stats.
>
> I found that if you expand stats_counts > zuul > job and then search for
> your project (sqlalchemy-migrate in this case), you can find the jobs and
> their graphs for load. In my case I care about stats for
> gate-sqlalchemy-migrate-python27.
>
> I'm having a little trouble interpreting the data though. From looking at
> what's out there for review now, there is one new patch created on 11/19
> and the last new one before that was on 11/15. I see spikes in the graph
> around 11/15, 11/18 and 11/19, but I'm not sure what the 11/18 spike is
> from?
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

2013-11-19 Thread Christopher Armstrong

On Mon, Nov 18, 2013 at 5:57 AM, Zane Bitter  wrote:

> On 16/11/13 11:15, Angus Salkeld wrote:
>
>> On 15/11/13 08:46 -0600, Christopher Armstrong wrote:
>>
>>> On Fri, Nov 15, 2013 at 3:57 AM, Zane Bitter  wrote:
>>>
>>>  On 15/11/13 02:48, Christopher Armstrong wrote:

  On Thu, Nov 14, 2013 at 5:40 PM, Angus Salkeld  > wrote:
>
> On 14/11/13 10:19 -0600, Christopher Armstrong wrote:
>
> http://docs.heatautoscale.__apiary.io/
>
> 
>
> I've thrown together a rough sketch of the proposed API for
> autoscaling.
> It's written in API-Blueprint format (which is a simple subset
> of Markdown)
> and provides schemas for inputs and outputs using JSON-Schema.
> The source
> document is currently at
> https://github.com/radix/heat/__raw/as-api-spike/
> autoscaling.__apibp
>
>
>  >
>
>
> Things we still need to figure out:
>
> - how to scope projects/domains. put them in the URL? get them
> from the
> token?
> - how webhooks are done (though this shouldn't affect the API
> too much;
> they're basically just opaque)
>
> Please read and comment :)
>
>
> Hi Chistopher
>
> In the group create object you have 'resources'.
> Can you explain what you expect in there? I thought we talked at
> summit about have a unit of scaling as a nested stack.
>
> The thinking here was:
> - this makes the new config stuff easier to scale (config get
> applied
> Â  per scaling stack)
>
> - you can potentially place notification resources in the scaling
> Â  stack (think marconi message resource - on-create it sends a
> Â  message)
>
> - no need for a launchconfig
> - you can place a LoadbalancerMember resource in the scaling stack
> Â  that triggers the loadbalancer to add/remove it from the lb.
>
>
> I guess what I am saying is I'd expect an api to a nested stack.
>
>
> Well, what I'm thinking now is that instead of "resources" (a
> mapping of
> resources), just have "resource", which can be the template definition
> for a single resource. This would then allow the user to specify a
> Stack
> resource if they want to provide multiple resources. How does that
> sound?
>
>
 My thought was this (digging into the implementation here a bit):

 - Basically, the autoscaling code works as it does now: creates a
 template
 containing OS::Nova::Server resources (changed from AWS::EC2::Instance),
 with the properties obtained from the LaunchConfig, and creates a
 stack in
 Heat.
 - LaunchConfig can now contain any properties you like (I'm not 100%
 sure
 about this one*).
 - The user optionally supplies a template. If the template is
 supplied, it
 is passed to Heat and set in the environment as the provider for the
 OS::Nova::Server resource.


  I don't like the idea of binding to OS::Nova::Server specifically for
>>> autoscaling. I'd rather have the ability to scale *any* resource,
>>> including
>>> nested stacks or custom resources. It seems like jumping through hoops to
>>>
>>
>> big +1 here, autoscaling should not even know what it is scaling, just
>> some resource. solum might want to scale all sorts of non-server
>> resources (and other users).
>>
>
> I'm surprised by the negative reaction to what I suggested, which is a
> completely standard use of provider templates. Allowing a user-defined
> stack of resources to stand in for an unrelated resource type is the entire
> point of providers. Everyone says that it's a great feature, but if you try
> to use it for something they call it a "hack". Strange.
>

To clarify this position (which I already did in IRC), replacing one
concrete resource with another that means something in a completely
different domain is a hack -- say, replacing "server" with "group of
related resources". However, replacing OS::Nova::Server with something
which still does something very much like creating a server is reasonable
-- e.g., using a different API like one for creating containers or using a
different cloud provider's API.


>
> So, allow me to make a slight modification to my proposal:
>
> - The autoscaling service manages a template containing
> OS::Heat::ScaledResource resources. This is an imaginary resource type that
> is not backed by a plugin in Heat.
> - If no template is supplied by the user, the environment declares another
> resource plugin as the provider for OS::Heat::ScaledResource (by default it
> would be OS::Nova::Serv

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Jay Pipes

On 11/19/2013 01:08 PM, Paula Ta-Shma wrote:



My apologies, I'm apparently coming into this quite late :) Would you
mind sharing a link to the HP proposal? I wasn't at the summit
unfortunately and am playing a bit of catch up.


Hi Jay,

Take a look at https://wiki.openstack.org/wiki/MetadataSearch
There are links there to the REST API proposed by HP and the slides
presented at the summit.


Thank you, Paula!

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Personally I would prefer #3 from the below. #2 I think will still have to
deal with consistency issues, just switching away from a DB doesn't make
magical ponies and unicorns appear (in-fact it can potentially make the
problem worse if its done incorrectly - and its pretty easy to get it
wrong IMHO). #1 could also work, but then u hit a vertical scaling limit
(works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer
#2 since I think it is honestly needed under all solutions.

On 11/19/13 9:29 AM, "Chris Friesen"  wrote:

>On 11/18/2013 06:47 PM, Joshua Harlow wrote:
>> An idea related to this, what would need to be done to make the DB have
>> the exact state that a compute node is going through (and therefore the
>> scheduler would not make unreliable/racey decisions, even when there are
>> multiple schedulers). It's not like we are dealing with a system which
>> can not know the exact state (as long as the compute nodes are connected
>> to the network, and a network partition does not occur).
>
>How would you synchronize the various schedulers with each other?
>Suppose you have multiple scheduler nodes all trying to boot multiple
>instances each.
>
>Even if each at the start of the process each scheduler has a perfect
>view of the system, each scheduler would need to have a view of what
>every other scheduler is doing in order to not make racy decisions.
>
>I see a few options:
>
>1) Push scheduling down into the database itself.  Implement scheduler
>filters as SQL queries or stored procedures.
>
>2) Get rid of the DB for scheduling.  It looks like people are working
>on this: https://blueprints.launchpad.net/nova/+spec/no-db-scheduler
>
>3) Do multi-stage scheduling.  Do a "tentative" schedule, then try and
>update the DB to reserve all the necessary resources.  If that fails,
>someone got there ahead of you so try again with the new data.
>
>Chris
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Sam Alba

On Tue, Nov 19, 2013 at 6:45 AM, Chuck Short  wrote:
> Hi
>
> I am excited to see containers getting such traction in the openstack
> project.
>
>
> On Mon, Nov 18, 2013 at 7:30 PM, Russell Bryant  wrote:
>>
>> On 11/18/2013 06:30 PM, Dan Smith wrote:
>> >> Not having been at the summit (maybe the next one), could somebody
>> >> give a really short explanation as to why it needs to be a separate
>> >> service? It sounds like it should fit within the Nova area. It is,
>> >> after all, just another hypervisor type, or so it seems.
>> >
>> > But it's not just another hypervisor. If all you want from your
>> > containers is lightweight VMs, then nova is a reasonable place to put
>> > that (and it's there right now). If, however, you want to expose the
>> > complex and flexible attributes of a container, such as being able to
>> > overlap filesystems, have fine-grained control over what is shared with
>> > the host OS, look at the processes within a container, etc, then nova
>> > ends up needing quite a bit of change to support that.
>> >
>> > I think the overwhelming majority of folks in the room, after discussing
>> > it, agreed that Nova is infrastructure and containers is more of a
>> > platform thing. Making it a separate service lets us define a mechanism
>> > to manage these that makes much more sense than treating them like VMs.
>> > Using Nova to deploy VMs that run this service is the right approach,
>> > IMHO. Clayton put it very well, I think:
>> >
>> >   If the thing you want to deploy has a kernel, then you need Nova. If
>> >   your thing runs on a kernel, you want $new_service_name.
>> >
>> > I agree.
>> >
>> > Note that this is just another service under the compute project (or
>> > program, or whatever the correct terminology is this week).
>>
>> The Compute program is correct.  That is established terminology as
>> defined by the TC in the last cycle.
>>
>> > So while
>> > distinct from Nova in terms of code, development should be tightly
>> > integrated until (and if at some point) it doesn't make sense.
>>
>> And it may share a whole bunch of the code.
>>
>> Another way to put this:  The API requirements people have for
>> containers include a number of features considered outside of the
>> current scope of Nova (short version: Nova's scope stops before going
>> *inside* the servers it creates, except file injection, which we plan to
>> remove anyway).  That presents a problem.  A new service is one possible
>> solution.
>>
>> My view of the outcome of the session was not "it *will* be a new
>> service".  Instead, it was, "we *think* it should be a new service, but
>> let's do some more investigation to decide for sure".
>>
>> The action item from the session was to go off and come up with a
>> proposal for what a new service would look like.  In particular, we
>> needed a proposal for what the API would look like.  With that in hand,
>> we need to come back and ask the question again of whether a new service
>> is the right answer.
>>
>> I see 3 possible solutions here:
>>
>> 1) Expand the scope of Nova to include all of the things people want to
>> be able to do with containers.
>>
>> This is my least favorite option.  Nova is already really big.  We've
>> worked to split things out (Networking, Block Storage, Images) to keep
>> it under control.  I don't think a significant increase in scope is a
>> smart move for Nova's future.
>>
>
> This is my least favorite option. Like a lot of other responses already I
> see a lot of code duplication  because Nova and the new nova container's
> project. This just doesn't include the scheduler but  things like config
> driver, etc.

Can we dig into this option? Honestly, I'd be glad to find a way to
avoid reimplementing everything again (a new compute service with
Keystone, Glance, Horizon integration, etc...). But I do understand
the limitation of changing Nova to improve containers support.

Can someone bring more details (maybe in the spec etherpad, in a new
section) about this 3rd option?

Since the API (in the front) and the virt API (in the back) have to be
different, I barely see how we can reuse most of Nova's code.

>>
>> 2) Declare containers as explicitly out of scope and start a new project
>> with its own API.
>>
>> That is what is being proposed here.
>>
>> 3) Some middle ground that is a variation of #2.  Consider Ironic.  The
>> idea is that Nova's API will still be used for basic provisioning, which
>> Nova will implement by talking to Ironic.  However, there are a lot of
>> baremetal management things that don't fit in Nova at all, and those
>> only exist in Ironic's API.
>
>
> This is my preferred choice  as well. If we could leverage the existing nova
> API and extend it to include containers and features that users who use
> containers in their existing production environment wants.
>>
>>
>> I wanted to mention this option for completeness, but I don't actually
>> think it's the right choice here.  With Ironic you have a physical

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

If you start adding these states you might really want to consider the
following work that is going on in other projects.

It surely appears that everyone is starting to hit the same problem (and
joining efforts would produce a more beneficial result).

Relevant icehouse etherpads:
- https://etherpad.openstack.org/p/CinderTaskFlowFSM
- https://etherpad.openstack.org/p/icehouse-oslo-service-synchronization

And of course my obvious plug for taskflow (which is designed to be a
useful library to help in all these usages).

- https://wiki.openstack.org/wiki/TaskFlow

The states u just mentioned start to line-up with
https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow

If this sounds like a useful way to go (joining efforts) then lets see how
we can make it possible.

IRC: #openstack-state-management is where I am usually at.

On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:

>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>Robert Kukura  wrote:
>
>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>> > Developers,
>> > 
>> > This topic has been discussed before but I do not remember if we have
>>a
>> > good solution or not.
>> 
>> The ML2 plugin addresses this by calling each MechanismDriver twice. The
>> create_network_precommit() method is called as part of the DB
>> transaction, and the create_network_postcommit() method is called after
>> the transaction has been committed. Interactions with devices or
>> controllers are done in the postcommit methods. If the postcommit method
>> raises an exception, the plugin deletes that partially-created resource
>> and returns the exception to the client. You might consider a similar
>> approach in your plugin.
>
>Splitting works into two phase, pre/post, is good approach.
>But there still remains race window.
>Once the transaction is committed, the result is visible to outside.
>So the concurrent request to same resource will be racy.
>There is a window after pre_xxx_yyy before post_xxx_yyy() where
>other requests can be handled.
>
>The state machine needs to be enhanced, I think. (plugins need
>modification)
>For example, adding more states like pending_{create, delete, update}.
>Also we would like to consider serializing between operation of ports
>and subnets. or between operation of subnets and network depending on
>performance requirement.
>(Or carefully audit complex status change. i.e.
>changing port during subnet/network update/deletion.)
>
>I think it would be useful to establish reference locking policy
>for ML2 plugin for SDN controllers.
>Thoughts or comments? If this is considered useful and acceptable,
>I'm willing to help.
>
>thanks,
>Isaku Yamahata
>
>> -Bob
>> 
>> > Basically, if concurrent API calls are sent to Neutron, all of them
>>are
>> > sent to the plug-in level where two actions have to be made:
>> > 
>> > 1. DB transaction ? No just for data persistence but also to collect
>>the
>> > information needed for the next action
>> > 2. Plug-in back-end implementation ? In our case is a call to the
>>python
>> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>> > 
>> > For instance:
>> > 
>> > def create_port(self, context, port):
>> > with context.session.begin(subtransactions=True):
>> > # Plugin DB - Port Create and Return port
>> > port_db = super(NeutronPluginPLUMgridV2,
>> > self).create_port(context,
>> >   
>> port)
>> > device_id = port_db["device_id"]
>> > if port_db["device_owner"] == "network:router_gateway":
>> > router_db = self._get_router(context, device_id)
>> > else:
>> > router_db = None
>> > try:
>> > LOG.debug(_("PLUMgrid Library: create_port() called"))
>> > # Back-end implementation
>> > self._plumlib.create_port(port_db, router_db)
>> > except Exception:
>> > Š
>> > 
>> > The way we have implemented at the plugin-level in Havana (even in
>> > Grizzly) is that both action are wrapped in the same "transaction"
>>which
>> > automatically rolls back any operation done to its original state
>> > protecting mostly the DB of having any inconsistency state or left
>>over
>> > data if the back-end part fails.=.
>> > The problem that we are experiencing is when concurrent calls to the
>> > same API are sent, the number of operation at the plug-in back-end are
>> > long enough to make the next concurrent API call to get stuck at the
>>DB
>> > transaction level, which creates a hung state for the Neutron Server
>>to
>> > the point that all concurrent API calls will fail.
>> > 
>> > This can be fixed if we include some "locking" system such as calling:
>> > 
>> > from neutron.common import utile
>> > Š
>> > 
>> > @utils.synchronized('any-name', external=True)
>> > def create_port(self, context, port):
>> > Š
>> > 
>> > Obviously, this will create a serialization of all concurrent calls
>> > which will ends up in having a really bad performance.

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Excerpts from Chris Friesen's message of 2013-11-19 09:29:00 -0800:
> On 11/18/2013 06:47 PM, Joshua Harlow wrote:
> > An idea related to this, what would need to be done to make the DB have
> > the exact state that a compute node is going through (and therefore the
> > scheduler would not make unreliable/racey decisions, even when there are
> > multiple schedulers). It's not like we are dealing with a system which
> > can not know the exact state (as long as the compute nodes are connected
> > to the network, and a network partition does not occur).
> 
> How would you synchronize the various schedulers with each other? 
> Suppose you have multiple scheduler nodes all trying to boot multiple 
> instances each.
> 
> Even if each at the start of the process each scheduler has a perfect 
> view of the system, each scheduler would need to have a view of what 
> every other scheduler is doing in order to not make racy decisions.
> 

Your question assumes they need to be "in sync" at a granular level.

Each scheduler process can own a different set of resources. If they
each grab instance requests in a round-robin fashion, then they will
fill their resources up in a relatively well balanced way until one
scheduler's resources are exhausted. At that time it should bow out of
taking new instances. If it can't fit a request in, it should kick the
request out for retry on another scheduler.

In this way, they only need to be in sync in that they need a way to
agree on who owns which resources. A distributed hash table that gets
refreshed whenever schedulers come and go would be fine for that.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

And also of course, nearly forgot a similar situation/review in heat.

https://review.openstack.org/#/c/49440/

Except theres was/is dealing with stack locking (a heat concept).

On 11/19/13 10:33 AM, "Joshua Harlow"  wrote:

>If you start adding these states you might really want to consider the
>following work that is going on in other projects.
>
>It surely appears that everyone is starting to hit the same problem (and
>joining efforts would produce a more beneficial result).
>
>Relevant icehouse etherpads:
>- https://etherpad.openstack.org/p/CinderTaskFlowFSM
>- https://etherpad.openstack.org/p/icehouse-oslo-service-synchronization
>
>And of course my obvious plug for taskflow (which is designed to be a
>useful library to help in all these usages).
>
>- https://wiki.openstack.org/wiki/TaskFlow
>
>The states u just mentioned start to line-up with
>https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow
>
>If this sounds like a useful way to go (joining efforts) then lets see how
>we can make it possible.
>
>IRC: #openstack-state-management is where I am usually at.
>
>On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:
>
>>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>>Robert Kukura  wrote:
>>
>>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>>> > Developers,
>>> > 
>>> > This topic has been discussed before but I do not remember if we have
>>>a
>>> > good solution or not.
>>> 
>>> The ML2 plugin addresses this by calling each MechanismDriver twice.
>>>The
>>> create_network_precommit() method is called as part of the DB
>>> transaction, and the create_network_postcommit() method is called after
>>> the transaction has been committed. Interactions with devices or
>>> controllers are done in the postcommit methods. If the postcommit
>>>method
>>> raises an exception, the plugin deletes that partially-created resource
>>> and returns the exception to the client. You might consider a similar
>>> approach in your plugin.
>>
>>Splitting works into two phase, pre/post, is good approach.
>>But there still remains race window.
>>Once the transaction is committed, the result is visible to outside.
>>So the concurrent request to same resource will be racy.
>>There is a window after pre_xxx_yyy before post_xxx_yyy() where
>>other requests can be handled.
>>
>>The state machine needs to be enhanced, I think. (plugins need
>>modification)
>>For example, adding more states like pending_{create, delete, update}.
>>Also we would like to consider serializing between operation of ports
>>and subnets. or between operation of subnets and network depending on
>>performance requirement.
>>(Or carefully audit complex status change. i.e.
>>changing port during subnet/network update/deletion.)
>>
>>I think it would be useful to establish reference locking policy
>>for ML2 plugin for SDN controllers.
>>Thoughts or comments? If this is considered useful and acceptable,
>>I'm willing to help.
>>
>>thanks,
>>Isaku Yamahata
>>
>>> -Bob
>>> 
>>> > Basically, if concurrent API calls are sent to Neutron, all of them
>>>are
>>> > sent to the plug-in level where two actions have to be made:
>>> > 
>>> > 1. DB transaction ? No just for data persistence but also to collect
>>>the
>>> > information needed for the next action
>>> > 2. Plug-in back-end implementation ? In our case is a call to the
>>>python
>>> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>>> > 
>>> > For instance:
>>> > 
>>> > def create_port(self, context, port):
>>> > with context.session.begin(subtransactions=True):
>>> > # Plugin DB - Port Create and Return port
>>> > port_db = super(NeutronPluginPLUMgridV2,
>>> > self).create_port(context,
>>> >  
>>> port)
>>> > device_id = port_db["device_id"]
>>> > if port_db["device_owner"] == "network:router_gateway":
>>> > router_db = self._get_router(context, device_id)
>>> > else:
>>> > router_db = None
>>> > try:
>>> > LOG.debug(_("PLUMgrid Library: create_port()
>>>called"))
>>> > # Back-end implementation
>>> > self._plumlib.create_port(port_db, router_db)
>>> > except Exception:
>>> > Š
>>> > 
>>> > The way we have implemented at the plugin-level in Havana (even in
>>> > Grizzly) is that both action are wrapped in the same "transaction"
>>>which
>>> > automatically rolls back any operation done to its original state
>>> > protecting mostly the DB of having any inconsistency state or left
>>>over
>>> > data if the back-end part fails.=.
>>> > The problem that we are experiencing is when concurrent calls to the
>>> > same API are sent, the number of operation at the plug-in back-end
>>>are
>>> > long enough to make the next concurrent API call to get stuck at the
>>>DB
>>> > transaction level, which creates a hung state for the Neutron Server
>>>to
>>> > the point that all concurrent API calls will fail.
>>> > 
>>> > This can be fi

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Eric Windisch

On Tue, Nov 19, 2013 at 1:02 PM, James Bottomley
 wrote:
> On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
>> Hey all
>>
>> Not having been at the summit (maybe the next one), could somebody
>> give a really short explanation as to why it needs to be a separate
>> service?
>> It sounds like it should fit within the Nova area. It is, after all,
>> just another hypervisor type, or so it seems.
>
> I can take a stab at this:  Firstly, a container is *not* a hypervisor.
> Hypervisor based virtualisation is done at the hardware level (so with
> hypervisors you boot a second kernel on top of the virtual hardware),
> container based virtualisation is done at the OS (kernel) level (so all
> containers share the same kernel ... and sometimes even huge chunks of
> the OS). With recent advances in the Linux Kernel, we can make a
> container behave like a hypervisor (full OS/IaaS virtualisation), but
> quite a bit of the utility of containers is that they can do much more
> than hypervisors, so they shouldn't be constrained by hypervisor APIs
> (which are effectively virtual hardware APIs).
>
> It is possible to extend the Nova APIs to control containers more fully,
> but there was resistance do doing this on the grounds that it's
> expanding the scope of Nova, hence the new project.

It might be worth noting that it was also brought up that
hypervisor-based virtualization can offer a number of features that
bridge some of these gaps, but are not supported in, nor may ever be
supported in Nova.

For example, Daniel brings up an interesting point with the
libvirt-sandbox feature. This is one of those features that bridges
some of the gaps. There are also solutions, however brittle, for
introspection that work on hypervisor-driven VMs. It is not clear what
the scope or desire for these features might be, how they might be
sufficiently abstracted between hypervisors and guest OSes, nor how
these would fit into any of the existing or planned compute API
buckets.

Having a separate service for managing containers draws a thick line
in the sand that will somewhat stiffen innovation around
hypervisor-based virtualization. That isn't necessarily a bad thing,
it will help maintain stability in the project. However, the choice
and the implications shouldn't be ignored.

-- 
Regards,
Eric Windisch

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Sorry that was I prefer #3 (not #2) at the end there. Keyboard failure ;)

On 11/19/13 10:27 AM, "Joshua Harlow"  wrote:

>Personally I would prefer #3 from the below. #2 I think will still have to
>deal with consistency issues, just switching away from a DB doesn't make
>magical ponies and unicorns appear (in-fact it can potentially make the
>problem worse if its done incorrectly - and its pretty easy to get it
>wrong IMHO). #1 could also work, but then u hit a vertical scaling limit
>(works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer
>#2 since I think it is honestly needed under all solutions.
>
>On 11/19/13 9:29 AM, "Chris Friesen"  wrote:
>
>>On 11/18/2013 06:47 PM, Joshua Harlow wrote:
>>> An idea related to this, what would need to be done to make the DB have
>>> the exact state that a compute node is going through (and therefore the
>>> scheduler would not make unreliable/racey decisions, even when there
>>>are
>>> multiple schedulers). It's not like we are dealing with a system which
>>> can not know the exact state (as long as the compute nodes are
>>>connected
>>> to the network, and a network partition does not occur).
>>
>>How would you synchronize the various schedulers with each other?
>>Suppose you have multiple scheduler nodes all trying to boot multiple
>>instances each.
>>
>>Even if each at the start of the process each scheduler has a perfect
>>view of the system, each scheduler would need to have a view of what
>>every other scheduler is doing in order to not make racy decisions.
>>
>>I see a few options:
>>
>>1) Push scheduling down into the database itself.  Implement scheduler
>>filters as SQL queries or stored procedures.
>>
>>2) Get rid of the DB for scheduling.  It looks like people are working
>>on this: https://blueprints.launchpad.net/nova/+spec/no-db-scheduler
>>
>>3) Do multi-stage scheduling.  Do a "tentative" schedule, then try and
>>update the DB to reserve all the necessary resources.  If that fails,
>>someone got there ahead of you so try again with the new data.
>>
>>Chris
>>
>>___
>>OpenStack-dev mailing list
>>OpenStack-dev@lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] [Ironic] [TripleO] scheduling flow with Ironic?

2013-11-19 Thread Devananda van der Veen

On Wed, Nov 13, 2013 at 10:11 PM, Alex Glikson  wrote:

> Thanks, I understand the Nova scheduler part. One of the gaps there is
> related to the blueprint we have are working on [1]. I was wondering
> regarding the role of Ironic, and the exact interaction between the user,
> Nova and Ironic.
>

The interaction from the point of "nova boot" onwards will be the same --
nova maintains a list of available (host, node) resources, the scheduler
picks one according to the request, dispatches the work to n-cond / n-cpu,
which in turn calls down to various methods in the nova/virt/driver API.
The implementation of the ironic driver is a wrapper around
python-ironicclient library, which will make calls out to the ironic API
service, which in turn performs the necessary work.

Where the interaction is different is around the management of physical
machines; eg, enrolling them with Ironic, temporarily marking a machine as
unavailable while doing maintenance on it, and other sorts of things we
haven't actually written the code for yet.

> In particular, initially I thought that Ironic is going to have its own
> scheduler, resolving some of the issues and complexity within Nova (which
> could focus on VM management, maybe even getting rid of hosts versus nodes,
> etc).

I'm not sure how putting a scheduler in Ironic would solve this problem at
all.

Conversely, I don't think there's any need for the whole (host, node)
thing. Chris Behrens and I talked at the last summit about a possible
rewrite to nova-conductor that would remove the need for this distinction
entirely. I would love to see Nova just track node, and I think this can
work for typical hypervisors (kvm, xen, ...) as well.

> But it seems that Ironic aims to stay at the level of virt driver API.. It
> is a bit unclear to me what is the desired architecture going forward -
> e.g., if the idea is to standardize virt driver APIs but keep the
> scheduling centralized,

AFAIK, the nova.virt.driver API is the standard that all the virt drivers
are written to. Unless you're referring to libvirt's API, in which case, I
don't understand the question.

> maybe we should take the rest of virt drivers into separate projects as
> well, and extend Nova to schedule beyond just compute (if it is already
> doing so for virt + bare-metal).

Why would Nova need to schedule anything besides compute resources? In this
context, Ironic is merely providing a different type of compute resource,
and Nova is still scheduling compute workloads. That this hypervisor driver
has different scheduling characteristics (eg, flavor == node resource;
extra_specs:cpu_arch == node arch; and so on) than other hypervisor drivers
doesn't mean it's not still a compute resource.

> Alternatively, each of them could have its own scheduler (like the
> approach we took when splitting out cinder, for example) - and then someone
> on top (e.g., Heat) would need to do the cross-project logic. Taking
> different architectural approaches in different cases confuses me a bit.
>

Yes, well, Cinder is a different type of resource (block storage).

HTH,
-Deva
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] RFC: Potential to increase min required libvirt version to 0.9.8 ?

Currently the Nova libvirt driver is declaring that it wants a minimum
of libvirt 0.9.6.

For cases where we use features newer than this, we have to do conditional
logic to ensure we operate correctly on old libvirt. We don't want to keep
adding conditionals forever since they complicate the code and thus impose
an ongoing dev burden.

I think it is wise to re-evaluate the min required libvirt version at the
start of each dev cycle. To do this we need to understand what versions
are available in the OS distributions we care about supporting for the
Icehouse release.

To this end I've created a wiki page covering Debian, Fedora, OpenSUSE,
RHEL and Ubuntu

  https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix

Considering those distros it looks like we can potentially increase the
min required libvirt to 0.9.8 for Icehouse.

If there are other distros I've missed which expect to support deployment
of Icehouse please add them to this list. Hopefully there won't be any
with libvirt software older than Ubuntu 12.04 LTS


The reason I'm asking this now, is that we're working to make the libvirt
python module a separate tar.gz that can build with multiple libvirt
versions, and I need to decide how ancient a libvirt we should support
for it.

Regards.
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread James Bottomley

On Tue, 2013-11-19 at 13:46 -0500, Eric Windisch wrote:
> On Tue, Nov 19, 2013 at 1:02 PM, James Bottomley
>  wrote:
> > On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
> >> Hey all
> >>
> >> Not having been at the summit (maybe the next one), could somebody
> >> give a really short explanation as to why it needs to be a separate
> >> service?
> >> It sounds like it should fit within the Nova area. It is, after all,
> >> just another hypervisor type, or so it seems.
> >
> > I can take a stab at this:  Firstly, a container is *not* a hypervisor.
> > Hypervisor based virtualisation is done at the hardware level (so with
> > hypervisors you boot a second kernel on top of the virtual hardware),
> > container based virtualisation is done at the OS (kernel) level (so all
> > containers share the same kernel ... and sometimes even huge chunks of
> > the OS). With recent advances in the Linux Kernel, we can make a
> > container behave like a hypervisor (full OS/IaaS virtualisation), but
> > quite a bit of the utility of containers is that they can do much more
> > than hypervisors, so they shouldn't be constrained by hypervisor APIs
> > (which are effectively virtual hardware APIs).
> >
> > It is possible to extend the Nova APIs to control containers more fully,
> > but there was resistance do doing this on the grounds that it's
> > expanding the scope of Nova, hence the new project.
> 
> It might be worth noting that it was also brought up that
> hypervisor-based virtualization can offer a number of features that
> bridge some of these gaps, but are not supported in, nor may ever be
> supported in Nova.
> 
> For example, Daniel brings up an interesting point with the
> libvirt-sandbox feature. This is one of those features that bridges
> some of the gaps. There are also solutions, however brittle, for
> introspection that work on hypervisor-driven VMs. It is not clear what
> the scope or desire for these features might be, how they might be
> sufficiently abstracted between hypervisors and guest OSes, nor how
> these would fit into any of the existing or planned compute API
> buckets.

It's certainly possible, but some of them are possible in the same way
as it's possible to get a square peg into a round hole by beating the
corners flat with a sledge hammer ... it works, but it's much less
hassle just to use a round peg because it actually fits the job.

> Having a separate service for managing containers draws a thick line
> in the sand that will somewhat stiffen innovation around
> hypervisor-based virtualization. That isn't necessarily a bad thing,
> it will help maintain stability in the project. However, the choice
> and the implications shouldn't be ignored.

How about this: we get the container API agreed and we use a driver
model like Nova (we have to anyway since there about four different
container technologies interested in this), then we see if someone wants
to do a hypervisor driver emulating the features.

James



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

On Tue, Nov 19, 2013 at 10:02:45AM -0800, James Bottomley wrote:
> On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
> > Hey all
> > 
> > Not having been at the summit (maybe the next one), could somebody
> > give a really short explanation as to why it needs to be a separate
> > service?
> > It sounds like it should fit within the Nova area. It is, after all,
> > just another hypervisor type, or so it seems.
> 
> I can take a stab at this:  Firstly, a container is *not* a hypervisor.
> Hypervisor based virtualisation is done at the hardware level (so with
> hypervisors you boot a second kernel on top of the virtual hardware),
> container based virtualisation is done at the OS (kernel) level (so all
> containers share the same kernel ... and sometimes even huge chunks of
> the OS). With recent advances in the Linux Kernel, we can make a
> container behave like a hypervisor (full OS/IaaS virtualisation), but
> quite a bit of the utility of containers is that they can do much more
> than hypervisors, so they shouldn't be constrained by hypervisor APIs
> (which are effectively virtual hardware APIs).
> 
> It is possible to extend the Nova APIs to control containers more fully,
> but there was resistance do doing this on the grounds that it's
> expanding the scope of Nova, hence the new project.

You're focusing on the low level technical differences between containers
and hypervisor here which IMHO is not the right comparison to be making.
Once you look at the high level use cases many of the referenced container
virt apps are trying to address things don't look anywhere near as clear
cut. As mentioned elsewhere libvirt-sandbox which provides a application
sandboxing toolkit is able to leverage either LXC or KVM virt to achieve
its high level goals. Based on my understanding of Docker, I believe it
would actually be possible to run Docker images under KVM without much
difficultly. There will certainly be some setups that aren't possible
todo with hypervisors, but I don't think those will be in the majority,
nor require starting again from scratch, throwing out Nova.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

2013-11-19 Thread Steven Dake


On 11/17/2013 01:57 PM, Steve Baker wrote:

On 11/15/2013 05:19 AM, Christopher Armstrong wrote:

http://docs.heatautoscale.apiary.io/

I've thrown together a rough sketch of the proposed API for
autoscaling. It's written in API-Blueprint format (which is a simple
subset of Markdown) and provides schemas for inputs and outputs using
JSON-Schema. The source document is currently
at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp


Apologies if I'm about to re-litigate an old argument, but...

At summit we discussed creating a new endpoint (and new pythonclient)
for autoscaling. Instead I think the autoscaling API could just be added
to the existing heat-api endpoint.

Arguments for just making auto scaling part of heat api include:
* Significantly less development, packaging and deployment configuration
of not creating a heat-autoscaling-api and python-autoscalingclient
* Autoscaling is orchestration (for some definition of orchestration) so
belongs in the orchestration service endpoint
* The autoscaling API includes heat template snippets, so a heat service
is a required dependency for deployers anyway
* End-users are still free to use the autoscaling portion of the heat
API without necessarily being aware of (or directly using) heat
templates and stacks
* It seems acceptable for single endpoints to manage many resources (eg,
the increasingly disparate list of resources available via the neutron API)

Arguments for making a new auto scaling api include:
* Autoscaling is not orchestration (for some narrower definition of
orchestration)
* Autoscaling implementation will be handled by something other than
heat engine (I have assumed the opposite)
(no doubt this list will be added to in this thread)
A separate process can be autoscaled independently of heat-api which is 
a big plus architecturally.


They really do different things, and separating their concerns at the 
process level is a good goal.


I prefer a separate process for these reasons.

Regards
-steve


What do you think?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

On 11/19/2013 12:35 PM, Clint Byrum wrote:


Each scheduler process can own a different set of resources. If they
each grab instance requests in a round-robin fashion, then they will
fill their resources up in a relatively well balanced way until one
scheduler's resources are exhausted. At that time it should bow out of
taking new instances. If it can't fit a request in, it should kick the
request out for retry on another scheduler.

In this way, they only need to be in sync in that they need a way to
agree on who owns which resources. A distributed hash table that gets
refreshed whenever schedulers come and go would be fine for that.


That has some potential, but at high occupancy you could end up refusing 
to schedule something because no one scheduler has sufficient resources 
even if the cluster as a whole does.


This gets worse once you start factoring in things like heat and 
instance groups that will want to schedule whole sets of resources 
(instances, IP addresses, network links, cinder volumes, etc.) at once 
with constraints on where they can be placed relative to each other.


Chris


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Version Negotiation Middleware Accept Header issue

2013-11-19 Thread Fuente, Pablo A

Hi,
I noticed that the Accept HTTP Header checked in the Version
Negotiation Middleware by Heat is the same MIME type used by Glance
("application/vnd.openstack.images-")
Is this OK, or should it be something like
"application/vnd.openstack.orchestration-"?
If this is the case I would proceed to file a bug.

Thanks.
Pablo.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

On 11/19/2013 12:27 PM, Joshua Harlow wrote:

Personally I would prefer #3 from the below. #2 I think will still have to
deal with consistency issues, just switching away from a DB doesn't make
magical ponies and unicorns appear (in-fact it can potentially make the
problem worse if its done incorrectly - and its pretty easy to get it
wrong IMHO). #1 could also work, but then u hit a vertical scaling limit
(works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer
#3 since I think it is honestly needed under all solutions.


Personally I think we need a combination of #3 (resource reservation) 
with something else to speed up scheduling.


We have multiple filters that currently loop over all the compute nodes, 
gathering a bunch of data from the DB and then ignoring most of that 
data while doing some simple logic in python.


There is really no need for the bulk of the resource information to be 
stored in the DB.  The compute nodes could broadcast their current state 
to all scheduler nodes, and the scheduler nodes could reserve resources 
directly from the compute nodes (triggering an update of all the other 
scheduler nodes).


Failing that, it should be possible to push at least some of the 
filtering down into the DB itself. Stuff like ramfilter or cpufilter 
would be trival (and fast) as an SQL query.


Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:
> On 11/19/2013 12:35 PM, Clint Byrum wrote:
> 
> > Each scheduler process can own a different set of resources. If they
> > each grab instance requests in a round-robin fashion, then they will
> > fill their resources up in a relatively well balanced way until one
> > scheduler's resources are exhausted. At that time it should bow out of
> > taking new instances. If it can't fit a request in, it should kick the
> > request out for retry on another scheduler.
> >
> > In this way, they only need to be in sync in that they need a way to
> > agree on who owns which resources. A distributed hash table that gets
> > refreshed whenever schedulers come and go would be fine for that.
> 
> That has some potential, but at high occupancy you could end up refusing 
> to schedule something because no one scheduler has sufficient resources 
> even if the cluster as a whole does.
> 

I'm not sure what you mean here. What resource spans multiple compute
hosts?

> This gets worse once you start factoring in things like heat and 
> instance groups that will want to schedule whole sets of resources 
> (instances, IP addresses, network links, cinder volumes, etc.) at once 
> with constraints on where they can be placed relative to each other.
> 

Actually that is rather simple. Such requests have to be serialized
into a work-flow. So if you say "give me 2 instances in 2 different
locations" then you allocate 1 instance, and then another one with
'not_in_location(1)' as a condition.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Rick Jones

On 11/19/2013 10:02 AM, James Bottomley wrote:

It is possible to extend the Nova APIs to control containers more fully,
but there was resistance do doing this on the grounds that it's
expanding the scope of Nova, hence the new project.


How well received would another CLI/API to learn be among the end-users?

rick jones

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

On 11/19/2013 01:51 PM, Clint Byrum wrote:

Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:

On 11/19/2013 12:35 PM, Clint Byrum wrote:

Each scheduler process can own a different set of resources. If they
each grab instance requests in a round-robin fashion, then they will
fill their resources up in a relatively well balanced way until one
scheduler's resources are exhausted. At that time it should bow out of
taking new instances. If it can't fit a request in, it should kick the
request out for retry on another scheduler.

In this way, they only need to be in sync in that they need a way to
agree on who owns which resources. A distributed hash table that gets
refreshed whenever schedulers come and go would be fine for that.

That has some potential, but at high occupancy you could end up refusing
to schedule something because no one scheduler has sufficient resources
even if the cluster as a whole does.

I'm not sure what you mean here. What resource spans multiple compute
hosts?

Imagine the cluster is running close to full occupancy, each scheduler
has room for 40 more instances. Now I come along and issue a single
request to boot 50 instances. The cluster has room for that, but none
of the schedulers do.

This gets worse once you start factoring in things like heat and
instance groups that will want to schedule whole sets of resources
(instances, IP addresses, network links, cinder volumes, etc.) at once
with constraints on where they can be placed relative to each other.

Actually that is rather simple. Such requests have to be serialized
into a work-flow. So if you say "give me 2 instances in 2 different
locations" then you allocate 1 instance, and then another one with
'not_in_location(1)' as a condition.

Actually, you don't want to serialize it, you want to hand the whole set
of resource requests and constraints to the scheduler all at once.

If you do them one at a time, then early decisions made with
less-than-complete knowledge can result in later scheduling requests
failing due to being unable to meet constraints, even if there are
actually sufficient resources in the cluster.

The "VM ensembles" document at
https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4UTwsmhw/edit?pli=1
has a good example of how one-at-a-time scheduling can cause spurious
failures.

And if you're handing the whole set of requests to a scheduler all at
once, then you want the scheduler to have access to as many resources as
possible so that it has the highest likelihood of being able to satisfy
the request given the constraints.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] SSL Termination write-up

2013-11-19 Thread Eugene Nikanorov

Hi Vijay,

Thanks for working on this. As was discussed at the summit, immediate
solution seems to be passing certificates via transient fields in Vip
object, which will avoid the need for certificate management (incl. storing
them).
If certificate management is concerned then I agree that it needs to be a
separate specialized service.

> My thought was to have independent certificate resource with VIP uuid as
one of the properties. VIP is already created and
> will help to identify the driver/device. The VIP property can be
depreciated in the long term.
I think it's a little bit forward looking at this moment, also I think that
certificates are not specific for load balancing.
Transient fields could help here too: client could pass certificate Id and
driver of corresponding device will communicate with external service
fetching corresponding certificate.

Thanks,
Eugene.


On Tue, Nov 19, 2013 at 5:48 PM, Vijay Venkatachalam <
vijay.venkatacha...@citrix.com> wrote:

>  Hi Sam, Eugene, & Avishay, etal,
>
>
>
> Today I spent some time to create a write-up for SSL
> Termination not exactly design doc. Please share your comments!
>
>
>
>
> https://docs.google.com/document/d/1tFOrIa10lKr0xQyLVGsVfXr29NQBq2nYTvMkMJ_inbo/edit
>
>
>
> Would like comments/discussion especially on the following note:
>
>
>
> SSL Termination requires certificate management. The ideal way is to
> handle this via an independent IAM service. This would take time to
> implement so the thought was to add the certificate details in VIP resource
> and send them directly to device. Basically don’t store the certificate key
> in the DB there by avoiding security concerns of maintaining certificates
> in controller.
>
>
>
> I would expect the certificates to become an independent resource in
> future thereby causing backward compatibility issues.
>
>
>
> Any ideas how to achieve this?
>
>
>
> My thought was to have independent certificate resource with VIP uuid as
> one of the properties. VIP is already created and will help to identify the
> driver/device. The VIP property can be depreciated in the long term.
>
>  Thanks,
>
> Vijay V.
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Russell Bryant

On 11/19/2013 03:08 PM, Rick Jones wrote:
> On 11/19/2013 10:02 AM, James Bottomley wrote:
>> It is possible to extend the Nova APIs to control containers more fully,
>> but there was resistance do doing this on the grounds that it's
>> expanding the scope of Nova, hence the new project.
> 
> How well received would another CLI/API to learn be among the end-users?

It depends.  It is it mostly a duplication of the compute API, but just
slightly different enough to be annoying?  Or is it something
significantly different enough that much better suits their use cases?

That's the important question to answer here.  How different is it, and
does the difference justify a split?

The opinions so far are quite vast.  Those interested in pushing this
are going to go off and work on a more detailed API proposal.  I think
we should revisit this discussion when they have something for us to
evaluate against.

Even if this ends up staying in Nova, the API work is still useful.  It
would help us toward defining a container specific extension to the
compute API.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

2013-11-19 Thread Steve Baker

On 11/19/2013 08:37 PM, Thomas Spatzier wrote:
> Steve Baker  wrote on 18.11.2013 21:52:04:
>> From: Steve Baker 
>> To: openstack-dev@lists.openstack.org,
>> Date: 18.11.2013 21:54
>> Subject: Re: [openstack-dev] [Heat] HOT software configuration
>> refined after design summit discussions
>>
>> On 11/19/2013 02:22 AM, Thomas Spatzier wrote:
>>> Hi all,
>>>
>>> I have reworked the wiki page [1] I created last week to reflect
>>> discussions we had on the mail list and in IRC. From ML discussions
> last
>>> week it looked like we were all basically on the same page (with some
>>> details to be worked out), and I hope the new draft eliminates some
>>> confusion that the original draft had.
>>>
>>> [1]
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
>> Thanks Thomas, this looks really good. I've actually started on a POC
>> which maps to this model.
> Good to hear that, Steve :-)
> Now that we are converging, should we consolidate the various wiki pages
> and just have one? E.g. copy the complete contents of
> hot-software-config-WIP to your original hot-software-config, or deprecate
> all others and make hot-software-config-WIP the master?
Lets just bless hot-software-config-WIP and add to it as we flesh out
the implementation.

>> I've used different semantics which you may actually prefer some of,
>> please comment below.
>>
>> Resource types:
>> SoftwareConfig -> SoftwareConfig (yay!)
>> SoftwareDeployment -> SoftwareApplier - less typing, less mouth-fatigue
> I'm ok with SoftwareApplier. If we don't hear objections, I can change it
> in the wiki.
>
>> SoftwareConfig properties:
>> parameters -> inputs - just because parameters is overloaded already.
> Makes sense.
>
>> Although if the CM tool has their own semantics for inputs then that
>> should be used in that SoftwareConfig resource implementation instead.
>> outputs -> outputs
>>
>> SoftwareApplier properties:
>> software_config -> apply_config - because there will sometimes be a
>> corresponding remove_config
> Makes sense, and the remove_config thought is a very good point!
>
>> server -> server
>> parameters -> input_values - to match the 'inputs' schema property in
>> SoftwareConfig
> Agree on input_values.
>
>> Other comments on hot-software-config-WIP:
>>
>> Regarding apply_config/remove_config, if a SoftwareApplier resource is
>> deleted it should trigger any remove_config and wait for the server to
>> acknowledge when that is complete. This allows for any
>> evacuation/deregistering workloads to be executed.
>>
>> I'm unclear yet what the SoftwareConfig 'role' is for, unless the role
>> specifies the contract for a given inputs and outputs schema? How would
>> this be documented or enforced? I'm inclined to leave it out for now.
> So about 'role', as I stated in the wiki, my thinking was that there will
> be different SoftwareConfig and SoftwareApplier implementations per CM tool
> (more on that below), since all CM tools will probably have their specific
> metadata and runtime implementation. So in my example I was using Chef, and
> 'role' is just a Chef concept, i.e. you take a cookbook and configure a
> specific Chef role on a server.
OK, its Chef specific; I'm fine with that.
>> It should be possible to write a SoftwareConfig type for a new CM tool
>> as a provider template. This has some nice implications for deployers
>> and users.
> I think provider templates are a good thing to have clean componentization
> for re-use. However, I think it still would be good to allow users to
> define their SoftwareConfigs inline in a template for simple use cases. I
> heard that requirement in several posts on the ML last week.
> The question is whether we can live with a single implementation of
> SoftwareConfig and SoftwareApplier then (see also below).
Yes, a provider template would encapsulate some base SoftwareConfig
resource type, but users would be free to use this type inline in their
template too.
>> My hope is that there will not need to be a different SoftwareApplier
>> type written for each CM tool. But maybe there will be one for each
>> delivery mechanism. The first implementation will use metadata polling
>> and signals, another might use Marconi. Bootstrapping an image to
>> consume a given CM tool and applied configuration data is something that
>> we need to do, but we can make it beyond the scope of this particular
>> proposal.
> I was thinking about a single implementation, too. However, I cannot really
> imagine how one single implementation could handle both the different
> metadata of different CM tools, and different runtime implementation. I
> think we would want to support at least a handful of the most favorite
> tools, but cannot see at the moment how to cover them all in one
> implementation. My thought was that there could be a super-class for common
> behavior, and then plugins with specific behavior for each tool.
>
> Anyway, all of that needs to be verified, so working on PoC patches is
> def

Re: [openstack-dev] [Ironic][Ceilometer] get IPMI data for ceilometer

2013-11-19 Thread Devananda van der Veen

On Mon, Nov 18, 2013 at 10:35 AM, Ladislav Smola  wrote:

>  Hello. I have a couple of additional questions.
>
> 1. What about IPMI data that we want to get by polling. E.g. temperatures,
> etc. Will the Ironic be polling these kind of
> data and send them directly to collector(or agent)? Not sure if this
> belongs to Ironic. It would have to support some
> pluggable architecture for vendor specific pollsters like Ceilometer.
>
>
If there is a fixed set of information (eg, temp, fan speed, etc) that
ceilometer will want, let's make a list of that and add a driver interface
within Ironic to abstract the collection of that information from physical
nodes. Then, each driver will be able to implement it as necessary for that
vendor. Eg., an iLO driver may poll its nodes differently than a generic
IPMI driver, but the resulting data exported to Ceilometer should have the
same structure.

I don't think we should, at least right now, support pluggable pollsters on
the Ceilometer->Ironic side. Let's start with a small set of data that
Ironic exposes, make it pluggable internally for different types of
hardware, and iterate if necessary.


> 2. I've seen in the etherpad that the SNMP agent(pollster) will be also
> part of the Ironic(running next to conductor). Is it true?
> Or that will be placed in Ceilometer central agent?
>

An SNMP agent doesn't fit within the scope of Ironic, as far as I see, so
this would need to be implemented by Ceilometer.

As far as where the SNMP agent would need to run, it should be on the same
host(s) as ironic-conductor so that it has access to the management network
(the physically-separate network for hardware management, IPMI, etc). We
should keep the number of applications with direct access to that network
to a minimum, however, so a thin agent that collects and forwards the SNMP
data to the central agent would be preferable, in my opinion.


Regards,
Devananda



>
>
> Thanks for response.
> Ladislav
>
>
>
> On 11/18/2013 06:25 PM, Devananda van der Veen wrote:
>
> Hi Lianhao Lu,
>
>  I briefly summarized my recollection of that session in this blueprint:
>
>  https://blueprints.launchpad.net/ironic/+spec/add-ceilometer-agent
>
>  I've responded to your questions inline as well.
>
>
> On Sun, Nov 17, 2013 at 10:24 PM, Lu, Lianhao wrote:
>
>> Hi stackers,
>>
>> During the summit session Expose hardware sensor (IPMI) data
>> https://etherpad.openstack.org/p/icehouse-summit-ceilometer-hardware-sensors,
>> it was proposed to deploy a ceilometer agent next to the ironic conductor
>> to the get the ipmi data. Here I'd like to ask some questions to figure out
>> what's the current missing pieces in ironic and ceilometer for that
>> proposal.
>>
>> 1. Just double check, ironic won't provide API to get IPMI data, right?
>>
>
>  Correct. This was generally felt to be unnecessary.
>
>>
>> 2. If deploying a ceilometer agent next to the ironic conductor, how does
>> the agent talk to the conductor? Through rpc?
>>
>
>  My understanding is that ironic-conductor will emit messages to the
> ceilimeter agent, and the communication is one-way. These could be
> triggered by a periodic task, or by some other event within Ironic, such as
> a change in the power state of a node.
>
>
>>
>> 3. Does the current ironic conductor have rpc_method to support getting
>> generic ipmi data, i.e. let the rpc_method caller specifying arbitrary
>> netfn/command to get any type of ipmi data?
>>
>
>  No, and as far as I understand, it doesn't need one.
>
>
>>
>> 4. I believe the ironic conductor uses some kind of node_id to associate
>> the bmc with its credentials, right? If so, how can the ceilometer agent
>> get those node_ids to ask the ironic conductor to poll the ipmi data? And
>> how can the ceilometer agent extract meaningful information from that
>> node_id to set those fields in the ceilometer Sample(e.g. recource_id,
>> project_id, user_id, etc.) to identify which physical node the ipmi data is
>> coming from?
>>
>
>  This question perhaps requires a longer answer.
>
>  Ironic references physical machines (nodes) internally with an integer
> node_id and externally with a standard uuid. When a Nova instance is
> created, it will be associated to a node, that node will have a reference
> to the nova instance_uuid which is exposed in our API, and can be passed to
> Ceilometer's agent. I believe that nova instance_uuid will enable
> ceilometer to detect the project, user, etc.
>
>  Should Ironic emit messages regarding nodes which are not provisioned?
> Physical machines that don't have a tenant instance on them are not
> associated to any project, user, tenant, quota, etc, so I suspect that we
> shouldn't notify about them. It would be like tracking the unused disks in
> a SAN.
>
>  Regards,
> Devananda
>
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> 
> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> deleted it should trigger any remove_config and wait for the server to
> acknowledge when that is complete. This allows for any
> evacuation/deregistering workloads to be executed.
> 

I'm a little worried about the road that leads us down. Most configuration
software defines forward progress only. Meaning, if you want something
not there, you don't remove it from your assertions, you assert that it
is not there.

The reason this is different than the way we operate with resources is
that resources are all under Heat's direct control via well defined
APIs. In-instance things, however, will be indirectly controlled. So I
feel like focusing on a "diff" mechanism for user-deployed tools may be
unnecessary and might confuse. I'd much rather have a "converge"
mechanism for the users to focus on.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

2013-11-19 Thread Steve Baker

On 11/20/2013 09:50 AM, Clint Byrum wrote:
> Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
>> Regarding apply_config/remove_config, if a SoftwareApplier resource is
>> deleted it should trigger any remove_config and wait for the server to
>> acknowledge when that is complete. This allows for any
>> evacuation/deregistering workloads to be executed.
>>
> I'm a little worried about the road that leads us down. Most configuration
> software defines forward progress only. Meaning, if you want something
> not there, you don't remove it from your assertions, you assert that it
> is not there.
>
> The reason this is different than the way we operate with resources is
> that resources are all under Heat's direct control via well defined
> APIs. In-instance things, however, will be indirectly controlled. So I
> feel like focusing on a "diff" mechanism for user-deployed tools may be
> unnecessary and might confuse. I'd much rather have a "converge"
> mechanism for the users to focus on.
>
>
A specific use-case I'm trying to address here is tripleo doing an
update-replace on a nova compute node. The remove_config contains the
workload to evacuate VMs and signal heat when the node is ready to be
shut down. This is more involved than just "uninstall the things".

Could you outline in some more detail how you think this could be done?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
> On 11/20/2013 09:50 AM, Clint Byrum wrote:
> > Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> >> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> >> deleted it should trigger any remove_config and wait for the server to
> >> acknowledge when that is complete. This allows for any
> >> evacuation/deregistering workloads to be executed.
> >>
> > I'm a little worried about the road that leads us down. Most configuration
> > software defines forward progress only. Meaning, if you want something
> > not there, you don't remove it from your assertions, you assert that it
> > is not there.
> >
> > The reason this is different than the way we operate with resources is
> > that resources are all under Heat's direct control via well defined
> > APIs. In-instance things, however, will be indirectly controlled. So I
> > feel like focusing on a "diff" mechanism for user-deployed tools may be
> > unnecessary and might confuse. I'd much rather have a "converge"
> > mechanism for the users to focus on.
> >
> >
> A specific use-case I'm trying to address here is tripleo doing an
> update-replace on a nova compute node. The remove_config contains the
> workload to evacuate VMs and signal heat when the node is ready to be
> shut down. This is more involved than just "uninstall the things".
> 
> Could you outline in some more detail how you think this could be done?
> 

So for that we would not remove the software configuration for the
nova-compute, we would assert that the machine needs vms evacuated.
We want evacuation to be something we explicitly do, not a side effect
of deleting things. Perhaps having delete hooks for starting delete
work-flows is right, but it set off a red flag for me so I want to make
sure we think it through.

Also IIRC, evacuation is not necessarily an in-instance thing. It looks
more like the weird thing we've been talking about lately which is
"how do we orchestrate tenant API's":

https://etherpad.openstack.org/p/orchestrate-tenant-apis

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration

2013-11-19 Thread Zane Bitter

On 19/11/13 19:03, Clint Byrum wrote:

Excerpts from Zane Bitter's message of 2013-11-15 12:41:53 -0800:

Good news, everyone! I have created the missing whiteboard diagram that
we all needed at the design summit:

https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram

I've documented 5 possibilities. (1) is the current implementation,
which we agree we want to get away from. I strongly favour (2) for the
reasons listed. I don't think (3) has many friends. (4) seems to be
popular despite the obvious availability problem and doubts that it is
even feasible. Finally, I can save us all some time by simply stating
that I will -2 on sight any attempt to implement (5).

When we're discussing this, please mention explicitly the number of the
model you are talking about at any given time.

If you have a suggestion for a different model, make your own diagram!
jk, you can sketch it or something for me and I'll see if I can add it.

Thanks for putting this together Zane. I just now got around to looking
closely.

Option 2 is good. I'd love for option 1 to be made automatic by making
the client smarter, but parsing templates in the client will require
some deep thought before we decide it is a good idea.

I'd like to consider a 2a, which just has the same Heat engines the user
is talking to being used to do the orchestration in whatever region
they are in. I think that is actually the intention of the diagram,
but it looks like there is a "special" one that talks to the engines
that actually do the work.

Yes, I think you're saying the same thing as the diagram was intended to
convey. (Each orange box is meant to be a Heat engine.)

So the user talks to the Heat endpoint in a region of their choice
(doesn't matter which, they're all the same). When a OS::Heat::Stack
resource has Region and/or Endpoint specified, it will use
python-heatclient to connect to the appropriate engine for that nested
stack.

2 may morph into 3 actually, if users don't like the nested stack
requirement for 2, we can do the work to basically make the engine create
a nested stack per region. So that makes 2 a stronger choice for first
implementation.

Yeah, if you run your own standalone Heat engine, you've effectively
created 3 with no code changes from 2 afaict. I wouldn't recommend that
operators deploy this way though.

4 has an unstated pro, which is that attack surface is reduced. This
makes more sense when you consider the TripleO case where you may want
the undercloud (hardware cloud) to orchestrate things existing in the
overcloud (vm cloud) but you don't want the overcloud administrators to
be able to control your entire stack.

Given CAP theorem, option 5, the global orchestrator, would be doable
with not much change as long as partition tolerance were the bit we gave

Well, in the real world partitions _do_ happen, so the choice is to give
up consistency, availability or both.

up. We would just have to have a cross-region RPC bus and database. Of
course, since regions are most likely to be partitioned, that is not
really a good choice. Trading partition tolerance for consistency lands
us in the complexity black hole. Trading out availability makes it no
better than option 4.

Yep, exactly.

cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Easier way of trying TripleO

2013-11-19 Thread James Slagle

I'd like to propose an idea around a simplified and complimentary version of
devtest that makes it easier for someone to get started and try TripleO.  

The goal being to get people using TripleO as a way to experience the
deployment of OpenStack, and not necessarily a way to get an experience of a
useable OpenStack cloud itself.

To that end, we could:

1) Provide an undercloud vm image so that you could effectively skip the entire
   seed setup.
2) Provide pre-built downloadable images for the overcloud and deployment
   kernel and ramdisk.
3) Instructions on how to use these images to deploy a running
   overcloud.

Images could be provided for Ubuntu and Fedora, since both those work fairly
well today.

The instructions would look something like:

1) Download all the images.
2) Perform initial host setup.  This would be much smaller than what is
   required for devtest and off the top of my head would mostly be:
   - openvswitch bridge setup
   - libvirt configuration
   - ssh configuration (for the baremetal virtual power driver)
3) Start the undercloud vm.  It would need to be bootstrapped with an initial
   static json file for the heat metadata, same as the seed works today.
4) Any last mile manual configuration, such as nova.conf edits for the virtual
   power driver user.
6) Use tuskar+horizon (running on the undercloud)  to deploy the overcloud.
7) Overcloud configuration (don't see this being much different than what is
   there today).

All the openstack clients, heat templates, etc., are on the undercloud vm, and
that's where they're used from, as opposed to from the host (results in less 
stuff
to install/configure on the host).

We could also provide instructions on how to configure the undercloud vm to
provision baremetal.  I assume this would be possible, given the correct
bridged networking setup.

It could make sense to use an all in one overcloud for this as well, given it's
going for simplification.

Obviously, this approach implies some image management on the community's part,
and I think we'd document and use all the existing tools (dib, elements) to
build images, etc.

Thoughts on this approach?  

--
-- James Slagle
--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Icehouse roadmap status

2013-11-19 Thread Russell Bryant

Greetings,

We've made a lot of progress on reviewing blueprints over the last week
and have the icehouse-1 [1] list in reasonable shape.  Note that
icehouse-1 development must be completed two weeks from today, and that
time frame includes a major holiday in the US.  Please adjust plans and
expectations accordingly.

Here are the goals we need to be aiming for over this next week:

1) File blueprints!  We talked about a *lot* of stuff in the design
summit that is still not reflected in blueprints.  Please get those
blueprints filed and targerted to the appropriate icehouse milestone ASAP!

2) nova-core reviewers should consider committing to reviewing
blueprints they care most about.  That will give us a better picture of
which blueprints are most likely to get reviewed in time to be merged.

The process here is to just add a note to the blueprint whiteboard
indicating yourself as a review sponsor.  For example:

"nova-core sponsors: russellb, alaski"

Once a blueprint has at least two sponsors, its priority can be raised
above Low.

3) nova-drivers: We need to keep up the pace on blueprint reviews.
There's still a lot to review in the overall icehouse list [2].  It's a
lot of work right now, early in the cycle while there's a lot of
planning going on.  It will start to taper off in coming weeks.


Please let me know if you have any questions.

Thanks everyone!


[1] https://launchpad.net/nova/+milestone/icehouse-1
[2] https://blueprints.launchpad.net/nova/icehouse

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] PTL election

2013-11-19 Thread Thierry Carrez

Thierry Carrez wrote:
> Thierry Carrez wrote:
>> The two candidates who nominated themselves in time for this election are:
>>
>> * David Lyle
>> * Matthias Runge
>>
>> The election will be set up tomorrow, and will stay open for voting for
>> a week.

The poll is now closed, and the winner is David Lyle !

You can see results at:

http://www.cs.cornell.edu/w8/~andru/cgi-perl/civs/results.pl?id=E_cd16dd051e519ef2

Congrats, David!

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] PTL election

2013-11-19 Thread Matthias Runge

On 11/19/2013 11:02 PM, Thierry Carrez wrote:

> 
> The poll is now closed, and the winner is David Lyle !
> 
David, well done and honestly deserved!

Matthias


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Salvatore Orlando

For what is worth we have considered this aspect from the perspective of
the Neutron plugin my team maintains (NVP) during the past release cycle.

The synchronous model that most plugins with a controller on the backend
currently implement is simple and convenient, but has some flaws:

- reliability: the current approach where the plugin orchestrates the
backend is not really optimal when it comes to ensuring your running
configuration (backend/control plane) is in sync with your desired
configuration (neutron/mgmt plane); moreover in some case, due to neutron
internals, API calls to the backend are wrapped in a transaction too,
leading to very long SQL transactions, which are quite dangerous indeed. It
is not easy to recover from a failure due to an eventlet thread deadlocking
with a mysql transaction, where by 'recover' I mean ensuring neutron and
backend state are in sync.

- maintainability: since handling rollback in case of failures on the
backend and/or the db is cumbersome, this often leads to spaghetti code
which is very hard to maintain regardless of the effort (ok, I agree here
that this also depends on how good the devs are - most of the guys in my
team are very good, but unfortunately they have me too...).

- performance & scalability:
-  roundtrips to the backend take a non-negligible toll on the duration
of an API call, whereas most Neutron API calls should probably just
terminate at the DB just like a nova boot call does not wait for the VM to
be ACTIVE to return.
- we need to keep some operation serialized in order to avoid the
mentioned race issues

For this reason we're progressively moving toward a change in the NVP
plugin with a series of patches under this umbrella-blueprint [1].

For answering the issues mentioned by Isaku, we've been looking at a task
management library with an efficient and reliable set of abstractions for
ensuring operations are properly ordered thus avoiding those races (I agree
on the observation on the pre/post commit solution).
We are currently looking at using celery [2] rather than taskflow; mostly
because we've already have expertise on how to use it into our
applications, and has very easy abstractions for workflow design, as well
as for handling task failures.
Said that, I think we're still open to switch to taskflow should we become
aware of some very good reason for using it.

Regards,
Salvatore

[1]
https://blueprints.launchpad.net/neutron/+spec/nvp-async-backend-communication
[2] http://docs.celeryproject.org/en/master/index.html



On 19 November 2013 19:42, Joshua Harlow  wrote:

> And also of course, nearly forgot a similar situation/review in heat.
>
> https://review.openstack.org/#/c/49440/
>
> Except theres was/is dealing with stack locking (a heat concept).
>
> On 11/19/13 10:33 AM, "Joshua Harlow"  wrote:
>
> >If you start adding these states you might really want to consider the
> >following work that is going on in other projects.
> >
> >It surely appears that everyone is starting to hit the same problem (and
> >joining efforts would produce a more beneficial result).
> >
> >Relevant icehouse etherpads:
> >- https://etherpad.openstack.org/p/CinderTaskFlowFSM
> >- https://etherpad.openstack.org/p/icehouse-oslo-service-synchronization
> >
> >And of course my obvious plug for taskflow (which is designed to be a
> >useful library to help in all these usages).
> >
> >- https://wiki.openstack.org/wiki/TaskFlow
> >
> >The states u just mentioned start to line-up with
> >https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow
> >
> >If this sounds like a useful way to go (joining efforts) then lets see how
> >we can make it possible.
> >
> >IRC: #openstack-state-management is where I am usually at.
> >
> >On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:
> >
> >>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
> >>Robert Kukura  wrote:
> >>
> >>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
> >>> > Developers,
> >>> >
> >>> > This topic has been discussed before but I do not remember if we have
> >>>a
> >>> > good solution or not.
> >>>
> >>> The ML2 plugin addresses this by calling each MechanismDriver twice.
> >>>The
> >>> create_network_precommit() method is called as part of the DB
> >>> transaction, and the create_network_postcommit() method is called after
> >>> the transaction has been committed. Interactions with devices or
> >>> controllers are done in the postcommit methods. If the postcommit
> >>>method
> >>> raises an exception, the plugin deletes that partially-created resource
> >>> and returns the exception to the client. You might consider a similar
> >>> approach in your plugin.
> >>
> >>Splitting works into two phase, pre/post, is good approach.
> >>But there still remains race window.
> >>Once the transaction is committed, the result is visible to outside.
> >>So the concurrent request to same resource will be racy.
> >>There is a window after pre_xxx_yyy before post_xxx_yyy() where
> >>other requests can be handled.
> >>
>

Re: [openstack-dev] [Heat] Version Negotiation Middleware Accept Header issue

2013-11-19 Thread Angus Salkeld

On 19/11/13 19:40 +, Fuente, Pablo A wrote:

Hi,
I noticed that the Accept HTTP Header checked in the Version
Negotiation Middleware by Heat is the same MIME type used by Glance
("application/vnd.openstack.images-")
Is this OK, or should it be something like
"application/vnd.openstack.orchestration-"?
If this is the case I would proceed to file a bug.


Yeah, that  looks wrong, I posted a fix here:
https://review.openstack.org/57338

-Angus



Thanks.
Pablo.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

2013-11-19 Thread Zane Bitter

On 19/11/13 19:14, Christopher Armstrong wrote:

On Mon, Nov 18, 2013 at 5:57 AM, Zane Bitter mailto:zbit...@redhat.com>> wrote:

On 16/11/13 11:15, Angus Salkeld wrote:

On 15/11/13 08:46 -0600, Christopher Armstrong wrote:

On Fri, Nov 15, 2013 at 3:57 AM, Zane Bitter
mailto:zbit...@redhat.com>> wrote:

On 15/11/13 02:48, Christopher Armstrong wrote:

On Thu, Nov 14, 2013 at 5:40 PM, Angus Salkeld
mailto:asalk...@redhat.com>
>> wrote:

 On 14/11/13 10:19 -0600, Christopher Armstrong
wrote:

http://docs.heatautoscale.__ap__iary.io/

 >

 I've thrown together a rough sketch of the
proposed API for
 autoscaling.
 It's written in API-Blueprint format (which
is a simple subset
 of Markdown)
 and provides schemas for inputs and outputs
using JSON-Schema.
 The source
 document is currently at
https://github.com/radix/heat/raw/as-api-spike/

autoscaling.__apibp

 >

 Things we still need to figure out:

 - how to scope projects/domains. put them
in the URL? get them
 from the
 token?
 - how webhooks are done (though this
shouldn't affect the API
 too much;
 they're basically just opaque)

 Please read and comment :)

 Hi Chistopher

 In the group create object you have 'resources'.
 Can you explain what you expect in there? I
thought we talked at
 summit about have a unit of scaling as a nested
stack.

 The thinking here was:
 - this makes the new config stuff easier to
scale (config get
applied
 Â  per scaling stack)

 - you can potentially place notification
resources in the scaling
 Â  stack (think marconi message resource -
on-create it sends a
 Â  message)

 - no need for a launchconfig
 - you can place a LoadbalancerMember resource
in the scaling stack
 Â  that triggers the loadbalancer to add/remove
it from the lb.

 I guess what I am saying is I'd expect an api
to a nested stack.

Well, what I'm thinking now is that instead of
"resources" (a
mapping of
resources), just have "resource", which can be the
template definition
for a single resource. This would then allow the
user to specify a
Stack
resource if they want to provide multiple resources.
How does that
sound?

My thought was this (digging into the implementation
here a bit):

- Basically, the autoscaling code works as it does now:
creates a
template
containing OS::Nova::Server resources (changed from
AWS::EC2::Instance),
with the properties obtained from the LaunchConfig, and
creates a
stack in
Heat.
- LaunchConfig can now contain any properties you like
(I'm not 100%
sure
about this one*).
- The user optionally supplies a template. If the
template is
supplied, it
is passed to Heat and set in the environment as the
provider for the
OS::Nova::Server resource.

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

Can u explain a little how using celery achieves workflow reliability and 
avoids races (or mitigates spaghetti code)?

To me celery acts as a way to distribute tasks, but does not deal with actually 
forming a easily understandable way of knowing that a piece of code that u 
design is actually going to go through the various state transitions (or states 
& workflows) that u expect (this is a higher level mechanism that u can build 
on-top of a distribution system). So this means that NVP (or neutron or other?) 
must be maintaining an orchestration/engine layer on-top of celery to add on 
this additional set of code that 'drives' celery to accomplish a given workflow 
in a reliable manner.

This starts to sound pretty similar to what taskflow is doing, not being a 
direct competitor to a distributed task queue such as celery but providing this 
higher level mechanism which adds on these benefits; since they are needed 
anyway.

To me these benefits currently are (may get bigger in the future):

1. A way to define a workflow (in a way that is not tied to celery, since 
celeries '@task' decorator ties u to celeries internal implementation).
 - This includes ongoing work to determine how to easily define a 
state-machine in a way that is relevant to cinder (and other projects).
2. A way to keep track of the state that the workflow goes through (this brings 
along resumption, progress information… when u track at the right level).
3. A way to execute that workflow reliably (potentially using celery, rpc, 
local threads, other future hotness)
 - This becomes important when u ask yourself how did u plan on testing 
celery in the gate/jenkins/CI?
4. A way to guarantee that the workflow upon failure is *automatically* resumed 
by some other entity.

More details @ http://www.slideshare.net/harlowja/taskflow-27820295

From: Salvatore Orlando mailto:sorla...@nicira.com>>
Date: Tuesday, November 19, 2013 2:22 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Cc: Joshua Harlow mailto:harlo...@yahoo-inc.com>>, 
Isaku Yamahata mailto:isaku.yamah...@gmail.com>>, 
Robert Kukura mailto:rkuk...@redhat.com>>
Subject: Re: [openstack-dev] [Neutron] Race condition between DB layer and 
plugin back-end implementation

For what is worth we have considered this aspect from the perspective of the 
Neutron plugin my team maintains (NVP) during the past release cycle.

The synchronous model that most plugins with a controller on the backend 
currently implement is simple and convenient, but has some flaws:

- reliability: the current approach where the plugin orchestrates the backend 
is not really optimal when it comes to ensuring your running configuration 
(backend/control plane) is in sync with your desired configuration 
(neutron/mgmt plane); moreover in some case, due to neutron internals, API 
calls to the backend are wrapped in a transaction too, leading to very long SQL 
transactions, which are quite dangerous indeed. It is not easy to recover from 
a failure due to an eventlet thread deadlocking with a mysql transaction, where 
by 'recover' I mean ensuring neutron and backend state are in sync.

- maintainability: since handling rollback in case of failures on the backend 
and/or the db is cumbersome, this often leads to spaghetti code which is very 
hard to maintain regardless of the effort (ok, I agree here that this also 
depends on how good the devs are - most of the guys in my team are very good, 
but unfortunately they have me too...).

- performance & scalability:
-  roundtrips to the backend take a non-negligible toll on the duration of 
an API call, whereas most Neutron API calls should probably just terminate at 
the DB just like a nova boot call does not wait for the VM to be ACTIVE to 
return.
- we need to keep some operation serialized in order to avoid the mentioned 
race issues

For this reason we're progressively moving toward a change in the NVP plugin 
with a series of patches under this umbrella-blueprint [1].

For answering the issues mentioned by Isaku, we've been looking at a task 
management library with an efficient and reliable set of abstractions for 
ensuring operations are properly ordered thus avoiding those races (I agree on 
the observation on the pre/post commit solution).
We are currently looking at using celery [2] rather than taskflow; mostly 
because we've already have expertise on how to use it into our applications, 
and has very easy abstractions for workflow design, as well as for handling 
task failures.
Said that, I think we're still open to switch to taskflow should we become 
aware of some very good reason for using it.

Regards,
Salvatore

[1] 
https://blueprints.launchpad.net/neutron/+spec/nvp-async-backend-communication
[2] http://docs.celeryproject.org/en/master/index.html



On 19 November 2013 19:42, Joshua Harlow 
mailto:harlo...@yahoo-inc.com>> wrote:
And also of course, nearly forgot a similar

Re: [openstack-dev] [Nova] New API requirements, review of GCE

2013-11-19 Thread Vishvananda Ishaya

On Nov 15, 2013, at 8:28 AM, Russell Bryant  wrote:

> Greetings,
> 
> We've talked a lot about requirements for new compute drivers [1].  I
> think the same sort of standards shold be applied for a new third-party
> API, such as the GCE API [2].
> 
> Before we can consider taking on a new API, it should have full test
> suite coverage.  Ideally this would be extensions to tempest.  It should
> also be set up to run on every Nova patch via CI.
> 
> Beyond that point, now is a good time to re-consider how we want to
> support new third-party APIs.  Just because EC2 is in the tree doesn't
> mean that has to be how we support them going forward.  Should new APIs
> go into their own repositories?
> 
> I used to be against this idea.  However, as Nova's has grown, the
> importance of finding the right spots to split is even higher.  My
> objection was primarily based on assuming we'd have to make the Python
> APIs stable.  I still do not think we should make them stable, but I
> don't think that's a huge issue, since it should be mitigated by running
> CI so the API maintainers quickly get notified when updates are necessary.
> 
> Taking on a whole new API seems like an even bigger deal than accepting
> a new compute driver, so it's an important question.
> 
> If we went this route, I would encourage new third-party APIs to build
> themselves up in a stackforge repo.  Once it's far enough along, we
> could then evaluate officially bringing it in as an official sub-project
> of the OpenStack Compute program.
> 
> Thoughts?

I like the idea of giving people +2/a rights to a certain part of the tree.
The ec2 api maintainers could have +2/a rights to api/ec2/*. Occasionally ec2
fixes will need to touch other parts of the tree, and this would require regular
core members to approve. A similar thing could be done for gce and occi when 
they
are up to snuff.

This seems like a way that we could keep the code together without adding 
massive
review burden on nova-core.

Vish

> 
> [1] https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan
> [2] https://blueprints.launchpad.net/nova/+spec/gce-api
> 
> -- 
> Russell Bryant
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Salvatore Orlando

Thanks for your reply Joshua,

some more comments inline; however I think I'm probably going off topic
here given the initial subject of this thread.

Talking about Taskflow, and moving aside for the plugins, which have not a
lot of community-wide interest, do you reckon Taskflow might be something
that we might look at to improve reliability and efficiency in the
nova/neutron interface? I have barely started to analyse how that could be
done, but it something which might deserve another thread of discussion.

Regards,
Salvatore


On 19 November 2013 23:56, Joshua Harlow  wrote:

>  Can u explain a little how using celery achieves workflow reliability
> and avoids races (or mitigates spaghetti code)?
>
>
Celery surely does not do that. I have probably not been precise enough in
my previous post. I meant to say that celery is merely the tool we are
going to use for managing a distributed task queue.


>  To me celery acts as a way to distribute tasks, but does not deal with
> actually forming a easily understandable way of knowing that a piece of
> code that u design is actually going to go through the various state
> transitions (or states & workflows) that u expect (this is a higher level
> mechanism that u can build on-top of a distribution system). So this means
> that NVP (or neutron or other?) must be maintaining an orchestration/engine
> layer on-top of celery to add on this additional set of code that 'drives'
> celery to accomplish a given workflow in a reliable manner.
>

That's pretty much correct. The neutron plugin will define workflows and
manage state transitions.


>
>  This starts to sound pretty similar to what taskflow is doing, not being
> a direct competitor to a distributed task queue such as celery but
> providing this higher level mechanism which adds on these benefits; since
> they are needed anyway.
>

>  To me these benefits currently are (may get bigger in the future):
>
>  1. A way to define a workflow (in a way that is not tied to celery,
> since celeries '@task' decorator ties u to celeries internal
> implementation).
>  - This includes ongoing work to determine how to easily define a
> state-machine in a way that is relevant to cinder (and other projects).
> 2. A way to keep track of the state that the workflow goes through (this
> brings along resumption, progress information… when u track at the right
> level).
> 3. A way to execute that workflow reliably (potentially using celery, rpc,
> local threads, other future hotness)
>  - This becomes important when u ask yourself how did u plan on
> testing celery in the gate/jenkins/CI?
> 4. A way to guarantee that the workflow upon failure is *automatically*
> resumed by some other entity.
>

Thanks for this clarification. I will surely look at how the NVP plugin can
leverage taskflow; surely I don't want to reinvent the wheel - or, in other
words, I surely don't want to write code if that code has already been
written by somebody else for me.


>  More details @ http://www.slideshare.net/harlowja/taskflow-27820295
>
>   From: Salvatore Orlando 
> Date: Tuesday, November 19, 2013 2:22 PM
>
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Cc: Joshua Harlow , Isaku Yamahata <
> isaku.yamah...@gmail.com>, Robert Kukura 
> Subject: Re: [openstack-dev] [Neutron] Race condition between DB layer
> and plugin back-end implementation
>
>   For what is worth we have considered this aspect from the perspective
> of the Neutron plugin my team maintains (NVP) during the past release
> cycle.
>
> The synchronous model that most plugins with a controller on the backend
> currently implement is simple and convenient, but has some flaws:
>
>  - reliability: the current approach where the plugin orchestrates the
> backend is not really optimal when it comes to ensuring your running
> configuration (backend/control plane) is in sync with your desired
> configuration (neutron/mgmt plane); moreover in some case, due to neutron
> internals, API calls to the backend are wrapped in a transaction too,
> leading to very long SQL transactions, which are quite dangerous indeed. It
> is not easy to recover from a failure due to an eventlet thread deadlocking
> with a mysql transaction, where by 'recover' I mean ensuring neutron and
> backend state are in sync.
>
>  - maintainability: since handling rollback in case of failures on the
> backend and/or the db is cumbersome, this often leads to spaghetti code
> which is very hard to maintain regardless of the effort (ok, I agree here
> that this also depends on how good the devs are - most of the guys in my
> team are very good, but unfortunately they have me too...).
>
>  - performance & scalability:
> -  roundtrips to the backend take a non-negligible toll on the
> duration of an API call, whereas most Neutron API calls should probably
> just terminate at the DB just like a nova boot call does not wait for the
> VM to be ACTIVE to return

Re: [openstack-dev] [Nova] New API requirements, review of GCE

On 16 November 2013 08:31, Joe Gordon  wrote:

>> and building on unstable Nova APIs. Anything which we accept is a part
>> of OpenStack should not get randomly made unusable by one contributor
>> while other contributors constantly have to scramble to catch up. Either
>> stuff winds up being broken too often or we stifle progress in Nova
>> because we're afraid to make breaking changes.
>
>
> the ceilometer plugin for nova hit this, and had to be scrapped.  It hooked
> into nova-compute and at one point made nova-compute hang there for minutes
> at a time.
>
> I agree, that hooking into our underlying python APIs is a bad idea and a
> recipe for disaster. But at the same time I do like having things live in a
> separate repo, at the very least until they are mature enough to be pulled
> into mainline.
>
> But if we do go with the separate repo solution, what are the issues with
> proxying third party APIs on top of OpenStack REST APIs?  Using the REST
> APIs would mean we have a stable contract for these third party APIs to
> consume, and we also get more feedback about fixing our own API at the same
> time.

As long as the metadataservice doesn't move out :) - that one I think
is pretty core and we have no native replacement [configdrive is not a
replacement :P].

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Easier way of trying TripleO

On 20 November 2013 10:40, James Slagle  wrote:
> I'd like to propose an idea around a simplified and complimentary version of
> devtest that makes it easier for someone to get started and try TripleO.

I think its a grand idea (in fact it's been floated many times). For a
while we ran our own jenkins with such downloadable images.

Right now I think we need to continue the two primary arcs we have:
CI/CD integration and a CD HA overcloud so that we are being tested,
and then work on making the artifacts from those tests available.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New API requirements, review of GCE

2013-11-19 Thread Sean Dague

On 11/19/2013 03:46 PM, Robert Collins wrote:

On 16 November 2013 08:31, Joe Gordon  wrote:


and building on unstable Nova APIs. Anything which we accept is a part
of OpenStack should not get randomly made unusable by one contributor
while other contributors constantly have to scramble to catch up. Either
stuff winds up being broken too often or we stifle progress in Nova
because we're afraid to make breaking changes.



the ceilometer plugin for nova hit this, and had to be scrapped.  It hooked
into nova-compute and at one point made nova-compute hang there for minutes
at a time.

I agree, that hooking into our underlying python APIs is a bad idea and a
recipe for disaster. But at the same time I do like having things live in a
separate repo, at the very least until they are mature enough to be pulled
into mainline.

But if we do go with the separate repo solution, what are the issues with
proxying third party APIs on top of OpenStack REST APIs?  Using the REST
APIs would mean we have a stable contract for these third party APIs to
consume, and we also get more feedback about fixing our own API at the same
time.


As long as the metadataservice doesn't move out :) - that one I think
is pretty core and we have no native replacement [configdrive is not a
replacement :P].


Slightly off tangent thread.

So we recently moved devstack gate to do con fig drive instead of 
metadata service, and life was good (no one really noticed). In what 
ways is configdrive insufficient compared to metadata service? And is 
that something that we should be tackling?


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New API requirements, review of GCE

2013-11-19 Thread Michael Still

On Wed, Nov 20, 2013 at 11:00 AM, Sean Dague  wrote:
> On 11/19/2013 03:46 PM, Robert Collins wrote:

>> As long as the metadataservice doesn't move out :) - that one I think
>> is pretty core and we have no native replacement [configdrive is not a
>> replacement :P].
>
> Slightly off tangent thread.
>
> So we recently moved devstack gate to do con fig drive instead of metadata
> service, and life was good (no one really noticed). In what ways is
> configdrive insufficient compared to metadata service? And is that something
> that we should be tackling?

Config drive and the metadata service present exactly the same data.
The only difference is how it is accessed. There is no reason to think
that config drive should go away any time soon.

Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Excerpts from Chris Friesen's message of 2013-11-19 12:18:16 -0800:
> On 11/19/2013 01:51 PM, Clint Byrum wrote:
> > Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:
> >> On 11/19/2013 12:35 PM, Clint Byrum wrote:
> >>
> >>> Each scheduler process can own a different set of resources. If they
> >>> each grab instance requests in a round-robin fashion, then they will
> >>> fill their resources up in a relatively well balanced way until one
> >>> scheduler's resources are exhausted. At that time it should bow out of
> >>> taking new instances. If it can't fit a request in, it should kick the
> >>> request out for retry on another scheduler.
> >>>
> >>> In this way, they only need to be in sync in that they need a way to
> >>> agree on who owns which resources. A distributed hash table that gets
> >>> refreshed whenever schedulers come and go would be fine for that.
> >>
> >> That has some potential, but at high occupancy you could end up refusing
> >> to schedule something because no one scheduler has sufficient resources
> >> even if the cluster as a whole does.
> >>
> >
> > I'm not sure what you mean here. What resource spans multiple compute
> > hosts?
> 
> Imagine the cluster is running close to full occupancy, each scheduler 
> has room for 40 more instances.  Now I come along and issue a single 
> request to boot 50 instances.  The cluster has room for that, but none 
> of the schedulers do.
> 

You're assuming that all 50 come in at once. That is only one use case
and not at all the most common.

> >> This gets worse once you start factoring in things like heat and
> >> instance groups that will want to schedule whole sets of resources
> >> (instances, IP addresses, network links, cinder volumes, etc.) at once
> >> with constraints on where they can be placed relative to each other.
> 
> > Actually that is rather simple. Such requests have to be serialized
> > into a work-flow. So if you say "give me 2 instances in 2 different
> > locations" then you allocate 1 instance, and then another one with
> > 'not_in_location(1)' as a condition.
> 
> Actually, you don't want to serialize it, you want to hand the whole set 
> of resource requests and constraints to the scheduler all at once.
> 
> If you do them one at a time, then early decisions made with 
> less-than-complete knowledge can result in later scheduling requests 
> failing due to being unable to meet constraints, even if there are 
> actually sufficient resources in the cluster.
> 
> The "VM ensembles" document at
> https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4UTwsmhw/edit?pli=1
>  
> has a good example of how one-at-a-time scheduling can cause spurious 
> failures.
> 
> And if you're handing the whole set of requests to a scheduler all at 
> once, then you want the scheduler to have access to as many resources as 
> possible so that it has the highest likelihood of being able to satisfy 
> the request given the constraints.

This use case is real and valid, which is why I think there is room for
multiple approaches. For instance the situation you describe can also be
dealt with by just having the cloud stay under-utilized and accepting
that when you get over a certain percentage utilized spurious failures
will happen. We have a similar solution in the ext3 filesystem on Linux.
Don't fill it up, or suffer a huge performance penalty.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New API requirements, review of GCE

On 20 November 2013 13:00, Sean Dague  wrote:
>> As long as the metadataservice doesn't move out :) - that one I think
>> is pretty core and we have no native replacement [configdrive is not a
>> replacement :P].
>
>
> Slightly off tangent thread.
>
> So we recently moved devstack gate to do con fig drive instead of metadata
> service, and life was good (no one really noticed). In what ways is
> configdrive insufficient compared to metadata service? And is that something
> that we should be tackling?

* The metadata service can be trivially updated live - and Heat wants
to use this to get rid of it's own metadata service... whereas config
drive requires unplugging the device, updating the data and replugging
- and thats a bit more invasive.

* Nova baremetal doesn't support config-drive today, and it's an open
question as to whether we ever will - and if we do we can't do the hot
unplug thing, so anyone using it would suffer downtime to update data.

* config drive permits no-control-plane-visibility for instances,
which some secure environments consider to be super important.

So I think we'll have both indefinitely at this point - they serve
overlapping but differing audiences.

We should be testing both.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [UX] Automatically post new threads from AskBot to the list

2013-11-19 Thread Stefano Maffulli

On 11/19/2013 08:19 AM, Julie Pichon wrote:
> I've been thinking about the AskBot UX website [0] and its lack of
> visibility, particularly for new community members.

Indeed, it's one of the drawbacks of splitting groups: information tends
not to flow very well.

> I think it would be valuable to have the first post from new
> conversation threads be posted to the -dev list with the appropriate
> [UX] tag, with a link to AskBot and a request to continue the
> conversation over there. 

If there is a way to do that, I'd say let's give it a try: if it's not
useful we can easily revert back.

I've heard that the UX team will be the first team to dogfood
Storyboard: do you have any idea of any ETA/deadline for when this will
happen?

/stef

-- 
Ask and answer questions on https://ask.openstack.org

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Introducing myself - looking for a IPv6 topology

2013-11-19 Thread Martinx - ジェームズ

Hello Stackers!

I'm Thiago and I'm here on dev list mostly to watch you guys...

Nevertheless, I want to say that I would love to test in deep, the IPv6
support in OpenStack IceHouse.

At at glance, what I'm looking for is more or less specified here, as
follows:

---
I have 1 native IPv6 /48 block and I would like to provide 1 (or more) IPv6
/64 subnet for each tenant (automatically configured when tenant requests a
subnet, tenant does not need to type the IPv6 network address, it will
never overlap).

** I have no plans to support or test NAT66 **

I'm planning to use a dual-stack topology that is described in details here:

http://www.nephos6.com/pdf/OpenStack-on-IPv6.pdf

http://www.nephos6.com/


Also, I would like to give, for example, a range of IPv6 address for each
Instance, instead of just only 1 fixed IPv6 address (based on Instance's
Mac Address / EUI-64) plus a few temporary address, but instead, I would
like to provide ~1000 IPv6 address for each Instance (i.e. DHCPv6 with
"range 2001:db8:1:1::1 2001:db8:1:1::1000" (or 2001:db8:1:1::1-1000). Of
course, assuming that IceHouse have native IPv6 support for IPv6 in first
place...

Desired: Dashboard should focus on DNSaaS, instead of IPs, when listing the
"Instances".

OBS: I really want to achieve a NAT-Less OpenStack Cloud, powered by
IPv6... I know I still need IPv4 (for metadata for example) but, IPv6 is
the way to go, I believe that OpenStack have a great future with IPv6!
---

Best!
Thiago
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing myself - looking for a IPv6 topology

2013-11-19 Thread Martinx - ジェームズ

One more thing...

I'm thinking about the "use case" for "Floating IPs" in a NAT-less IPv6
OpenStack environment.

I can think in two "use cases" in a IPv6-Only Tenant Subnet:

1- the Floating IP might be used to allocate more IPv6 address for an
Instance (since there is no plans for NAT66, I believe it is not desired)
but, instead of allocating it from the "allocation pool", get it from the
tenant subnet directly. This way, the "IPv6 Floating IP" will appear within
the Instance it self, not at the L3 Namespace Router as it is with IPv4
today.

2- we can provide a IPv4 Floating IP address (for a IPv6-Only tenant) and
the L3 Namespace Router will do the NAT46. This way, the "old Internet"
will be able to seamless reach a IPv6-Only network.

What do you guys have in mind / roadmap?!

Cheers!
Thiago

On 19 November 2013 22:57, Martinx - ジェームズ wrote:

> Hello Stackers!
>
> I'm Thiago and I'm here on dev list mostly to watch you guys...
>
> Nevertheless, I want to say that I would love to test in deep, the IPv6
> support in OpenStack IceHouse.
>
> At at glance, what I'm looking for is more or less specified here, as
> follows:
>
> ---
> I have 1 native IPv6 /48 block and I would like to provide 1 (or more)
> IPv6 /64 subnet for each tenant (automatically configured when tenant
> requests a subnet, tenant does not need to type the IPv6 network address,
> it will never overlap).
>
> ** I have no plans to support or test NAT66 **
>
> I'm planning to use a dual-stack topology that is described in details
> here:
>
> http://www.nephos6.com/pdf/OpenStack-on-IPv6.pdf
>
> http://www.nephos6.com/
>
>
> Also, I would like to give, for example, a range of IPv6 address for each
> Instance, instead of just only 1 fixed IPv6 address (based on Instance's
> Mac Address / EUI-64) plus a few temporary address, but instead, I would
> like to provide ~1000 IPv6 address for each Instance (i.e. DHCPv6 with
> "range 2001:db8:1:1::1 2001:db8:1:1::1000" (or 2001:db8:1:1::1-1000). Of
> course, assuming that IceHouse have native IPv6 support for IPv6 in first
> place...
>
> Desired: Dashboard should focus on DNSaaS, instead of IPs, when listing
> the "Instances".
>
> OBS: I really want to achieve a NAT-Less OpenStack Cloud, powered by
> IPv6... I know I still need IPv4 (for metadata for example) but, IPv6 is
> the way to go, I believe that OpenStack have a great future with IPv6!
> ---
>
> Best!
> Thiago
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic] A question about getting IPMI data

2013-11-19 Thread Gao, Fengqian

Re-send again because been rejected by system.

Best Regards,

-fengqian

From: Gao, Fengqian
Sent: Tuesday, November 19, 2013 2:06 PM
To: openstack-dev@lists.openstack.org
Cc: 'Devananda van der Veen'; 'jbjoh...@us.ibm.com'; Lu, Lianhao; Wang, Shane
Subject: [Ironic] A question about getting IPMI data

Hi, all,
As the summit session 
https://etherpad.openstack.org/p/icehouse-summit-ceilometer-hardware-sensors 
said, ironic will expose IPMI data to ceilometer, but I have a couple questions 
here.

1.   What kind of IPMI data will be collected? I assume that power, 
temperature or other sensor data is needed, right?

2.   How do we get all these data?

IIUC, ironic did not involve too much about IPMI for now, only used ipmitool 
and pyghmi module.
Using IPMItool to execute command and parsing the output is easy to understand. 
But seems not a good way for me. The pyghmi only supports 
out-of-band(net-payload) now. So, I am wondering if we can extend the function 
of pyghmi to provide more interfaces?
Be specifically, I think pyghmi can provide in-band IPMI interface including 
IPMB or system interface etc. and allowed to get IPMI data in the OS.

Is all this make sense to you?

Thanks for your response

Best Regards,

-fengqian
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

At yahoo at least 50+ simultaneous will be the common case (maybe we are
special).

Think of what happens on www.yahoo.com say on the olympics, news.yahoo.com
could need 50+ very very quickly (especially if say a gold medal is won by
some famous person). So I wouldn't discount those being the common case
(may not be common for some, but is common for others). In fact any
website with spurious/spikey traffic will have the same desire; so it
might be a target use-case for website like companies (or ones that can't
upfront predict spikes).

Overall though I think what u said about 'don't fill it up' is good
general knowledge. Filling up stuff beyond a certain threshold is
dangerous just in general (one should only push the limits so far before
madness).

On 11/19/13 4:08 PM, "Clint Byrum"  wrote:

>Excerpts from Chris Friesen's message of 2013-11-19 12:18:16 -0800:
>> On 11/19/2013 01:51 PM, Clint Byrum wrote:
>> > Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:
>> >> On 11/19/2013 12:35 PM, Clint Byrum wrote:
>> >>
>> >>> Each scheduler process can own a different set of resources. If they
>> >>> each grab instance requests in a round-robin fashion, then they will
>> >>> fill their resources up in a relatively well balanced way until one
>> >>> scheduler's resources are exhausted. At that time it should bow out
>>of
>> >>> taking new instances. If it can't fit a request in, it should kick
>>the
>> >>> request out for retry on another scheduler.
>> >>>
>> >>> In this way, they only need to be in sync in that they need a way to
>> >>> agree on who owns which resources. A distributed hash table that
>>gets
>> >>> refreshed whenever schedulers come and go would be fine for that.
>> >>
>> >> That has some potential, but at high occupancy you could end up
>>refusing
>> >> to schedule something because no one scheduler has sufficient
>>resources
>> >> even if the cluster as a whole does.
>> >>
>> >
>> > I'm not sure what you mean here. What resource spans multiple compute
>> > hosts?
>> 
>> Imagine the cluster is running close to full occupancy, each scheduler
>> has room for 40 more instances.  Now I come along and issue a single
>> request to boot 50 instances.  The cluster has room for that, but none
>> of the schedulers do.
>> 
>
>You're assuming that all 50 come in at once. That is only one use case
>and not at all the most common.
>
>> >> This gets worse once you start factoring in things like heat and
>> >> instance groups that will want to schedule whole sets of resources
>> >> (instances, IP addresses, network links, cinder volumes, etc.) at
>>once
>> >> with constraints on where they can be placed relative to each other.
>> 
>> > Actually that is rather simple. Such requests have to be serialized
>> > into a work-flow. So if you say "give me 2 instances in 2 different
>> > locations" then you allocate 1 instance, and then another one with
>> > 'not_in_location(1)' as a condition.
>> 
>> Actually, you don't want to serialize it, you want to hand the whole
>>set 
>> of resource requests and constraints to the scheduler all at once.
>> 
>> If you do them one at a time, then early decisions made with
>> less-than-complete knowledge can result in later scheduling requests
>> failing due to being unable to meet constraints, even if there are
>> actually sufficient resources in the cluster.
>> 
>> The "VM ensembles" document at
>> 
>>https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4U
>>Twsmhw/edit?pli=1
>> has a good example of how one-at-a-time scheduling can cause spurious
>> failures.
>> 
>> And if you're handing the whole set of requests to a scheduler all at
>> once, then you want the scheduler to have access to as many resources
>>as 
>> possible so that it has the highest likelihood of being able to satisfy
>> the request given the constraints.
>
>This use case is real and valid, which is why I think there is room for
>multiple approaches. For instance the situation you describe can also be
>dealt with by just having the cloud stay under-utilized and accepting
>that when you get over a certain percentage utilized spurious failures
>will happen. We have a similar solution in the ext3 filesystem on Linux.
>Don't fill it up, or suffer a huge performance penalty.
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder][Glance] OSLO update

2013-11-19 Thread John Griffith

On Mon, Nov 18, 2013 at 3:53 PM, Mark McLoughlin  wrote:
> On Mon, 2013-11-18 at 17:24 +, Duncan Thomas wrote:
>> Random OSLO updates with no list of what changed, what got fixed etc
>> are unlikely to get review attention - doing such a review is
>> extremely difficult. I was -2ing them and asking for more info, but
>> they keep popping up. I'm really not sure what the best way of
>> updating from OSLO is, but this isn't it.
>
> Best practice is to include a list of changes being synced, for example:
>
>   https://review.openstack.org/54660
>
> Every so often, we throw around ideas for automating the generation of
> this changes list - e.g. cinder would have the oslo-incubator commit ID
> for each module stored in a file in git to tell us when it was last
> synced.
>
> Mark.
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Been away on vacation so I'm afraid I'm a bit late on this... but;

I think the point Duncan is bringing up here is that there are some
VERY large and significant patches coming from OSLO pulls.  The DB
patch in particular being over 1K lines of code to a critical portion
of the code is a bit unnerving to try and do a review on.  I realize
that there's a level of trust that goes with the work that's done in
OSLO and synchronizing those changes across the projects, but I think
a few key concerns here are:

1. Doing huge pulls from OSLO like the DB patch here are nearly
impossible to thoroughly review and test.  Over time we learn a lot
about real usage scenarios and the database and tweak things as we go,
so seeing a patch set like this show up is always a bit unnerving and
frankly nobody is overly excited to review it.

2. Given a certain level of *trust* for the work that folks do on the
OSLO side in submitting these patches and new additions, I think some
of the responsibility on the review of the code falls on the OSLO
team.  That being said there is still the issue of how these changes
will impact projects *other* than Nova which I think is sometimes
neglected.  There have been a number of OSLO synchs pushed to Cinder
that fail gating jobs, some get fixed, some get abandoned, but in
either case it shows that there wasn't any testing done with projects
other than Nova (PLEASE note, I'm not referring to this particular
round of patches or calling any patch set out, just stating a
historical fact).

3. We need better documentation in commit messages explaining why the
changes are necessary and what they do for us.  I'm sorry but in my
opinion the answer "it's the latest in OSLO and Nova already has it"
is not enough of an answer in my opinion.  The patches mentioned in
this thread in my opinion met the minimum requirements because they at
least reference the OSLO commit which is great.  In addition I'd like
to see something to address any discovered issues or testing done with
the specific projects these changes are being synced to.

I'm in no way saying I don't want Cinder to play nice with the common
code or to get in line with the way other projects do things but I am
saying that I think we have a ways to go in terms of better
communication here and in terms of OSLO code actually keeping in mind
the entire OpenStack eco-system as opposed to just changes that were
needed/updated in Nova.  Cinder in particular went through some pretty
massive DB re-factoring and changes during Havana and there was a lot
of really good work there but it didn't come without a cost and the
benefits were examined and weighed pretty heavily.  I also think that
some times the indirection introduced by adding some of the
openstack.common code is unnecessary and in some cases makes things
more difficult than they should be.

I'm just not sure that we always do a very good ROI investigation or
risk assessment on changes, and that opinion applies to ALL changes in
OpenStack projects, not OSLO specific or anything else.

All of that being said, a couple of those syncs on the list are
outdated.  We should start by doing a fresh pull for these and if
possible add some better documentation in the commit messages as to
the justification for the patches that would be great.  We can take a
closer look at the changes and the history behind them and try to get
some review progress made here.  Mark mentioned some good ideas
regarding capturing commit ID's from synchronization pulls and I'd
like to look into that a bit as well.

Thanks,
John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] The three API server multi-worker process patches.

2013-11-19 Thread Zhongyue Luo

Carl, Yingjun,

I'm still getting the 2006 error even by configuring idle_interval to 1.

I applied the patch to the RDO havana dist on centos 6.4.

Are there any other options I should be considering such as min/max pool
size or use_tpool?

Thanks.



On Sat, Sep 7, 2013 at 3:33 AM, Baldwin, Carl (HPCS Neutron) <
carl.bald...@hp.com> wrote:

> This pool_recycle parameter is already configurable using the idle_timeout
> configuration variable in neutron.conf.  I tested this with a value of 1
> as suggested and it did get rid of the mysql server gone away messages.
>
> This is a great clue but I think I would like a long-term solution that
> allows the end-user to still configure this like they were before.
>
> I'm currently thinking along the lines of calling something like
> pool.dispose() in each child immediately after it is spawned.  I think
> this should invalidate all of the existing connections so that when a
> connection is checked out of the pool a new one will be created fresh.
>
> Thoughts?  I'll be testing.  Hopefully, I'll have a fixed patch up soon.
>
> Cheers,
> Carl
>
> From:  Yingjun Li 
> Reply-To:  OpenStack Development Mailing List
> 
> Date:  Thursday, September 5, 2013 8:28 PM
> To:  OpenStack Development Mailing List  >
> Subject:  Re: [openstack-dev] [Neutron] The three API server multi-worker
> process patches.
>
>
> +1 for Carl's patch, and i have abandoned my patch..
>
> About the `MySQL server gone away` problem, I fixed it by set
> 'pool_recycle' to 1 in db/api.py.
>
> 在 2013年9月6日星期五，Nachi Ueno 写道：
>
> Hi Folks
>
> We choose https://review.openstack.org/#/c/37131/ <-- This patch to go on.
> We are also discussing in this patch.
>
> Best
> Nachi
>
>
>
> 2013/9/5 Baldwin, Carl (HPCS Neutron) :
> > Brian,
> >
> > As far as I know, no consensus was reached.
> >
> > A problem was discovered that happens when spawning multiple processes.
> > The mysql connection seems to "go away" after between 10-60 seconds in my
> > testing causing a seemingly random API call to fail.  After that, it is
> > okay.  This must be due to some interaction between forking the process
> > and the mysql connection pool.  This needs to be solved but I haven't had
> > the time to look in to it this week.
> >
> > I'm not sure if the other proposal suffers from this problem.
> >
> > Carl
> >
> > On 9/4/13 3:34 PM, "Brian Cline"  wrote:
> >
> >>Was any consensus on this ever reached? It appears both reviews are still
> >>open. I'm partial to review 37131 as it attacks the problem a more
> >>concisely, and, as mentioned, combined the efforts of the two more
> >>effective patches. I would echo Carl's sentiments that it's an easy
> >>review minus the few minor behaviors discussed on the review thread
> >>today.
> >>
> >>We feel very strongly about these making it into Havana -- being confined
> >>to a single neutron-server instance per cluster or region is a huge
> >>bottleneck--essentially the only controller process with massive CPU
> >>churn in environments with constant instance churn, or sudden large
> >>batches of new instance requests.
> >>
> >>In Grizzly, this behavior caused addresses not to be issued to some
> >>instances during boot, due to quantum-server thinking the DHCP agents
> >>timed out and were no longer available, when in reality they were just
> >>backlogged (waiting on quantum-server, it seemed).
> >>
> >>Is it realistically looking like this patch will be cut for h3?
> >>
> >>--
> >>Brian Cline
> >>Software Engineer III, Product Innovation
> >>
> >>SoftLayer, an IBM Company
> >>4849 Alpha Rd, Dallas, TX 75244
> >>214.782.7876 direct  |  bcl...@softlayer.com
> >>
> >>
> >>-Original Message-
> >>From: Baldwin, Carl (HPCS Neutron) [mailto:carl.bald...@hp.com]
> >>Sent: Wednesday, August 28, 2013 3:04 PM
> >>To: Mark McClain
> >>Cc: OpenStack Development Mailing List
> >>Subject: [openstack-dev] [Neutron] The three API server multi-worker
> >>process patches.
> >>
> >>All,
> >>
> >>We've known for a while now that some duplication of work happened with
> >>respect to adding multiple worker processes to the neutron-server.  There
> >>were a few mistakes made which led to three patches being done
> >>independently of each other.
> >>
> >>Can we settle on one and accept it?
> >>
> >>I have changed my patch at the suggestion of one of the other 2 authors,
> >>Peter Feiner, in attempt to find common ground.  It now uses openstack
> >>common code and therefore it is more concise than any of the original
> >>three and should be pretty easy to review.  I'll admit to some bias
> >>toward
> >>my own implementation but most importantly, I would like for one of these
> >>implementations to land and start seeing broad usage in the community
> >>earlier than later.
> >>
> >>Carl Baldwin
> >>
> >>PS Here are the two remaining patches.  The third has been abandoned.
> >>
> >>https://review.openstack.org/#/c/37131/
> >>https://review.openstack.org/#/c/36487/
> >>
> >>
> >>

Re: [openstack-dev] [Neutron][Tempest] Tempest API test for Neutron LBaaS

2013-11-19 Thread Sean Dague

On 11/17/2013 09:55 PM, Eugene Nikanorov wrote:

Hi folks,

I'm working on major change to Neutron LBaaS API, obviously it will
break existing tempest API tests for LBaaS.
What would be the right process to deal with this? I guess I can't just
push fixed tests to tempest because they will not pass against existing
neutron code, and vice versa.


The answer is, you don't. If it's a published API than it has to be 
versioned. Changing an existing API in a non compatible way is not allowed.


So please explain the deprecation and versioning plan for LBaaS, then we 
can probably figure out the right path forward.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Tomoe Sugihara

Hi Edgar,

We had a similar issue and worked around by something like the following
(which I believe similar to what Aaron said):

@@ -45,6 +45,8 @@ SNAT_RULE_PROPERTY = {OS_TENANT_ROUTER_RULE_KEY: SNAT_RULE}
 class MidonetResourceNotFound(q_exc.NotFound):
 message = _('MidoNet %(resource_type)s %(id)s could not be found')

+from eventlet.semaphore import Semaphore
+PORT_ALLOC_SEM = Semaphore()

 class MidonetPluginV2(db_base_plugin_v2.QuantumDbPluginV2,
   l3_db.L3_NAT_db_mixin):
@@ -428,21 +430,31 @@ class MidonetPluginV2(db_base_plugin_v2.QuantumDbPluginV2,
 # set midonet port id to quantum port id and create a DB record.
 port_data['id'] = bridge_port.get_id()

-session = context.session
-with session.begin(subtransactions=True):
-qport = super(MidonetPluginV2, self).create_port(context, port)
-if is_compute_interface:
-# get ip and mac from DB record.
-fixed_ip = qport['fixed_ips'][0]['ip_address']
-mac = qport['mac_address']
-
+qport = None
+with PORT_ALLOC_SEM:
+session = context.session
+with session.begin(subtransactions=True):
+qport = super(MidonetPluginV2, self).create_port(context, port)
+if is_compute_interface:
+# get ip and mac from DB record.
+id = qport['id']
+fixed_ip = qport['fixed_ips'][0]['ip_address']
+mac = qport['mac_address']
+
+if qport and is_compute_interface:
+try:
 # create dhcp host entry under the bridge.
 dhcp_subnets = bridge.get_dhcp_subnets()
 if len(dhcp_subnets) > 0:
 dhcp_subnets[0].add_dhcp_host().ip_addr(fixed_ip)\
.mac_addr(mac)\
.create()
-return qport
+return qport
+except Exception:
+self.delete_port(context, id)
+return None
+else:
+return qport

 def update_port(self, context, id, port):
 """


We are also looking to fix this for upstream icehouce.

Also, I have just submitted a (regression) test for this in tempest:
https://review.openstack.org/#/c/57355

Hope the test makes sense.

Thanks,
Tomoe



On Tue, Nov 19, 2013 at 5:25 AM, Edgar Magana  wrote:

> Developers,
>
> This topic has been discussed before but I do not remember if we have a
> good solution or not.
> Basically, if concurrent API calls are sent to Neutron, all of them are
> sent to the plug-in level where two actions have to be made:
>
> 1. DB transaction – No just for data persistence but also to collect the
> information needed for the next action
> 2. Plug-in back-end implementation – In our case is a call to the python
> library than consequentially calls PLUMgrid REST GW (soon SAL)
>
> For instance:
>
> def create_port(self, context, port):
> with context.session.begin(subtransactions=True):
> # Plugin DB - Port Create and Return port
> port_db = super(NeutronPluginPLUMgridV2,
> self).create_port(context,
>
>  port)
> device_id = port_db["device_id"]
> if port_db["device_owner"] == "network:router_gateway":
> router_db = self._get_router(context, device_id)
> else:
> router_db = None
> try:
> LOG.debug(_("PLUMgrid Library: create_port() called"))
> # Back-end implementation
> self._plumlib.create_port(port_db, router_db)
> except Exception:
> …
>
> The way we have implemented at the plugin-level in Havana (even in
> Grizzly) is that both action are wrapped in the same "transaction" which
> automatically rolls back any operation done to its original state
> protecting mostly the DB of having any inconsistency state or left over
> data if the back-end part fails.=.
> The problem that we are experiencing is when concurrent calls to the same
> API are sent, the number of operation at the plug-in back-end are long
> enough to make the next concurrent API call to get stuck at the DB
> transaction level, which creates a hung state for the Neutron Server to the
> point that all concurrent API calls will fail.
>
> This can be fixed if we include some "locking" system such as calling:
>
> from neutron.common import utile
> …
>
> @utils.synchronized('any-name', external=True)
> def create_port(self, context, port):
> …
>
> Obviously, this will create a serialization of all concurrent calls which
> will ends up in having a really bad performance. Does anyone has a better
> solution?
>
> Thanks,
>
> Edgar
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/li

Re: [openstack-dev] [Nova] New API requirements, review of GCE

Excerpts from Robert Collins's message of 2013-11-19 16:22:41 -0800:
> On 20 November 2013 13:00, Sean Dague  wrote:
> >> As long as the metadataservice doesn't move out :) - that one I think
> >> is pretty core and we have no native replacement [configdrive is not a
> >> replacement :P].
> >
> >
> > Slightly off tangent thread.
> >
> > So we recently moved devstack gate to do con fig drive instead of metadata
> > service, and life was good (no one really noticed). In what ways is
> > configdrive insufficient compared to metadata service? And is that something
> > that we should be tackling?
> 
> * The metadata service can be trivially updated live - and Heat wants
> to use this to get rid of it's own metadata service... whereas config
> drive requires unplugging the device, updating the data and replugging
> - and thats a bit more invasive.
> 

This one is key. Both Trove and Savanna have run into the same
limitation as Heat: networks that cannot reach the API endpoints don't
get to have their API specific Metadata that updates over time. By
putting it in the EC2 metadata service, we can access it via the Neutron
proxy and then Heat/Trove/Savanna can update it later to provide a
control bus for in-instance tools.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing myself - looking for a IPv6 topology

2013-11-19 Thread Kyle Mestery (kmestery)

On Nov 19, 2013, at 7:23 PM, Martinx - ジェームズ  wrote:
> 
> One more thing...
> 
> I'm thinking about the "use case" for "Floating IPs" in a NAT-less IPv6 
> OpenStack environment.
> 
> 
> I can think in two "use cases" in a IPv6-Only Tenant Subnet:
> 
> 1- the Floating IP might be used to allocate more IPv6 address for an 
> Instance (since there is no plans for NAT66, I believe it is not desired) 
> but, instead of allocating it from the "allocation pool", get it from the 
> tenant subnet directly. This way, the "IPv6 Floating IP" will appear within 
> the Instance it self, not at the L3 Namespace Router as it is with IPv4 today.
> 
> 2- we can provide a IPv4 Floating IP address (for a IPv6-Only tenant) and the 
> L3 Namespace Router will do the NAT46. This way, the "old Internet" will be 
> able to seamless reach a IPv6-Only network.
> 
> 
> What do you guys have in mind / roadmap?!
> 
> Cheers!
> Thiago
> 
Hi Thiago:

An IPV6 subteam in Neutron was formed for the Icehouse release. The
team will have weekly meetings in #openstack-meeting-alt on freenode
Thursday's at 2100 UTV. See the meeting page here [1]. If you're planning
to work on IPV6 in any form, it would be great to participate in these and
help shape the IPV6 direction for Neutron.

Thanks, and welcome aboard!
Kyle

[1] https://wiki.openstack.org/wiki/Meetings/Neutron-IPv6-Subteam

> 
> 
> On 19 November 2013 22:57, Martinx - ジェームズ  wrote:
> Hello Stackers!
> 
> I'm Thiago and I'm here on dev list mostly to watch you guys...
> 
> Nevertheless, I want to say that I would love to test in deep, the IPv6 
> support in OpenStack IceHouse.
> 
> At at glance, what I'm looking for is more or less specified here, as follows:




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New API requirements, review of GCE

2013-11-19 Thread Steve Baker

On 11/20/2013 03:54 PM, Clint Byrum wrote:
> Excerpts from Robert Collins's message of 2013-11-19 16:22:41 -0800:
>> On 20 November 2013 13:00, Sean Dague  wrote:
 As long as the metadataservice doesn't move out :) - that one I think
 is pretty core and we have no native replacement [configdrive is not a
 replacement :P].
>>>
>>> Slightly off tangent thread.
>>>
>>> So we recently moved devstack gate to do con fig drive instead of metadata
>>> service, and life was good (no one really noticed). In what ways is
>>> configdrive insufficient compared to metadata service? And is that something
>>> that we should be tackling?
>> * The metadata service can be trivially updated live - and Heat wants
>> to use this to get rid of it's own metadata service... whereas config
>> drive requires unplugging the device, updating the data and replugging
>> - and thats a bit more invasive.
>>
> This one is key. Both Trove and Savanna have run into the same
> limitation as Heat: networks that cannot reach the API endpoints don't
> get to have their API specific Metadata that updates over time. By
> putting it in the EC2 metadata service, we can access it via the Neutron
> proxy and then Heat/Trove/Savanna can update it later to provide a
> control bus for in-instance tools.
>
>
I've got some nova changes which I will resurrect when I get the chance.

They allow nova instance userdata to be updated via the nova API:
https://review.openstack.org/#/c/53732/
https://review.openstack.org/#/c/49971/
https://review.openstack.org/#/c/49973/


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Blueprint: standard specification of guest CPU topology

2013-11-19 Thread Wangpan

Hi Daniel,

Thanks for your help in advance.
I have read your wiki page and it explains this issue very clearly.
But I have a question about the 'technical design', you give us a prototype 
method as below:
def get_guest_cpu_topology(self, inst_type, image, preferred_topology, 
mandatory_topology):
my question is that, how/where we can get these two parameters 
'preferred_topology, mandatory_topology'?
from the nova config file? or get from the hypervisor?

Thanks again.

2013-11-20



Wangpan



发件人："Daniel P. Berrange" 
发送时间：2013-11-19 20:15
主题：[openstack-dev] [Nova] Blueprint: standard specification of guest CPU 
topology
收件人："openstack-dev"
抄送：

For attention of maintainers of Nova virt drivers 

A while back there was a bug requesting the ability to set the CPU 
topology (sockets/cores/threads) for guests explicitly 

   https://bugs.launchpad.net/nova/+bug/1199019 

I countered that setting explicit topology doesn't play well with 
booting images with a variety of flavours with differing vCPU counts. 

This led to the following change which used an image property to 
express maximum constraints on CPU topology (max-sockets/max-cores/ 
max-threads) which the libvirt driver will use to figure out the 
actual topology (sockets/cores/threads) 

  https://review.openstack.org/#/c/56510/ 

I believe this is a prime example of something we must co-ordinate 
across virt drivers to maximise happiness of our users. 

There's a blueprint but I find the description rather hard to 
follow 

  https://blueprints.launchpad.net/nova/+spec/support-libvirt-vcpu-topology 

So I've created a standalone wiki page which I hope describes the 
idea more clearly 

  https://wiki.openstack.org/wiki/VirtDriverGuestCPUTopology 

Launchpad doesn't let me link the URL to the blueprint since I'm not 
the blurprint creator :-( 

Anyway this mail is to solicit input on the proposed standard way to 
express this which is hypervisor portable and the addition of some 
shared code for doing the calculations which virt driver impls can 
just all into rather than re-inventing 

I'm looking for buy-in to the idea from the maintainers of each 
virt driver that this conceptual approach works for them, before 
we go merging anything with the specific impl for libvirt. 

Regards, 
Daniel 
--  
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :| 
|: http://libvirt.org  -o- http://virt-manager.org :| 
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :| 
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :| 

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity

2013-11-19 Thread Tomoe Sugihara

Hi Salvatore, et al,

On Mon, Nov 18, 2013 at 9:19 PM, Salvatore Orlando wrote:

> Hi Yair,
>
> I had in mind of doing something similar. I also registered a tempest
> blueprint for it.
> Basically I think we can assume test machines have access to the Internet,
> but the devstack deployment might not be able to route packets from VMs to
> the Internet.
>
> Being able to ping the external network gateway, which by default is
> 172.24.4.225 is a valuable connectivity test IMHO (and that's your #1 item)
> For items #2 and #3 I'm not sure of your intentions; precisely so far I'm
> not sure if we're adding any coverage to Neutron. I assume you want to
> check servers such as www.google.com are reachable, but the routing from
> the external_gateway_ip to the final destination is beyond's Neutron
> control. DNS resolution might be interesting, but however I think
> resolution of external names is too beyond Neutron's control.
>
> Two more things to consider on external network connectivity tests:
> 1) SNAT can be enabled or not. In this case we need a test that can tell
> us the SRC IP of the host connecting to the public external gateway,
> because I think that if SNAT kicks in, it should be an IP on the ext
> network, otherwise it should be an IP on the internal network. In this case
> we can use netcat to this aim, emulating a web server and use verbose
> output to print the source IP
>
2) When the connection happens from a port associated with a floating IP it
> is important that the SNAT happens with the floating IP address, and not
> with the default SNAT address. This is actually a test which would have
> avoided us a regression in the havana release cycle.
>


As far as I know from the code (I'm new to tempest and might be missing
something), test_network_basic_ops launches a single VM with a floating IP
associated and test is performed by accessing from the tempest host to the
guest VM using floating ip.

So, I have some questions:

- How can we test the internal network connectivity (when the tenant
networks are not accessible from the host, which I believe is the case for
most of the plugins)?

- For external connectivity, how can we test connectivity without floating
ip?
  - should we have another vm and control that from the access VM e.g. by
ssh remote command? or
  - spawn specific VMs which sends traffic upon boot (e.g. metadata server
+ userdata with cloud init installed VM, etc) to public and assert traffics
on the tempest host side?

Thanks,
Tomoe



> Regards,
> Salvatore
>
>
> On 18 November 2013 13:13, Giulio Fidente  wrote:
>
>> On 11/18/2013 11:41 AM, Yair Fried wrote:
>>
>>> I'm editing tempest/scenario/test_network_basic_ops.py for external
>>> connectivity as the TODO listed in its docstring.
>>>
>>> the test cases are for pinging against external ip and url to test
>>> connectivity and dns respectivly.
>>> since default deployement (devstack gate) doesn't have external
>>> connectivity I was thinking on one or all of the following
>>>
>>
>> I think it's a nice thing to have!
>>
>>   2. add fields in tempest.conf for
>>>   * external connectivity = False/True
>>>   * external ip to test against (ie 8.8.8.8)
>>>
>>
>> I like this option.
>>
>> One can easily disable it entirely OR pick a "more relevant" ip address
>> if needed. Seems to me it would give the greatest flexibility.
>> --
>> Giulio Fidente
>> GPG KEY: 08D733BA | IRC: giulivo
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Potential to increase min required libvirt version to 0.9.8 ?

On 20 November 2013 08:02, Daniel P. Berrange  wrote:
> Currently the Nova libvirt driver is declaring that it wants a minimum
> of libvirt 0.9.6.
...
> If there are other distros I've missed which expect to support deployment
> of Icehouse please add them to this list. Hopefully there won't be any
> with libvirt software older than Ubuntu 12.04 LTS
>
>
> The reason I'm asking this now, is that we're working to make the libvirt
> python module a separate tar.gz that can build with multiple libvirt
> versions, and I need to decide how ancient a libvirt we should support
> for it.

Fantastic!!!

The Ubuntu cloud archive
https://wiki.ubuntu.com/ServerTeam/CloudArchive is how OpenStack is
delivered by Canonical for Ubuntu LTS users. So I think you can go
with e.g. 0.9.11 or even 0.9.12 depending on what the Suse folk say.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Plugin and Driver Inclusion Requirements

2013-11-19 Thread Tomoe Sugihara

Hello Neutron team,

>From the "Testing Requirements" section, Tempest is mentioned as a
requirement.
Does that mean all the thirdparty vendors' system should run all the tests
in Tempest?

It might not make perfect sense to test image API for changes in the
neutron code.
So can we define some subset of tests that are relevant to neutron (e.g.
tempest.api.network, tempest.scenario.test_network_* ).

Also, some plugins may not implement full set of APIs defined by Neutron,
(for example. Vendor X hasn't supported floating IP API).
What should be the acceptance criteria for the API compatibility?

Thanks,
Tomoe

On Mon, Nov 18, 2013 at 10:59 PM, Russell Bryant  wrote:

> On 11/18/2013 08:42 AM, Kyle Mestery (kmestery) wrote:
> > Yong, if you read Mark's proposal closely, the third party tests will
> only run when the specific third party code is touched, or when the Jenkins
> user submits code.
> >
> > On Nov 17, 2013, at 11:50 PM, Yongsheng Gong 
> wrote:
> >
> >> For third party testing, I am afraid these tests will make the patch
> merge process much longer since each patch, which even has nothing to do
> with the specific plugins, will triggers the unwanted third party testing
> jobs.
>
> And note that if it's not gating, we do not necessarily have to wait for
> every third party system to report in before merging the patch.
>
> --
> Russell Bryant
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Potential to increase min required libvirt version to 0.9.8 ?

2013-11-19 Thread Tom Fifield

On 20/11/13 14:33, Robert Collins wrote:

On 20 November 2013 08:02, Daniel P. Berrange  wrote:

Currently the Nova libvirt driver is declaring that it wants a minimum
of libvirt 0.9.6.

...

If there are other distros I've missed which expect to support deployment
of Icehouse please add them to this list. Hopefully there won't be any
with libvirt software older than Ubuntu 12.04 LTS


The reason I'm asking this now, is that we're working to make the libvirt
python module a separate tar.gz that can build with multiple libvirt
versions, and I need to decide how ancient a libvirt we should support
for it.


Fantastic!!!

The Ubuntu cloud archive
https://wiki.ubuntu.com/ServerTeam/CloudArchive is how OpenStack is
delivered by Canonical for Ubuntu LTS users. So I think you can go
with e.g. 0.9.11 or even 0.9.12 depending on what the Suse folk say.


Just confirming that the documentation for Ubuntu sets users up with the 
Cloud Archive. In Havana, Libvirt is 1.1.1 and qemu is 1.5.0. I've added 
a row to the table.


Regards,

Tom


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Mike Wilson

I've been thinking about this use case for a DHT-like design, I think I
want to do what other people have alluded to here and try and intercept
problematic requests like this one in some sort of "pre sending to
ring-segment" stage. In this case the "pre-stage" could decide to send this
off to a scheduler that has a more complete view of the world.
Alternatively, don't make a single request for 50 instances, just send 50
requests for one? Is that a viable thing to do for this use case?

-Mike


On Tue, Nov 19, 2013 at 7:03 PM, Joshua Harlow wrote:

> At yahoo at least 50+ simultaneous will be the common case (maybe we are
> special).
>
> Think of what happens on www.yahoo.com say on the olympics, news.yahoo.com
> could need 50+ very very quickly (especially if say a gold medal is won by
> some famous person). So I wouldn't discount those being the common case
> (may not be common for some, but is common for others). In fact any
> website with spurious/spikey traffic will have the same desire; so it
> might be a target use-case for website like companies (or ones that can't
> upfront predict spikes).
>
> Overall though I think what u said about 'don't fill it up' is good
> general knowledge. Filling up stuff beyond a certain threshold is
> dangerous just in general (one should only push the limits so far before
> madness).
>
> On 11/19/13 4:08 PM, "Clint Byrum"  wrote:
>
> >Excerpts from Chris Friesen's message of 2013-11-19 12:18:16 -0800:
> >> On 11/19/2013 01:51 PM, Clint Byrum wrote:
> >> > Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:
> >> >> On 11/19/2013 12:35 PM, Clint Byrum wrote:
> >> >>
> >> >>> Each scheduler process can own a different set of resources. If they
> >> >>> each grab instance requests in a round-robin fashion, then they will
> >> >>> fill their resources up in a relatively well balanced way until one
> >> >>> scheduler's resources are exhausted. At that time it should bow out
> >>of
> >> >>> taking new instances. If it can't fit a request in, it should kick
> >>the
> >> >>> request out for retry on another scheduler.
> >> >>>
> >> >>> In this way, they only need to be in sync in that they need a way to
> >> >>> agree on who owns which resources. A distributed hash table that
> >>gets
> >> >>> refreshed whenever schedulers come and go would be fine for that.
> >> >>
> >> >> That has some potential, but at high occupancy you could end up
> >>refusing
> >> >> to schedule something because no one scheduler has sufficient
> >>resources
> >> >> even if the cluster as a whole does.
> >> >>
> >> >
> >> > I'm not sure what you mean here. What resource spans multiple compute
> >> > hosts?
> >>
> >> Imagine the cluster is running close to full occupancy, each scheduler
> >> has room for 40 more instances.  Now I come along and issue a single
> >> request to boot 50 instances.  The cluster has room for that, but none
> >> of the schedulers do.
> >>
> >
> >You're assuming that all 50 come in at once. That is only one use case
> >and not at all the most common.
> >
> >> >> This gets worse once you start factoring in things like heat and
> >> >> instance groups that will want to schedule whole sets of resources
> >> >> (instances, IP addresses, network links, cinder volumes, etc.) at
> >>once
> >> >> with constraints on where they can be placed relative to each other.
> >>
> >> > Actually that is rather simple. Such requests have to be serialized
> >> > into a work-flow. So if you say "give me 2 instances in 2 different
> >> > locations" then you allocate 1 instance, and then another one with
> >> > 'not_in_location(1)' as a condition.
> >>
> >> Actually, you don't want to serialize it, you want to hand the whole
> >>set
> >> of resource requests and constraints to the scheduler all at once.
> >>
> >> If you do them one at a time, then early decisions made with
> >> less-than-complete knowledge can result in later scheduling requests
> >> failing due to being unable to meet constraints, even if there are
> >> actually sufficient resources in the cluster.
> >>
> >> The "VM ensembles" document at
> >>
> >>
> https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4U
> >>Twsmhw/edit?pli=1
> >> has a good example of how one-at-a-time scheduling can cause spurious
> >> failures.
> >>
> >> And if you're handing the whole set of requests to a scheduler all at
> >> once, then you want the scheduler to have access to as many resources
> >>as
> >> possible so that it has the highest likelihood of being able to satisfy
> >> the request given the constraints.
> >
> >This use case is real and valid, which is why I think there is room for
> >multiple approaches. For instance the situation you describe can also be
> >dealt with by just having the cloud stay under-utilized and accepting
> >that when you get over a certain percentage utilized spurious failures
> >will happen. We have a similar solution in the ext3 filesystem on Linux.
> >Don't fill it up, or suffer a huge perfo

[openstack-dev] OFF TOPIC Re: Image without host-id

2013-11-19 Thread Tom Fifield

On 20/11/13 15:43, Santosh Kumar wrote:

I am following Havana guide for creating three node set up.

Everything has been installed and configured.

However not able to create network for VMs , That's why when creating VM .. 
they come out be without host-id ( VM gets lauch ).

#Nova network-create , is not working for me. Any pointer for the same.



Hi Santosh,

Thank you for supporting OpenStack.

Unfortunately, this discussion is OFF TOPIC on this list.

This list for the developers of OpenStack to discuss development issues
and roadmap.

It is focused on the next release of OpenStack: you should post on this
list if you are a contributor to OpenStack or are very familiar with
OpenStack development and want to discuss very precise topics,
contribution ideas and similar. Do not ask support requests on this
list.

Please move the discussion to the General mailing list
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack or
https://ask.openstack.org

Regards,

Tom

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] OFF TOPIC Re: Image without host-id

On 20 November 2013 17:55, Tom Fifield  wrote:
> Hi Santosh,
>
> Thank you for supporting OpenStack.
...://ask.openstack.org

Thanks for doing this Tom. Can I suggest however, that when replying
one not change the topic? In gmail (which a /huge/ proportion of folk
use nowadays) that breaks the thread and makes it look like the
original off topic post has not been replied to. Which will lead to
multiple replies saying the same thing.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Propose "project story wiki" idea

2013-11-19 Thread Boris Pavlovic

Hi stackers,


Currently what I see is growing amount of interesting projects, that at
least I would like to track. But reading all mailing lists, and reviewing
all patches in all interesting projects to get high level understanding of
what is happing in project now, is quite hard or even impossible task (at
least for me). Especially after 2 weeks vacation =)


The idea of this proposal is that every OpenStack project should have
"story" wiki page. It means to publish every week one short message that
contains most interesting updates for the last week, and high level road
map for future week. So reading this for 10-15 minutes you can see what
changed in project, and get better understanding of high level road map of
the project.

E.g. we start doing this in Rally:
https://wiki.openstack.org/wiki/Rally/Updates


I think that the best way to organize this, is to have person (or few
persons) that will track all changes in project and prepare such updates
each week.



Best regards,
Boris Pavlovic
--
Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Propose "project story wiki" idea

2013-11-19 Thread Michael Bright

+1


On 20 November 2013 06:33, Boris Pavlovic  wrote:

> Hi stackers,
>
>
> Currently what I see is growing amount of interesting projects, that at
> least I would like to track. But reading all mailing lists, and reviewing
> all patches in all interesting projects to get high level understanding of
> what is happing in project now, is quite hard or even impossible task (at
> least for me). Especially after 2 weeks vacation =)
>
>
> The idea of this proposal is that every OpenStack project should have
> "story" wiki page. It means to publish every week one short message that
> contains most interesting updates for the last week, and high level road
> map for future week. So reading this for 10-15 minutes you can see what
> changed in project, and get better understanding of high level road map of
> the project.
>
> E.g. we start doing this in Rally:
> https://wiki.openstack.org/wiki/Rally/Updates
>
>
> I think that the best way to organize this, is to have person (or few
> persons) that will track all changes in project and prepare such updates
> each week.
>
>
>
> Best regards,
> Boris Pavlovic
> --
> Mirantis Inc.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Alex Glikson

Another possible approach could be that only part of the 50 succeeds 
(reported back to the user), and then a retry mechanism at a higher level 
would potentially approach the other partition/scheduler - similar to 
today's retries.

Regards,
Alex

From:   Mike Wilson 
To: "OpenStack Development Mailing List (not for usage questions)" 
, 
Date:   20/11/2013 05:53 AM
Subject:Re: [openstack-dev] [Nova] Does Nova really need an SQL 
database?

I've been thinking about this use case for a DHT-like design, I think I 
want to do what other people have alluded to here and try and intercept 
problematic requests like this one in some sort of "pre sending to 
ring-segment" stage. In this case the "pre-stage" could decide to send 
this off to a scheduler that has a more complete view of the world. 
Alternatively, don't make a single request for 50 instances, just send 50 
requests for one? Is that a viable thing to do for this use case?

-Mike

On Tue, Nov 19, 2013 at 7:03 PM, Joshua Harlow  
wrote:
At yahoo at least 50+ simultaneous will be the common case (maybe we are
special).

Think of what happens on www.yahoo.com say on the olympics, news.yahoo.com
could need 50+ very very quickly (especially if say a gold medal is won by
some famous person). So I wouldn't discount those being the common case
(may not be common for some, but is common for others). In fact any
website with spurious/spikey traffic will have the same desire; so it
might be a target use-case for website like companies (or ones that can't
upfront predict spikes).

Overall though I think what u said about 'don't fill it up' is good
general knowledge. Filling up stuff beyond a certain threshold is
dangerous just in general (one should only push the limits so far before
madness).

On 11/19/13 4:08 PM, "Clint Byrum"  wrote:

>Excerpts from Chris Friesen's message of 2013-11-19 12:18:16 -0800:
>> On 11/19/2013 01:51 PM, Clint Byrum wrote:
>> > Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:
>> >> On 11/19/2013 12:35 PM, Clint Byrum wrote:
>> >>
>> >>> Each scheduler process can own a different set of resources. If 
they
>> >>> each grab instance requests in a round-robin fashion, then they 
will
>> >>> fill their resources up in a relatively well balanced way until one
>> >>> scheduler's resources are exhausted. At that time it should bow out
>>of
>> >>> taking new instances. If it can't fit a request in, it should kick
>>the
>> >>> request out for retry on another scheduler.
>> >>>
>> >>> In this way, they only need to be in sync in that they need a way 
to
>> >>> agree on who owns which resources. A distributed hash table that
>>gets
>> >>> refreshed whenever schedulers come and go would be fine for that.
>> >>
>> >> That has some potential, but at high occupancy you could end up
>>refusing
>> >> to schedule something because no one scheduler has sufficient
>>resources
>> >> even if the cluster as a whole does.
>> >>
>> >
>> > I'm not sure what you mean here. What resource spans multiple compute
>> > hosts?
>>
>> Imagine the cluster is running close to full occupancy, each scheduler
>> has room for 40 more instances.  Now I come along and issue a single
>> request to boot 50 instances.  The cluster has room for that, but none
>> of the schedulers do.
>>
>
>You're assuming that all 50 come in at once. That is only one use case
>and not at all the most common.
>
>> >> This gets worse once you start factoring in things like heat and
>> >> instance groups that will want to schedule whole sets of resources
>> >> (instances, IP addresses, network links, cinder volumes, etc.) at
>>once
>> >> with constraints on where they can be placed relative to each other.
>>
>> > Actually that is rather simple. Such requests have to be serialized
>> > into a work-flow. So if you say "give me 2 instances in 2 different
>> > locations" then you allocate 1 instance, and then another one with
>> > 'not_in_location(1)' as a condition.
>>
>> Actually, you don't want to serialize it, you want to hand the whole
>>set
>> of resource requests and constraints to the scheduler all at once.
>>
>> If you do them one at a time, then early decisions made with
>> less-than-complete knowledge can result in later scheduling requests
>> failing due to being unable to meet constraints, even if there are
>> actually sufficient resources in the cluster.
>>
>> The "VM ensembles" document at
>>
>>
https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4U
>>Twsmhw/edit?pli=1
>> has a good example of how one-at-a-time scheduling can cause spurious
>> failures.
>>
>> And if you're handing the whole set of requests to a scheduler all at
>> once, then you want the scheduler to have access to as many resources
>>as
>> possible so that it has the highest likelihood of being able to satisfy
>> the request given the constraints.
>
>This use case is real and valid, which is why I think there is room for
>multiple approaches. For instance the situation

Re: [openstack-dev] [Neutron][LBaaS] SSL Termination write-up

2013-11-19 Thread Vijay Venkatachalam

Hi Eugene,

The proposal is simple, create a separate resource called 
certificate (which will also be handled by Neutron+LBaaS plugin) rather than 
having them in the VIP. It will maintain the original thought of security, the 
certificate confidential parameters will  be transient and the certificate will 
be directly sent to the device/driver and will not be stored. This is done by 
having VIP as a property in certificate.

Thanks,
Vijay V.

From: Eugene Nikanorov [mailto:enikano...@mirantis.com]
Sent: Wednesday, November 20, 2013 1:59 AM
To: Vijay Venkatachalam
Cc: Samuel Bercovici; Avishay Balderman; openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Neutron][LBaaS] SSL Termination write-up

Hi Vijay,

Thanks for working on this. As was discussed at the summit, immediate solution 
seems to be passing certificates via transient fields in Vip object, which will 
avoid the need for certificate management (incl. storing them).
If certificate management is concerned then I agree that it needs to be a 
separate specialized service.

> My thought was to have independent certificate resource with VIP uuid as one 
> of the properties. VIP is already created and
> will help to identify the driver/device. The VIP property can be depreciated 
> in the long term.
I think it's a little bit forward looking at this moment, also I think that 
certificates are not specific for load balancing.
Transient fields could help here too: client could pass certificate Id and 
driver of corresponding device will communicate with external service fetching 
corresponding certificate.

Thanks,
Eugene.

On Tue, Nov 19, 2013 at 5:48 PM, Vijay Venkatachalam 
mailto:vijay.venkatacha...@citrix.com>> wrote:
Hi Sam, Eugene, & Avishay, etal,

Today I spent some time to create a write-up for SSL 
Termination not exactly design doc. Please share your comments!

https://docs.google.com/document/d/1tFOrIa10lKr0xQyLVGsVfXr29NQBq2nYTvMkMJ_inbo/edit

Would like comments/discussion especially on the following note:

SSL Termination requires certificate management. The ideal way is to handle 
this via an independent IAM service. This would take time to implement so the 
thought was to add the certificate details in VIP resource and send them 
directly to device. Basically don't store the certificate key in the DB there 
by avoiding security concerns of maintaining certificates in controller.

I would expect the certificates to become an independent resource in future 
thereby causing backward compatibility issues.

Any ideas how to achieve this?

My thought was to have independent certificate resource with VIP uuid as one of 
the properties. VIP is already created and will help to identify the 
driver/device. The VIP property can be depreciated in the long term.
Thanks,
Vijay V.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Top Gate Bugs

2013-11-19 Thread Joe Gordon

Hi All,

As many of you have noticed the gate has been in very bad shape over the
past few days.  Here is a list of some of the top open bugs (without
pending patches, and many recent hits) that we are hitting.  Gate won't be
stable, and it will be hard to get your code merged, until we fix these
bugs.

1) https://bugs.launchpad.net/bugs/1251920
 nova
468 Hits
2) https://bugs.launchpad.net/bugs/1251784
 neutron, Nova
 328 Hits
3) https://bugs.launchpad.net/bugs/1249065
 neutron
  122 hits
4) https://bugs.launchpad.net/bugs/1251448
 neutron
65 Hits

Raw Data:


Note: If a bug has any hits for anything besides failure, it means the
fingerprint isn't perfect.

Elastic recheck known issues
 Bug: https://bugs.launchpad.net/bugs/1251920 => message:"assertionerror:
console output was empty" AND filename:"console.html" Title: Tempest
failures due to failure to return console logs from an instance Project:
Status nova: Confirmed Hits FAILURE: 468 Bug:
https://bugs.launchpad.net/bugs/1251784 => message:"Connection to neutron
failed: Maximum attempts reached" AND filename:"logs/screen-n-cpu.txt"
Title: nova+neutron scheduling error: Connection to neutron failed: Maximum
attempts reached Project: Status neutron: New nova: New Hits FAILURE: 328
UNSTABLE: 13 SUCCESS: 275 Bug: https://bugs.launchpad.net/bugs/1240256 =>
message:" 503" AND filename:"logs/syslog.txt" AND
syslog_program:"proxy-server" Title: swift proxy-server returning 503
during tempest run Project: Status openstack-ci: Incomplete swift: New
tempest: New Hits FAILURE: 136 SUCCESS: 83
Pending Patch Bug: https://bugs.launchpad.net/bugs/1249065 => message:"No
nw_info cache associated with instance" AND
filename:"logs/screen-n-api.txt" Title: Tempest failure:
tempest/scenario/test_snapshot_pattern.py Project: Status neutron: New
nova: Confirmed Hits FAILURE: 122 Bug:
https://bugs.launchpad.net/bugs/1252514 => message:"Got error from Swift:
put_object" AND filename:"logs/screen-g-api.txt" Title: glance doesn't
recover if Swift returns an error Project: Status devstack: New glance: New
swift: New Hits FAILURE: 95
Pending Patch Bug: https://bugs.launchpad.net/bugs/1244255 =>
message:"NovaException: Unexpected vif_type=binding_failed" AND
filename:"logs/screen-n-cpu.txt" Title: binding_failed because of l2 agent
assumed down Project: Status neutron: Fix Committed Hits FAILURE: 92
SUCCESS: 29 Bug: https://bugs.launchpad.net/bugs/1251448 => message:"
possible networks found, use a Network ID to be more specific. (HTTP 400)"
AND filename:"console.html" Title: BadRequest: Multiple possible networks
found, use a Network ID to be more specific. Project: Status neutron: New
Hits FAILURE: 65 Bug: https://bugs.launchpad.net/bugs/1239856 =>
message:"tempest/services" AND message:"/images_client.py" AND
message:"wait_for_image_status" AND filename:"console.html" Title:
"TimeoutException: Request timed out" on
tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestXML
Project: Status glance: New Hits FAILURE: 62 Bug:
https://bugs.launchpad.net/bugs/1235435 => message:"One or more ports have
an IP allocation from this subnet" AND message:" SubnetInUse: Unable to
complete operation on subnet" AND filename:"logs/screen-q-svc.txt" Title:
'SubnetInUse: Unable to complete operation on subnet UUID. One or more
ports have an IP allocation from this subnet.' Project: Status neutron:
Incomplete nova: Fix Committed tempest: New Hits FAILURE: 48 Bug:
https://bugs.launchpad.net/bugs/1224001 =>
message:"tempest.scenario.test_network_basic_ops AssertionError: Timed out
waiting for" AND filename:"console.html" Title: test_network_basic_ops
fails waiting for network to become available Project: Status neutron: In
Progress swift: Invalid tempest: Invalid Hits FAILURE: 42 Bug:
https://bugs.launchpad.net/bugs/1218391 => message:"Cannot 'createImage'"
AND filename:"console.html" Title:
tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestXML.test_delete_image_that_is_not_yet_active
spurious failure Project: Status nova: Confirmed swift: Confirmed tempest:
Confirmed Hits FAILURE: 25



best,
Joe Gordon
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Propose "project story wiki" idea

2013-11-19 Thread Alex Glikson

I think the idea in general is very good (I would be a heavy consumer of 
such a thing myself). I am not sure how sustainable is to do it manually 
though, especially for larger projects. 
Maybe there is a reasonable way to automate this.. For example, if we 
could generate a 'dashboard' for each project, aggregating weekly progress 
with patches and blueprints, as well as mailing list threads (potentially 
grouped by tags associated with projects). Maybe stackalytics could be 
used as a platform..

Regards,
Alex




From:   Boris Pavlovic 
To: OpenStack Development Mailing List 
, 
Date:   20/11/2013 07:34 AM
Subject:[openstack-dev] Propose "project story wiki" idea



Hi stackers,


Currently what I see is growing amount of interesting projects, that at 
least I would like to track. But reading all mailing lists, and reviewing 
all patches in all interesting projects to get high level understanding of 
what is happing in project now, is quite hard or even impossible task (at 
least for me). Especially after 2 weeks vacation =)


The idea of this proposal is that every OpenStack project should have 
"story" wiki page. It means to publish every week one short message that 
contains most interesting updates for the last week, and high level road 
map for future week. So reading this for 10-15 minutes you can see what 
changed in project, and get better understanding of high level road map of 
the project.

E.g. we start doing this in Rally: 
https://wiki.openstack.org/wiki/Rally/Updates


I think that the best way to organize this, is to have person (or few 
persons) that will track all changes in project and prepare such updates 
each week. 



Best regards,
Boris Pavlovic
--
Mirantis Inc. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Propose "project story wiki" idea

2013-11-19 Thread Tom Fifield

Love the idea Boris - a nice read :)

Regards,

Tom

On 20/11/13 16:45, Michael Bright wrote:

+1


On 20 November 2013 06:33, Boris Pavlovic mailto:bpavlo...@mirantis.com>> wrote:

Hi stackers,


Currently what I see is growing amount of interesting projects, that
at least I would like to track. But reading all mailing lists, and
reviewing all patches in all interesting projects to get high level
understanding of what is happing in project now, is quite hard or
even impossible task (at least for me). Especially after 2 weeks
vacation =)


The idea of this proposal is that every OpenStack project should
have "story" wiki page. It means to publish every week one short
message that contains most interesting updates for the last week,
and high level road map for future week. So reading this for 10-15
minutes you can see what changed in project, and get better
understanding of high level road map of the project.

E.g. we start doing this in Rally:
https://wiki.openstack.org/wiki/Rally/Updates


I think that the best way to organize this, is to have person (or
few persons) that will track all changes in project and prepare such
updates each week.



Best regards,
Boris Pavlovic
--
Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] I18n team meeting tomorrow

2013-11-19 Thread Ying Chun Guo


Hi, all

There will be OpenStack I18n team meeting at UTC on Thursday (Nov 21th)
in IRC channel #openstack-meeting.
The time, we use Asia/America friendly time. Welcome to join the meeting.

This is the first I18n meeting after Icehouse design summit.
Summarize of the design summit in I18n areas can be found here:
http://lists.openstack.org/pipermail/openstack-i18n/2013-November/000294.html.

This time, we will cover following topics this time:

* Horizon translation update
* I18n team structure discussion
* Reorganize wiki page
* Blueprints & bugs tracking
* Plans in Icehouse cycle
* Open discussion

For more details, please look into
https://wiki.openstack.org/wiki/Meetings/I18nTeamMeeting.

You can also contact us through IRC channel #openstack-translation, or
mailing address: openstack-i...@list.openstack.org.
Please refer to our wiki page for more details in the I18n area:
https://wiki.openstack.org/wiki/I18nTeam

Regards
Ying Chun Guo (Daisy)___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [openstack][nova][social-apects] Social aspects shouldn't impact on dev process

2013-11-19 Thread Boris Pavlovic

Hello Stackers,

I faced some social problems in community.

We started working on purge engine for DB (before HK summit)

This is very important, because at this moment we don't have any working
way to purge DB... so admins should make it by hand.


And we made this BP (in october)
https://blueprints.launchpad.net/nova/+spec/db-purge-engine

And made patch that makes this work.
But only because our BP wasn't approved we got -2 from Joe Gordon.
(https://review.openstack.org/#/c/51523/ ) And there was long discussion to
remove this -2.

And now after summit David Ripton made the similar BP (probably he didn't
know):
https://blueprints.launchpad.net/nova/+spec/db-purge2
That is already approved by Joe Gordon. (that already know that we are
working on same problem)

Why?

(btw question about Purge Engine was raised by me on the summit and
community accepted that)

Joe could you explain please?


Best regards,
Boris Pavlovic
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

2013-11-19 Thread Thomas Spatzier

Excerpts from Steve Baker's message on 19.11.2013 21:40:54:
> From: Steve Baker 
> To: openstack-dev@lists.openstack.org,
> Date: 19.11.2013 21:43
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>

> I think there needs to a CM tool specific agent delivered to the server
> which os-collect-config invokes. This agent will transform the config
> data (input values, CM script, CM specific specialness) to a CM tool
> invocation.
>
> How to define and deliver this agent is the challenge. Some options are:
> 1) install it as part of the image customization/bootstrapping (golden
> images or cloud-init)
> 2) define a (mustache?) template in the SoftwareConfig which
> os-collect-config transforms into the agent script, which
> os-collect-config then executes
> 3) a CM tool specific implementation of SoftwareApplier builds and
> delivers a complete agent to os-collect-config which executes it
>
> I may be leaning towards 3) at the moment. Hopefully any agent can be
> generated with a sufficiently sophisticated base SoftwareApplier type,
> plus maybe some richer intrinsic functions.

This is good summary of options; about the same we had in mind. And we were
also leaning towards 3. Probably the approach we would take is to get a
SoftwareApplier running for one CM tool (e.g. Chef), then look at another
tool (base shell scripts), and then see what the generic parts art that can
be factored into a base class.

> >> The POC I'm working on is actually backed by a REST API which does
dumb
> >> (but structured) storage of SoftwareConfig and SoftwareApplier
entities.
> >> This has some interesting implications for managing SoftwareConfig
> >> resources outside the context of the stack which uses them, but lets
not
> >> worry too much about that *yet*.
> > Sounds good. We are also defining some blueprints to break down the
overall
> > software config topic. We plan to share them later this week, and then
we
> > can consolidate with your plans and see how we can best join forces.
> >
> >
> At this point it would be very helpful to spec out how specific CM tools
> are invoked with given inputs, script, and CM tool specific options.

That's our plan; and we would probably start with scripts and chef.

>
> Maybe if you start with shell scripts, cfn-init and chef then we can all
> contribute other CM tools like os-config-applier, puppet, ansible,
> saltstack.
>
> Hopefully by then my POC will at least be able to create resources, if
> not deliver some data to servers.

We've been thinking about getting metadata to the in-instance parts on the
server and whether the resources you are building can serve the purpose.
I.e. pass and endpoint to the SoftwareConfig resources to the instance and
let the instance query the metadata from the resource. Sounds like this is
what you had in mind, so that would be a good point for integrating the
work. In the meantime, we can think of some shortcuts.

>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] SSL Termination write-up

2013-11-19 Thread Andrew Hutchings

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 19/11/13 16:33, Clint Byrum wrote:
> Excerpts from Vijay Venkatachalam's message of 2013-11-19 05:48:43
> -0800:
>> Hi Sam, Eugene, & Avishay, etal,
>> 
>> Today I spent some time to create a write-up for SSL Termination
>> not exactly design doc. Please share your comments!
>> 
>> https://docs.google.com/document/d/1tFOrIa10lKr0xQyLVGsVfXr29NQBq2nYTvMkMJ_inbo/edit
>>
>>
>> 
Would like comments/discussion especially on the following note:
>> 
>> SSL Termination requires certificate management. The ideal way is
>> to handle this via an independent IAM service. This would take
>> time to implement so the thought was to add the certificate
>> details in VIP resource and send them directly to device.
>> Basically don't store the certificate key in the DB there by
>> avoiding security concerns of maintaining certificates in
>> controller.
>> 
>> I would expect the certificates to become an independent resource
>> in future thereby causing backward compatibility issues.
>> 
> 
> Perhaps Barbican can be leveraged for this, it seems that it was

Indeed, the planned Libra solution for this is to use Barbican.  We
did some investigation into this when Barbican was a very new project
but unfortunately it wasn't quite ready at the time.  We will be
looking into this again in the coming months.

Kind Regards
- -- 
Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSjGpOAAoJEJfFW8uxApB2VQQH/RdqegsTw5TsqIeARRwv4XPP
9VgpbuPuUmENJ2zBa9LPM1+ytY61LQRXUpLtYMxN3iefndgACvEgJ/hfVeiu7F6T
YDQyu3K6XtSJ4E1bizJQyyU0Q1bXzsb/3J/Nh3E5RRZ/vQCRAvnihiehDr8R002e
YdzOIorMP45kmHg77CCq9wKEffUDGzr/vqa+5xERSOCkt3TvB7T8BW+f9DipVH/G
9pbTvkqWsHB0ppytgTM7rv1P7ltAvSDPdC6ALh7q/oQU7QAMr1XxZSEbJCYzXLqb
WAt1QYC0EddxKArQmPA5LmMoOW02c76sJqKHeZbrZEERB4nxbZMPvjAKf4F+8Mw=
=uJGy
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Configuration validation

2013-11-19 Thread Oleg Gelbukh

Tracy,

I've created etherpad with the summary:
https://etherpad.openstack.org/p/w5BwMtCG6z

--
Best regards,
Oleg Gelbukh


On Mon, Nov 18, 2013 at 11:17 PM, Tracy Jones  wrote:

> Oleg - this is great!  I tried to find you on IRC to recommend we put this
> on a etherpad so several of us can collaborate together on requirements and
> brainstorming.
>
> On Nov 18, 2013, at 10:54 AM, Oleg Gelbukh  wrote:
>
> Hi,
>
> I summarized a list of requirements to the config validation framework
> from this thread and other sources, including IRC discussions so far:
>
> https://gist.github.com/ogelbukh/7533029
>
> Seems like we have 2 work items here, one being extension which adds more
> complex flags types to Oslo.config, and the other is standalone service for
> stateful validation of cfg across services. We're working on design for
> such service in frame of Rubick project.
>
> I'd really appreciate any help with prioritization of requirements from
> the list above.
>
> --
> Best regards,
> Oleg Gelbukh
> Mirantis Labs
> > To be fair, we test only the subset that is set via devstack on the
> > stable side. That should be a common subset, but it is far from fully
> > comprehensive. Nova has over 600 config variables, so additional tooling
> > here would be goodness.
>
> I'm surely not arguing against additional testing of config stuff, I'm
> just saying I'm not sure there's an upgrade impact here. What we care
> about across upgrades is that the conf stays legit. A set of offline
> tests that look at the config shouldn't have anything useful to verify
> or report, other than just "this thingy is now deprecated, beware!"
>
> If we care about validating that reviewers didn't let some config
> element be removed that wasn't already deprecated, that's something that
> needs visibility into the two ends of the upgrade. I would think grenade
> would be the place to do this since it has both hashes, and I definitely
> think that could be instrumented easily. However, I don't think that has
> much overlap with the config checker thing that was discussed at summit,
> simply because it requires visibility into two trees.
>
> --Dan
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>  ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][LBaaS] Subteam meeting on Thursday, 21

2013-11-19 Thread Eugene Nikanorov

Hi folks,

Let's meet on #openstack-meeting on Thrsday, 21, at 14-00 UTC
We'll discuss current progress and design of some of proposed features.

Thanks,
Eugene.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config

2013-11-19 Thread Flavio Percoco

On 18/11/13 16:46 +, Mark McLoughlin wrote:

On Mon, 2013-11-18 at 17:37 +0100, Julien Danjou wrote:

On Mon, Nov 18 2013, Mark McLoughlin wrote:

> I'm struggling to care about this until I have some insight into why
> it's important. And it's a bit frustrating to have to guess the
> rationale for this. Like commit messages, blueprints should be as much
> about the why as the what.

Sure. I ought to think that having an application that wants to leverage
oslo.messaging but is not using oslo.config and is retrieving its
parameter from another way is a good enough argument.

It's a theoretical benefit versus the very practical "design an API for
the use cases that are actually important to OpenStack projects".

I agree with Mark here. I don't think the discussion here should be
around whether not depending on oslo.config is the right thing. As far
as I can see, we all agree on the fact that libraries should have few
dependencies and they shouldn't force anything on the application
using them.

That being said, I think the discussion here should go around whether
this is the right time to make the change or not. The benefit of doing
it now is that we haven't released oslo.messaging yet, which means
there are no *strong* API requirements. However, I'd like us to
consider how that will help with the migration of existing projects
from oslo-rpc to oslo.messaging.

I think this is - or at least should be - our priority right now as
far as oslo.messaging is concerned.

Cheers,
FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

On Mon, Nov 18, 2013 at 07:10:53PM -0500, Krishna Raman wrote:
> We should probably meet on IRC or conf. call and discuss the suggested 
> architecture, open questions and REST API.
> 
> I have set up a doodle poll at http://doodle.com/w7y5qcdvq9i36757 to gather a 
> times when we can meet.
> Please sign up or let me know if none of the time slots works for you.

The time slots are pretty west coast centric there. Only the 8am/9am slots
work for people in Europe really.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reg : Security groups implementation using openflows in quantum ovs plugin

2013-11-19 Thread Kanthi P

Hi All,

Thanks for the response!
Amir,Mike: Is your implementation being done according to ML2 plugin

Regards,
Kanthi


On Tue, Nov 19, 2013 at 1:43 AM, Mike Wilson  wrote:

> Hi Kanthi,
>
> Just to reiterate what Kyle said, we do have an internal implementation
> using flows that looks very similar to security groups. Jun Park was the
> guy that wrote this and is looking to get it upstreamed. I think he'll be
> back in the office late next week. I'll point him to this thread when he's
> back.
>
> -Mike
>
>
> On Mon, Nov 18, 2013 at 3:39 PM, Kyle Mestery (kmestery) <
> kmest...@cisco.com> wrote:
>
>> On Nov 18, 2013, at 4:26 PM, Kanthi P  wrote:
>> > Hi All,
>> >
>> > We are planning to implement quantum security groups using openflows
>> for ovs plugin instead of iptables which is the case now.
>> >
>> > Doing so we can avoid the extra linux bridge which is connected between
>> the vnet device and the ovs bridge, which is given as a work around since
>> ovs bridge is not compatible with iptables.
>> >
>> > We are planning to create a blueprint and work on it. Could you please
>> share your views on this
>> >
>> Hi Kanthi:
>>
>> Overall, this idea is interesting and removing those extra bridges would
>> certainly be nice. Some people at Bluehost gave a talk at the Summit [1] in
>> which they explained they have done something similar, you may want to
>> reach out to them since they have code for this internally already.
>>
>> The OVS plugin is in feature freeze during Icehouse, and will be
>> deprecated in favor of ML2 [2] at the end of Icehouse. I would advise you
>> to retarget your work at ML2 when running with the OVS agent instead. The
>> Neutron team will not accept new features into the OVS plugin anymore.
>>
>> Thanks,
>> Kyle
>>
>> [1]
>> http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/towards-truly-open-and-commoditized-software-defined-networks-in-openstack
>> [2] https://wiki.openstack.org/wiki/Neutron/ML2
>>
>> > Thanks,
>> > Kanthi
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Thierry Carrez

Russell Bryant wrote:
> My view of the outcome of the session was not "it *will* be a new
> service".  Instead, it was, "we *think* it should be a new service, but
> let's do some more investigation to decide for sure".
> 
> The action item from the session was to go off and come up with a
> proposal for what a new service would look like.  In particular, we
> needed a proposal for what the API would look like.  With that in hand,
> we need to come back and ask the question again of whether a new service
> is the right answer.

+1

I can see how a separate service can be a good way to avoid making Nova
support container-specific use cases and make it even bigger than it is.
However if you end up duplicating most of the code, I'm not sure there
would be any net gain.

I'm not talking about the base service infrastructure and the scheduler
(which are well-known duplication already) but more around specific nova
features (metadata/config drive, networking, image handling, etc.) that
would apply to VMs and containers alike.

So it would be great to come out of this first round of analysis with a
plan detailing how different (and how similar) from nova that new
service would be, and how much code duplication is to be expected.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers