Re: [openstack-dev] [novalcient]Proposal for removing 'bash_completion'

[Alexei Kornienko]

Hello Lingxian,

Actually I'm planning to remove both of them.

I'm working on integration of argcomplete 
(https://github.com/kislyuk/argcomplete) library to openstack clients:

https://review.openstack.org/#/c/65863/
https://review.openstack.org/#/c/65864/
https://review.openstack.org/#/c/65865/

With such approach we'll remove both commands and simplify competition 
implementation.


Regards,
Alexei Kornienko

On 01/15/2014 08:51 AM, Lingxian Kong wrote:
Now, novaclient supports 'bash-completion' and 'bash_completion' 
commands at the same time, after I checked the commit history, I found 
that the '_add_bash_completion_subparser' function and 
'do_bash_completion' function were added in at 2011/12/16 by Rick 
Harris, with the Change-Id: If882f7a822ef6b1e58666b3af6f7166ab0a230fe.


then, there are two parsers with the same usage, but with different 
commands, one is 'bash-completion', and the other is 
'bash_completion', I think it's not what the author supposed to do.


and by the way, there are two things should be noticed:
1. there is only unittest for 'bash-completion'.
2. the command 'bash_completion' can't be found with 'nova help', 
which means it's not seen by users.


so, I purpose removing 'bash_completion'. Thoughts?

--
*---*
*Lingxian Kong*
Huawei Technologies Co.,LTD.
IT Product Line CloudOS PDU
China, Xi'an
Mobile: +86-18602962792
Email: konglingx...@huawei.com <mailto:konglingx...@huawei.com>; 
anlin.k...@gmail.com <mailto:anlin.k...@gmail.com>



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Disk Eraser

[Alexei Kornienko]

If you are working on linux system following can help you:

dd if=/dev/urandom of=/dev/sda bs=4k

:)
Best Regards,

On 01/15/2014 04:31 PM, Alan Kavanagh wrote:


Hi fellow OpenStackers

Does anyone have any recommendations on open source tools for disk 
erasure/data destruction software. I have so far looked at DBAN and 
disk scrubber and was wondering if ironic team have some better 
recommendations?


BR

Alan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

>I did notice, however, that neutronclient is
conspicuously absent from the Work Items in the blueprint's Whiteboard.
It will surely be added later. We already working on several things in
parallel and we will add neutronclient soon.

>I would love to see a bit more detail on the structure of the lib(s), the
blueprint really doesn't discuss the design/organization/intended API of
the libs.  For example, I would hope the distinction between the various
layers of a client stack don't get lost, i.e. not mixing the low-level REST
API bits with the higher-level CLI parsers and decorators.
>Does the long-term goals include a common caching layer?

Distinction between client layers won't get lost and would only be
improved. My basic idea is the following:
1) Transport layer would handle all transport related stuff - HTTP, JSON
encoding, auth, caching, etc.
2) Model layer (Resource classes, BaseManager, etc.) will handle data
representation, validation
3) API layer will handle all project specific stuff - url mapping, etc.
(This will be imported to use client in other applications)
4) Cli level will handle all stuff related to cli mapping - argparse,
argcomplete, etc.

>I believe the current effort referenced by the blueprint is focusing on
moving existing code into the incubator for reuse, to make it easier to
restructure later. Alexei, do I have that correct?
You are right. The first thing we do is try to make all clients look/work
in similar way. After we'll continue our work with improving overall
structure.




2014/1/16 Noorul Islam K M 

> Doug Hellmann  writes:
>
> > Several people have mentioned to me that they are interested in, or
> > actively working on, code related to a "common" client library --
> something
> > meant to be reused directly as a basis for creating a common library for
> > all of the openstack clients to use. There's a blueprint [1] in oslo,
> and I
> > believe the keystone devs and unified CLI teams are probably interested
> in
> > ensuring that the resulting API ends up meeting all of our various
> > requirements.
> >
> > If you're interested in this effort, please subscribe to the blueprint
> and
> > use that to coordinate efforts so we don't produce more than one common
> > library. ;-)
> >
>
> Solum is already using it https://review.openstack.org/#/c/58067/
>
> I would love to watch this space.
>
> Regards,
> Noorul
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

On 01/16/2014 05:25 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:07 AM, Joe Gordon > wrote:






On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:



On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah
mailto:chmo...@enovance.com>> wrote:



On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones mailto:c...@tenshu.net>> wrote:

Once a common library is in place, is there any intention to
(or resistance against) collapsing the clients into a single
project or even a single command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient


After speaking with people working on OSC and looking at the code
base in depth; I don't think this addresses what Chris is
implying: OSC wraps the individual CLIs built by each project
today, instead of the inverse: a common backend that the
individual CLIs can wrap - the latter is an important distinction
as currently, building a single binary install of OSC for say,
Windows is difficult given the dependency tree incurred by each
of the wrapped CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes /
functions / methods means that the layer that presents a
consistent user interface (OSC) to the user is made more complex
juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a
developer of user interfaces (CLIs, Applications, Web apps
(horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(...)

def my_command(cli):
compute = Compute(Auth(cli.tentant..., connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or
wanted to keep their specific CLIs, they would be able to use a
not "least common denominator" back end (each service can have a
rich common.api.compute.py  or
api.compute/client.py and extend where needed. However tools like
horizon / openstackclient can choose not to leverage the "power
user/operator/admin" components and present a simplified user
interface.

I'm working on a wiki page + blueprint to brainstorm how we could
accomplish this based off of what work is in flight today (see
doug's linked blueprint) and sussing out a layout / API strawman
for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each
sub-project delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper
unit testing


I am happy to see this work being done, there is definitely a lot of 
work to be done on the clients.


This blueprint sounds like its still being fleshed out, so I am 
wondering what the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z


Those patches mainly sync cliutils and apiutils from oslo into the 
assorted clients. But if this blueprint is about the python API and 
not the CLI (as that would be the openstack-pythonclient), why sync 
in apiutils?


Also does this need to go through oslo-incubator or can this start 
out as a library? Making this a library earlier on will reduce the 
number of patches needed to get 20+ repositories to use this.




Alexei and others have at least started the first stage of a rollout - 
the blueprint(s) needs additional work, planning and discussion, but 
his work is a good first step (reduce the duplication of code) 
although I am worried that the libraries and APIs / namespaces will 
need to change if we continue these discussions which potentially 
means re-doing work.


If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a 
migration of common code into oslo today, into the structure defined 
by 1 & 2


So, I sort of agree: moving / collapsing code now might be premature. 
I do strongly agree it should stand on its own as a library rather 
than an oslo incubator however. We should start with a single, clean 
namespace / library rather than depending on oslo directly.
Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall 
approach may take years to be complete.
And after they'll be approved it will become clear that this 
architecture is already outdated.

We try to use iterative approach for clients refactoring.
We started our work from removing code duplication becau

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

On 01/16/2014 06:15 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:54 AM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:



On 01/16/2014 05:25 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:07 AM, Joe Gordon <mailto:joe.gord...@gmail.com>> wrote:






On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:



On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah
mailto:chmo...@enovance.com>> wrote:



On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones mailto:c...@tenshu.net>> wrote:

Once a common library is in place, is there any intention
to (or resistance against) collapsing the clients into a
single project or even a single command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient


After speaking with people working on OSC and looking at the
code base in depth; I don't think this addresses what Chris is
implying: OSC wraps the individual CLIs built by each project
today, instead of the inverse: a common backend that the
individual CLIs can wrap - the latter is an important
distinction as currently, building a single binary install of
OSC for say, Windows is difficult given the dependency tree
incurred by each of the wrapped CLIs, difference in
dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes
/ functions / methods means that the layer that presents a
consistent user interface (OSC) to the user is made more
complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as
a developer of user interfaces (CLIs, Applications, Web apps
(horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(...)

def my_command(cli):
compute = Compute(Auth(cli.tentant..., connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or
wanted to keep their specific CLIs, they would be able to use a
not "least common denominator" back end (each service can have
a rich common.api.compute.py <http://common.api.compute.py/> or
api.compute/client.py and extend where needed. However tools
like horizon / openstackclient can choose not to leverage the
"power user/operator/admin" components and present a simplified
user interface.

I'm working on a wiki page + blueprint to brainstorm how we
could accomplish this based off of what work is in flight today
(see doug's linked blueprint) and sussing out a layout / API
strawman for discussion.

Some of the additions that came out of this email threads and
others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each
sub-project delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper
unit testing


I am happy to see this work being done, there is definitely a lot 
of work to be done on the clients.


This blueprint sounds like its still being fleshed out, so I am 
wondering what the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z


Those patches mainly sync cliutils and apiutils from oslo into the 
assorted clients. But if this blueprint is about the python API and 
not the CLI (as that would be the openstack-pythonclient), why sync 
in apiutils?


Also does this need to go through oslo-incubator or can this start 
out as a library? Making this a library earlier on will reduce the 
number of patches needed to get 20+ repositories to use this.




Alexei and others have at least started the first stage of a rollout 
- the blueprint(s) needs additional work, planning and discussion, 
but his work is a good first step (reduce the duplication of code) 
although I am worried that the libraries and APIs / namespaces will 
need to change if we continue these discussions which potentially 
means re-doing work.


If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a 
migration of common code into oslo today, into the structure defined 
by 1 & 2


So, I sort of agree: moving / collapsing code now might be 
premature. I do strongly agree it should stand on its own as a 
library rather than an oslo incubator however. We should start with 
a single, clean namespace / library rather than depending on oslo 
directly.
Knowing usual openstack workflow I'm afraid that #1,#2 with a 
waterfall approach may

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

Hello Joe,

continuous refactoring and syncing across 22+ repositories sounds like 
a nightmare, one that I would like to avoid.

You are right this is not easy.

However I have several reasons to do that:
The hardest part is to bring basic stuff in sync across all projects 
(That's what we are doing now). Later we'll work directly with oslo lib 
and just sync changes from it.


We could introduce a standalone library to avoid the need to sync oslo 
code across all projects but it brings additional problems:


1) We would have to maintain rationale versioning and backwards 
compatibility of this library. If we start library from scratch we'll 
have to add/change lots of stuff before we'll reach some stability period.


2)Another option would be to follow waterfall process and create a solid 
library interface before including it to all client projects. However 
such approach this period can take unknown amount of time and can be 
easily failed during integration stage cause requirements change or some 
other reason.


Please let me know what you think.

Best Regards,
Alexei

On 01/16/2014 08:16 PM, Joe Gordon wrote:




On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:


On 01/16/2014 06:15 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:54 AM, Alexei Kornienko
mailto:alexei.kornie...@gmail.com>>
wrote:


On 01/16/2014 05:25 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:07 AM, Joe Gordon mailto:joe.gord...@gmail.com>> wrote:





On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller
mailto:jesse.nol...@rackspace.com>> wrote:


On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah
mailto:chmo...@enovance.com>> wrote:



On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones
mailto:c...@tenshu.net>> wrote:

Once a common library is in place, is there any
intention to (or resistance against) collapsing the
clients into a single project or even a single
command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient


After speaking with people working on OSC and looking at
the code base in depth; I don't think this addresses what
Chris is implying: OSC wraps the individual CLIs built by
each project today, instead of the inverse: a common
backend that the individual CLIs can wrap - the latter is
an important distinction as currently, building a single
binary install of OSC for say, Windows is difficult given
the dependency tree incurred by each of the wrapped CLIs,
difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client
classes / functions / methods means that the layer that
presents a consistent user interface (OSC) to the user is
made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single
backend); as a developer of user interfaces (CLIs,
Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(...)

def my_command(cli):
compute = Compute(Auth(cli.tentant..., connect=True))
compute.list_flavors()

This would mean that *even if the individual clients
needed or wanted to keep their specific CLIs, they would
be able to use a not "least common denominator" back end
(each service can have a rich common.api.compute.py
<http://common.api.compute.py/> or api.compute/client.py
and extend where needed. However tools like horizon /
openstackclient can choose not to leverage the "power
user/operator/admin" components and present a simplified
user interface.

I'm working on a wiki page + blueprint to brainstorm how
we could accomplish this based off of what work is in
flight today (see doug's linked blueprint) and sussing out
a layout / API strawman for discussion.

Some of the additions that came out of this email threads
and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and
each sub-project delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for
proper unit testing


I am happy to see this work being done, there is definitely a
lot of work to be done on the clients.

This blueprint sounds like its still being fleshed out, so I
am wondering what the value is of the current patches
https://review.opens

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

Hello,

I would like to end this requirements talk cause it doesn't make any sense
in term of python clients.
Initially the discussion was about "many clients projects with separate
requirements" VS "single client project with single requirements list".

At that moment we should have stop and actually open requirements lists for
python clients.
Basically all clients have the same requirement (cause they all do the same
stuff - sending HTTP requests K.O.)
There is absolutely no difference in the situation of many clients vs
single client.

Answering another question about "user only needs X (keystone) and we
install package with clients for all openstack services":
Size of keystone client (and any other client I suppose) is ~300Kb I don't
think that it's a big difference for the user to install package that is
~300Kb or ~10Mb (unless we are using openstack from Android).

>From the user perspective I think it's much easier to use client with
"everything included" rather than try to google for client package for some
rarely used service.

Regards,
Alexei




2014/1/21 Sean Dague 

> On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
> >
> > On 17 Jan 2014, at 22:00, Jamie Lennox  > > wrote:
> >
> >> (I don't buy the problem with large amounts of dependencies, if you
> >> have a meta-package you just have one line in requirements and pip
> >> will figure the rest out.)
> >
> > +1
> >
> > Renat Akhmerov
> > @ Mirantis Inc.
>
> Man, where were you then when we had to spend 3 weeks unwinding global
> requirements in the gate because pip was figuring it out all kinds of
> wrong, and we'd do things like uninstall and reinstall
> python-keystoneclient 6 times during an install. Because after that
> experience, I'm very anti "pip will figure the rest out".
>
> Because it won't, not in python, where we're talking about libraries
> that are in the global namespace, where python can only have 1 version
> of a dependency installed.
>
> If the the solution is every openstack project should install a venv for
> all it's dependencies to get around this issue, then we're talking a
> different problem (and a different architecture from what we've been
> trying to do). But I find the idea of having 12 copies of
> python-keystone client installed on my openstack environment to be
> distasteful.
>
> So come spend a month working on requirements updates in OpenStack
> gate... and if you still believe "pip will figure it out", well you are
> a braver man than I. :)
>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

I would like to propose to use this thread to gather and discuss software
requirements that our clients should meet.
Later we'll summarize all the requirements and use them during our work of
improving the clients.
By reaching listed requirements we'll be able to evaluate the success of
our refactoring.

To start this discussion:
* API versioning should be encapsulated inside of the client.

>From the end user perspective using the code should look something like:

import openstack.nova

nova = openstack.nova.create_client(...params TBD...)
nova.start_server(...)

Reasoning for this requirement:
If user would have to specify version explicitly his code will only work
with specific version of client/api or user would have to write several
code paths for different api versions.

Implementation:
We will have an interface (abstact class) that would contain all the
methods available in latest API.
Different versions of the API would implement this methods. In case method
cannot be implemented with existing API it should raise
"UnsupportedOperation" exception.

Glossary:
user - developer who uses the client/openstack API

Please share your thoughts about this requirement/other requirements.

Looking forward for your replies,
Alexei Kornienko
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

>It is when most openstack clouds don’t just run keystone. Or nova, or
swift. Or when each client acts, smells and behaves differently. It matters
a LOT when you’re trying to write applications on top of a mature openstack
deployment.

I still don't understand the problem here. Installed packages usually just
lie quietly on your disk (which size is now measured in terabytes) and they
don't act or stink unless you ask them to. I'm pretty sure that most of the
people are not aware of ALL packages installed in their distribution and
few packages that are not used won't make it any worse


 >I have actual experience here as a person running a cloud and supporting
end-users & developers: the overwhelming customer feedback (paid and unpaid
(exploratory users)) is that the 22+ clients are confusing, difficult to
use, prone to error. There are two ways of resolving this if you’re in my
shoes - or in a role where the primary focus is not openstack developers or
builders; the first is you coordinate work focusing on developer & end user
experience upstream, working with openstack and the teams to come up with
something that benefits everyone, or, you fork, and build the openstack
equivalent to boto / awscli so you can provide a unified “one obvious way
to consume openstack clouds”.

I agree that many clients can be confusing however I think that actuall
problem here is not the clients number but discrepancies in client
parameters naming and missing/unclear help.
In our work of clients refactoring I will also update bash completition
that we have for our clients. So when you use nova start-server you will
see options applicable to this command etc.

For the unified client we can make a nice commands hierarhcy and I don't
think it will be confusing to users for example:

$ openstack help
nova cloud computing fabric controller
glance  discovering, registering, retrieving and storing virtual
machine images
cinder
...
22 records

$ openstack help nova
OpenStack Nova provides a cloud computing fabric controller, supporting a
wide variety of virtualization technologies, including KVM, Xen, LXC,
VMware, and more. In addition to its native API, it includes compatibility
with the commonly encountered Amazon EC2 and S3 APIs.

start-server
...

With proper bash completition it should be quite easy to use.



2014/1/21 Jesse Noller 

>
>  On Jan 21, 2014, at 12:19 PM, Alexei Kornienko <
> alexei.kornie...@gmail.com> wrote:
>
>  Hello,
>
> I would like to end this requirements talk cause it doesn't make any sense
> in term of python clients.
>  Initially the discussion was about "many clients projects with separate
> requirements" VS "single client project with single requirements list".
>
>
>  Then I suspect we’re at an impasse; I do plan on proceeding with a new
> wiki page and blueprint for a unified client, SDK (backend) for end users.
> I for one do not think things are as clear cut as you make them - and the
> +1s to the unified client thoughts clearly express the discontent with the
> current structure & packaging.
>
>  At that moment we should have stop and actually open requirements lists
> for python clients.
> Basically all clients have the same requirement (cause they all do the
> same stuff - sending HTTP requests K.O.)
>  There is absolutely no difference in the situation of many clients vs
> single client.
>
>  Answering another question about "user only needs X (keystone) and we
> install package with clients for all openstack services":
>  Size of keystone client (and any other client I suppose) is ~300Kb I
> don't think that it's a big difference for the user to install package that
> is ~300Kb or ~10Mb (unless we are using openstack from Android).
>
>
>  It is when most openstack clouds don’t just run keystone. Or nova, or
> swift. Or when each client acts, smells and behaves differently. It matters
> a LOT when you’re trying to write applications on top of a mature openstack
> deployment.
>
>
>  From the user perspective I think it's much easier to use client with
> "everything included" rather than try to google for client package for some
> rarely used service.
>
>
>  I have actual experience here as a person running a cloud and supporting
> end-users & developers: the overwhelming customer feedback (paid and unpaid
> (exploratory users)) is that the 22+ clients are confusing, difficult to
> use, prone to error. There are two ways of resolving this if you’re in my
> shoes - or in a role where the primary focus is not openstack developers or
> builders; the first is you coordinate work focusing on developer & end user
> experience upstream, working with openstack and the teams to come up with
> something that benefits e

Re: [openstack-dev] savann-ci, Re: [savanna] Alembic migrations and absence of DROP column in sqlite

[Alexei Kornienko]

Hi


I'm currently working on moving on the MySQL for savanna-ci
We are working on same task in ceilometer so maybe you could use some of 
our patches as reference:


https://review.openstack.org/#/c/59489/
https://review.openstack.org/#/c/63049/

Regards,
Alexei

On 02/05/2014 02:06 PM, Sergey Kolekonov wrote:

I'm currently working on moving on the MySQL for savanna-ci


On Wed, Feb 5, 2014 at 3:53 PM, Sergey Lukjanov 
mailto:slukja...@mirantis.com>> wrote:


Agreed, let's move on to the MySQL for savanna-ci to run
integration tests against production-like DB.


On Wed, Feb 5, 2014 at 1:54 AM, Andrew Lazarev
mailto:alaza...@mirantis.com>> wrote:

Since sqlite is not in the list of "databases that would be
used in production", CI should use other DB for testing.

Andrew.


On Tue, Feb 4, 2014 at 1:13 PM, Alexander Ignatov
mailto:aigna...@mirantis.com>> wrote:

Indeed. We should create a bug around that and move our
savanna-ci to mysql.

Regards,
Alexander Ignatov



On 05 Feb 2014, at 01:01, Trevor McKay mailto:tmc...@redhat.com>> wrote:

> This brings up an interesting problem:
>
> In https://review.openstack.org/#/c/70420/ I've added a
migration that
> uses a drop column for an upgrade.
>
> But savann-ci is apparently using a sqlite database to
run.  So it can't
> possibly pass.
>
> What do we do here?  Shift savanna-ci tests to non sqlite?
>
> Trevor
>
> On Sat, 2014-02-01 at 18:17 +0200, Roman Podoliaka wrote:
>> Hi all,
>>
>> My two cents.
>>
>>> 2) Extend alembic so that op.drop_column() does the
right thing
>> We could, but should we?
>>
>> The only reason alembic doesn't support these
operations for SQLite
>> yet is that SQLite lacks proper support of ALTER
statement. For
>> sqlalchemy-migrate we've been providing a work-around
in the form of
>> recreating of a table and copying of all existing rows
(which is a
>> hack, really).
>>
>> But to be able to recreate a table, we first must have
its definition.
>> And we've been relying on SQLAlchemy schema reflection
facilities for
>> that. Unfortunately, this approach has a few drawbacks:
>>
>> 1) SQLAlchemy versions prior to 0.8.4 don't support
reflection of
>> unique constraints, which means the recreated table
won't have them;
>>
>> 2) special care must be taken in 'edge' cases (e.g.
when you want to
>> drop a BOOLEAN column, you must also drop the
corresponding CHECK (col
>> in (0, 1)) constraint manually, or SQLite will raise an
error when the
>> table is recreated without the column being dropped)
>>
>> 3) special care must be taken for 'custom' type columns
(it's got
>> better with SQLAlchemy 0.8.x, but e.g. in 0.7.x we had
to override
>> definitions of reflected BIGINT columns manually for each
>> column.drop() call)
>>
>> 4) schema reflection can't be performed when alembic
migrations are
>> run in 'offline' mode (without connecting to a DB)
>> ...
>> (probably something else I've forgotten)
>>
>> So it's totally doable, but, IMO, there is no real
benefit in
>> supporting running of schema migrations for SQLite.
>>
>>> ...attempts to drop schema generation based on models
in favor of migrations
>>
>> As long as we have a test that checks that the DB
schema obtained by
>> running of migration scripts is equal to the one
obtained by calling
>> metadata.create_all(), it's perfectly OK to use model
definitions to
>> generate the initial DB schema for running of
unit-tests as well as
>> for new installations of OpenStack (and this is
actually faster than
>> running of migration scripts). ... and if we have
strong objections
>> against doing metadata.create_all(), we can always use
migration
>> scripts for both new installations and upgrades for all
DB backends,
>> except SQLite.
>>
>> Thanks,
>> Roman
>>
>> On Sat, Feb 1, 2014 

Re: [openstack-dev] [rally] Proposing changes in Rally core team

[Alexei Kornienko]

but IMO no need to rush with Alexei's removal
I'm not working actively on rally at this moment and already far behind 
of the current code base.

Cause of this I'm ok to step down from core.

Regards,

On 02/05/2014 01:17 PM, Ilya Kharin wrote:

+1 for Hugh


On Wed, Feb 5, 2014 at 2:22 PM, Sergey Skripnick 
mailto:sskripn...@mirantis.com>> wrote:



+1 for Hugh, but IMO no need to rush with Alexei's removal

Hi stackers,

I would like to:

1) Nominate Hugh Saunders to Rally core, he is doing a lot of
good reviews (and always testing patches=) ):
http://stackalytics.com/report/reviews/rally/30

2) Remove Alexei from core team, because unfortunately he is
not able to work on Rally at this moment. Thank you Alexei for
all work that you have done.


Thoughts?


Best regards,
Boris Pavlovic


-- 
Regards,

Sergey Skripnick

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-*client] moving to WebOb exceptions

[Alexei Kornienko]

Hi,

Could you please explain in more details what causes pain in adding
additional requirement to clients (in most cases when client is used inside
other openstack projects it's there already)?
My IMHO is that having to write ugly hacks to handle several exception
classes with the same name causes much more pain.

Regards,
Alexei


2014-02-26 20:02 GMT+02:00 Dean Troyer :

> On Wed, Feb 26, 2014 at 11:20 AM, Andrey Kurilin wrote:
>
>> While working on unification of clients code we try to unify various
>> exceptions in different client projects.
>> We have module apiclient.exceptions in oslo-incubator[1]. Since our
>> apiclient is an oslo-inclubator library and not a standalone lib this
>> doesn't help in case we need to process exceptions from several clients.
>>
> [...]
>
>> The solution would be to use exceptions from external library - Module
>> WebOb.exc[2] for example (since WebOb is already used in other openstack
>> projects). This exceptions cover all our custom http exceptions.
>>
>
> I would oppose adding WebOb as a requirement for the client libraries.  I
> see keystoneclient has it today but that is only because the middleware is
> still in that repo (moving those is another topic).
>
> The pain of installing the exiting client libs and their prereqs is bad
> enough, adding to it is not tenable and is part of what is motivating the
> SDK efforts.
>
> dt
>
> --
>
> Dean Troyer
> dtro...@gmail.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] The future of nosetests with Tempest

[Alexei Kornienko]

Hi,

Let me express my concerns on this topic:

With some recent changes made to Tempest compatibility with
nosetests is going away.
I think that we should not drop nosetests support from tempest or any 
other project. The problem with testrepository is that it's not 
providing any debugger support at all (and will never provide). It also 
has some issues with proving error traces in human readable form and it 
can be quite hard to find out what is actually broken.


Because of this I think we should try to avoid any kind of test 
libraries that break compatibility with conventional test runners.


Our tests should be able to run correctly with nosetests, teststools or 
plain old unittest runner. If for some reason test libraries (like 
testscenarios) doesn't provide support for this we should fix this 
libraries or avoid their usage.


Regards,
Alexei Kornienko

On 02/27/2014 06:36 PM, Frittoli, Andrea (HP Cloud) wrote:

This is another example of achieving the same result (exclusion from a
list):
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.py
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.txt

andrea

-Original Message-
From: Matthew Treinish [mailto:mtrein...@kortar.org]
Sent: 27 February 2014 15:49
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA] The future of nosetests with Tempest

On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:


On 2/12/2014 1:57 PM, Matthew Treinish wrote:

On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:


On 1/17/2014 8:34 AM, Matthew Treinish wrote:

On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:

On 01/16/2014 10:56 PM, Matthew Treinish wrote:

Hi everyone,

With some recent changes made to Tempest compatibility with
nosetests is going away. We've started using newer features that
nose just doesn't support. One example of this is that we've
started using testscenarios and we're planning to do this in more

places moving forward.

So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the requirements list and all the workarounds and
references to nose will be pulled out of the tree. Tempest will
also start raising an unsupported exception when you try to run
it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've
brought it up a couple of times before, but I believe it is time
to do this now. I feel for tempest to move forward we need to do this

now so that there isn't any ambiguity as we add even more features and new
types of testing.

I'm with you up to here.

Now, this will have implications for people running tempest with
python 2.6 since up until now we've set nosetests. There is a
workaround for getting tempest to run with python 2.6 and testr see:

https://review.openstack.org/#/c/59007/1/README.rst

but essentially this means that when nose is marked as
unsupported on tempest python 2.6 will also be unsupported by
Tempest. (which honestly it basically has been for while now just
we've gone without making it official)

The way we handle different runners/os can be categorized as
"tested in gate", "unsupported" (should work, possibly some hacks
needed), and "hostile". At present, both nose and py2.6 I would
say are in the unsupported category. The title of this message and
the content up to here says we are moving nose to the hostile
category. With only 2 months to feature freeze I see no
justification in moving
py2.6 to the hostile category. I don't see what new testing
features scheduled for the next two months will be enabled by
saying that tempest cannot and will not run on 2.6. It has been
agreed I think by all projects that py2.6 will be dropped in J. It
is OK that py2.6 will require some hacks to work and if in the
next few months it needs a few more then that is ok. If I am
missing another connection between the py2.6 and nose issues, please

explain.

So honestly we're already at this point in tempest. Nose really
just doesn't work with tempest, and we're adding more features to
tempest, your negative test generator being one of them, that
interfere further with nose. I've seen several

I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues.  I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7.  Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and t

Re: [openstack-dev] [oslo-incubator] removal of slave_connection from db.sqlalchemy.session

[Alexei Kornienko]

Hello Darren,

This option is removed since oslo.db will no longer manage engine 
objects on it's own. Since it will not store engines it cannot handle 
query dispatching.


Every project that wan't to use slave_connection will have to implement 
this logic (creation of the slave engine and query dispatching) on it's own.


Regards,

On 03/05/2014 05:18 PM, Darren Birkett wrote:

Hi,

I'm wondering why in this commit:

https://github.com/openstack/oslo-incubator/commit/630d3959b9d001ca18bd2ed1cf757f2eb44a336f

...the slave_connection option was removed.  It seems like a useful 
option to have, even if a lot of projects weren't yet using it.


Darren


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][HA][RabbitMQ] Rabbitmq-related HA issues

[Alexei Kornienko]

Hi,

No good news from me now.
I think I need to spend additional time to reproduce this issue. I've done
some code research and it seems that problem may be related to how nova
uses oslo.messaging library.

P.S. I've seen a patch in fuel changing tcp_keepalive option and it says
that it fixed the problem, am I right?

Regards,
On May 13, 2014 2:05 PM, "Vladimir Kuklin"  wrote:

> Alexey Kornienko promised to give us an update. Alexey, do you have newer
> info?
> 12 мая 2014 г. 13:22 пользователь "Vladimir Kuklin" 
> написал:
>
>> Guys, I suggest that we meet at Mirantis booth as it is in the same hall
>> with lunch
>> 12 мая 2014 г. 13:02 пользователь "Andrew Woodward" 
>> написал:
>>
>>> Correct. Openstack HA
>>> On May 12, 2014 12:43 PM, "Mike Scherbakov" 
>>> wrote:
>>>
 A little note - it's going to be about OpenStack HA issues, not about
 Fuel master node HA, right?
 On May 12, 2014 8:37 PM, "Vladimir Kuklin" 
 wrote:

> Fuelers,
>
>
> We are going to discuss rabbit/kombu related HA issues today. I
> suggest we do this at 1pm today in Hall B2 during lunch. Feel free to
> provide alternative suggestions if you have any objections. Also bring in
> anyone who may be of help.
>
> List of problems and bugs is here:
> https://etherpad.openstack.org/p/fuel-ha-rabbitmq
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 45bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com 
> www.mirantis.ru
> vkuk...@mirantis.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hi,

Please find some answers inline.

Regards,
Alexei

On 06/10/2014 03:06 PM, Flavio Percoco wrote:

On 10/06/14 15:03 +0400, Dina Belova wrote:

Hello, stackers!


Oslo.messaging is future of how different OpenStack components 
communicate with
each other, and really I'd love to start discussion about how we can 
make this

library even better then it's now and how can we refactor it make more
production-ready.


As we all remember, oslo.messaging was initially inspired to be 
created as a

logical continuation of nova.rpc - as a separated library, with lots of
transports supported, etc. That's why oslo.messaging inherited not only
advantages of now did the nova.rpc work (and it were lots of them), 
but also
some architectural decisions that currently sometimes lead to the 
performance
issues (we met some of them while Ceilometer performance testing [1] 
during the

Icehouse).


For instance, simple testing messaging server (with connection pool and
eventlet) can process 700 messages per second. The same functionality
implemented using plain kombu (without connection pool and eventlet)  
driver is

processing ten times more - 7000-8000 messages per second.


So we have the following suggestions about how we may make this 
process better

and quicker (and really I'd love to collect your feedback, folks):


1) Currently we have main loop running in the Executor class, and I 
guess it'll
be much better to move it to the Server class, as it'll make 
relationship
between the classes easier and will leave Executor only one task - 
process the
message and that's it (in blocking or eventlet mode). Moreover, this 
will make

further refactoring much easier.


To some extent, the executors are part of the server class since the
later is the one actually controlling them. If I understood your
proposal, the server class would implement the event loop, which means
we would have an EventletServer / BlockingServer, right?

If what I said is what you meant, then I disagree. Executors keep the
eventloop isolated from other parts of the library and this is really
important for us. One of the reason is to easily support multiple
python versions - by having different event loops.

Is my assumption correct? Could you elaborate more?
No It's not how we plan it. Server will do the loop and pass received 
message to dispatcher and executor. It means that we would still have 
blocking executor and eventlet executor in the same server class. We 
would just change the implementation part to make it more consistent and 
easier to control.






2) Some of the drivers implementations (such as impl_rabbit and 
impl_qpid, for

instance) are full of useless separated classes that in reality might be
included to other ones. There are already some changes making the whole
structure easier [2], and after the 1st issue will be solved 
Dispatcher and

Listener also will be able to be refactored.


This was done on purpose. The idea was to focus on backwards
compatibility rather than cleaning up/improving the drivers. That
said, sounds like those drivers could user some clean up. However, I
think we should first extend the test suite a bit more before hacking
the existing drivers.




3) If we'll separate RPC functionality and messaging functionality 
it'll make

code base clean and easily reused.


What do you mean with this?
We mean that current drivers are written with RPC code hardcoded inside 
(ReplyWaiter, etc.). Thats not how messaging library is supposed to 
work. We can move RPC to a separate layer and this would be beneficial 
for both rpc (code will become more clean and less error-prone) and core 
messaging part (we'll be able to implement messaging in way that will 
work much faster).




4) Connection pool can be refactored to implement more efficient 
connection

reusage.


Please, elaborate. What changes do you envision?
Currently there is a class that is called ConnectionContext that is used 
to manage pool. Additionaly it can be accessed/configured in several 
other places. If we refactor it a little bit it would be much easier to 
use connections from the pool.


As Dims suggested, I think filing some specs for this (and keeping the
proposals separate) would help a lot in understanding what the exact
plan is.

Glad to know you're looking forward to help improving oslo.messaging.

Thanks,
Flavio

Folks, are you ok with such a plan? Alexey Kornienko already started 
some of
this work [2], but really we want to be sure that we chose the 
correct vector

of development here.


Thanks!


[1] https://docs.google.com/document/d/
1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing

[2] https://review.openstack.org/#/q/
status:open+owner:akornienko+project:openstack/oslo.messaging,n,z


Best regards,

Dina Belova

Software Engineer

Mirantis Inc.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

On 06/10/2014 03:59 PM, Gordon Sim wrote:

On 06/10/2014 12:03 PM, Dina Belova wrote:

Hello, stackers!


Oslo.messaging is future of how different OpenStack components
communicate with each other, and really I’d love to start discussion
about how we can make this library even better then it’s now and how can
we refactor it make more production-ready.


As we all remember, oslo.messaging was initially inspired to be created
as a logical continuation of nova.rpc - as a separated library, with
lots of transports supported, etc. That’s why oslo.messaging inherited
not only advantages of now did the nova.rpc work (and it were lots of
them), but also some architectural decisions that currently sometimes
lead to the performance issues (we met some of them while Ceilometer
performance testing [1] during the Icehouse).


For instance, simple testing messaging server (with connection pool and
eventlet) can process 700 messages per second. The same functionality
implemented using plain kombu (without connection pool and eventlet)
driver is processing ten times more - 7000-8000 messages per second.


So we have the following suggestions about how we may make this process
better and quicker (and really I’d love to collect your feedback, 
folks):



1) Currently we have main loop running in the Executor class, and I
guess it’ll be much better to move it to the Server class, as it’ll make
relationship between the classes easier and will leave Executor only one
task - process the message and that’s it (in blocking or eventlet mode).
Moreover, this will make further refactoring much easier.

2) Some of the drivers implementations (such as impl_rabbit and
impl_qpid, for instance) are full of useless separated classes that in
reality might be included to other ones. There are already some changes
making the whole structure easier [2], and after the 1st issue will be
solved Dispatcher and Listener also will be able to be refactored.

3) If we’ll separate RPC functionality and messaging functionality it’ll
make code base clean and easily reused.

4) Connection pool can be refactored to implement more efficient
connection reusage.


Folks, are you ok with such a plan? Alexey Kornienko already started
some of this work [2], but really we want to be sure that we chose the
correct vector of development here.


For the impl_qpid driver, I think there would need to be quite 
significant changes to make it efficient. At present there are several 
synchronous roundtrips for every RPC call made[1]. Notifications are 
not treated any differently than RPCs (and sending a call is no 
different to sending a cast).


I agree the connection pooling is not efficient. For qpid at least it 
creates too many connections for no real benefit[2].


I think this may be a result of trying to fit the same high-level 
design to two entirely different underlying APIs.


For me at least, this also makes it hard to spot issues by reading the 
code. The qpid specific 'unit' tests for oslo.messaging also fail for 
me everytime when an actual qpidd broker is running (I haven't yet got 
to the bottom of that).


I'm personally not sure that the changes to impl_qpid you linked to 
have much impact on either efficiency or readability, safety of the code. 
Indeed it was only to remove some of the unnecessary complexity of the 
code. We'll see more improvement after we'll implement points 1,2 from 
the original email (cause the will allow us to proceed to further 
improvement)


I think there could be a lot of work required to significantly improve 
that driver, and I wonder if that would be better spent on e.g. the 
AMQP 1.0 driver which I believe will perform much better and will 
offer more choice in deployment.
I agree with you on this. However I'm not sure that we can do such a 
decision. If we focus on amqp driver only we should mention it 
explicitly and deprecate qpid driver completely. There is no point in 
keeping driver that is not really functional.


--Gordon

[1] For both the request and the response, the sender is created every 
time, which results in at least one roundtrip to the broker. Again, 
for both the request and the response, the message is then sent with a 
blocking send, meaning a further synchronous round trip for each. So 
for an RPC call, instead of just one roundtrip, there are at least four.


[2] In my view, what matters more than per-connection throughput for 
olso.messaging, is the scalability of the system as you add many RPC 
clients and servers. Excessive creation of connections by each process 
will have a negative impact on this. I don't believe the current code 
gets anywhere close to the limits of the underlying connection and 
suspect it would be more efficient and faster to multiplex different 
streams down the same connection. This would be especially true where 
using eventlet I suspect.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.opensta

Re: [openstack-dev] does exception need localize or not?

[Alexei Kornienko]

On 03/13/2014 10:44 PM, Doug Hellmann wrote:




On Thu, Feb 27, 2014 at 3:45 AM, yongli he > wrote:


refer to :
https://wiki.openstack.org/wiki/Translations

now some exception use _ and some not.  the wiki suggest do not to
do that. but i'm not sure.

what's the correct way?


F.Y.I


What To Translate

At present the convention is to translate/all/user-facing strings.
This means API messages, CLI responses, documentation, help text, etc.

There has been a lack of consensus about the translation of log
messages; the current ruling is that while it is not against
policy to mark log messages for translation if your project feels
strongly about it, translating log messages is not actively
encouraged.


I've updated the wiki to replace that paragraph with a pointer to 
https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation which 
explains the log translation rules. We will be adding the job needed 
to have different log translations during Juno.


Exception text should/not/be marked for translation, becuase if an
exception occurs there is no guarantee that the translation
machinery will be functional.


This makes no sense to me. Exceptions should be translated. By far the 
largest number of errors will be presented to users through the API or 
through Horizon (which gets them from the API). We will ensure that 
the translation code does its best to fall back to the original string 
if the translation fails.
There is another option: exception can contain non localized string and 
a thin wrapper will translate them on API layer right before output.

Something like:
print _(str(exception)).

It seems a cleaner solution to me since we don't need to add 
translations all over the code and we call a gettext just once when it's 
actually needed.


Doug



Regards
Yongli He


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Alexei Kornienko]

On 03/14/2014 09:37 AM, Radomir Dopieralski wrote:

Hello,

I also think that this thread is going in the wrong direction, but I
don't think the direction Boris wants is the correct one either. Frankly
I'm a little surprised that nobody mentioned another advantage that soft
delete gives us, the one that I think it was actually used for originally.

You see, soft delete is an optimization. It's there to make the system
work faster as a whole, have less code and be simpler to maintain and debug.

How does it do it, when, as clearly shown in the first post in this
thread, it makes the queries slower, requires additional indices in the
database and more logic in the queries? The answer is, by doing more
with those queries, by making you write less code, execute fewer queries
to the databases and avoid duplicating the same data in multiple places.

OpenStack is a big, distributed system of multiple databases that
sometimes rely on each other and cross-reference their records. It's not
uncommon to have some long-running operation started, that uses some
data, and then, in the middle of its execution, have that data deleted.
With soft delete, that's not a problem -- the operation can continue
safely and proceed as scheduled, with the data it was started with in
the first place -- it still has access to the deleted records as if
nothing happened. You simply won't be able to schedule another operation
like that with the same data, because it has been soft-deleted and won't
pass the validation at the beginning (or even won't appear in the UI or
CLI). This solves a lot of race conditions, error handling, additional
checks to make sure the record still exists, etc.
1) Operation in SQL are working in transactions so deleted records will 
be visible for other clients until transaction commit.
2) If someone inside the same transaction will try to use record that is 
already deleted it's definitely an error in our code and should be fixed.
I don't think that such use case can be used as an argument to keep soft 
deleted records.


Without soft delete, you need to write custom code every time to handle
the case of a record being deleted mid-operation, including all the
possible combinations of which record and when. Or you need to copy all
the relevant data in advance over to whatever is executing that
operation. This cannot be abstracted away entirely (although tools like
TaskFlow help), as this is specific to the case you are handling. And
it's not easy to find all the places where you can have a race condition
like that -- especially when you are modifying existing code that has
been relying on soft delete before. You can have bugs undetected for
years, that only appear in production, on very large deployments, and
are impossible to reproduce reliably.

There are more similar cases like that, including cascading deletes and
more advanced stuff, but I think this single case already shows that
the advantages of soft delete out-weight its disadvantages.

On 13/03/14 19:52, Boris Pavlovic wrote:

Hi all,


I would like to fix direction of this thread. Cause it is going in wrong
direction.

To assume:
1) Yes restoring already deleted recourses could be useful.
2) Current approach with soft deletion is broken by design and we should
get rid of them.

More about why I think that it is broken:
1) When you are restoring some resource you should restore N records
from N tables (e.g. VM)
2) Restoring sometimes means not only restoring DB records.
3) Not all resources should be restorable (e.g. why I need to restore
fixed_ip? or key-pairs?)


So what we should think about is:
1) How to implement restoring functionally in common way (e.g. framework
that will be in oslo)
2) Split of work of getting rid of soft deletion in steps (that I
already mention):
a) remove soft deletion from places where we are not using it
b) replace internal code where we are using soft deletion to that framework
c) replace API stuff using ceilometer (for logs) or this framework (for
restorable stuff)


To put in a nutshell: Restoring Delete resources / Delayed Deletion !=
Soft deletion.


Best regards,
Boris Pavlovic



On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson mailto:geekinu...@gmail.com>> wrote:

 For some guests we use the LVM imagebackend and there are times when
 the guest is deleted on accident. Humans, being what they are, don't
 back up their files and don't take care of important data, so it is
 not uncommon to use lvrestore and "undelete" an instance so that
 people can get their data. Of course, this is not always possible if
 the data has been subsequently overwritten. But it is common enough
 that I imagine most of our operators are familiar with how to do it.
 So I guess my saying that we do it on a regular basis is not quite
 accurate. Probably would be better to say that it is not uncommon to
 do this, but definitely not a daily task or something of that ilk.

 I have personally "undeleted" an 

Re: [openstack-dev] [Ceilometer][QA][Tempest][Infra] Ceilometer tempest testing in gate

[Alexei Kornienko]

On 03/21/2014 12:15 AM, Jay Pipes wrote:

On Fri, 2014-03-21 at 00:03 +0200, Alexei Kornienko wrote:

Hello,

We've done some profiling and results are quite interesting:
during 1,5 hour ceilometer inserted 59755 events (59755 calls to
record_metering_data)
this calls resulted in total 2591573 SQL queries.

Yes, this matches my own experience with Ceilo+MySQL. But do not assume
that there are 2591573/59755 or around 43 queries per record meter
event. That is misleading. In fact, the number of queries per record
meter event increases over time, as the number of retries climbs due to
contention between readers and writers.


And the most interesting part is that 291569 queries were ROLLBACK
queries.

Yep, I noted that as well. But, this is not unique to Ceilometer by any
means. Just take a look at any database serving Nova, Cinder, Glance, or
anything that uses the common SQLAlchemy code. You will see a huge
percentage of entire number of queries taken up by ROLLBACK statements.
The problem in Ceilometer is just that the write:read ratio is much
higher than any of the other projects.

I had a suspicion that the rollbacks have to do with the way that the
oslo.db retry logic works, but I never had a chance to investigate it
further. Would be really interested to see similar stats against
PostgreSQL and see if the rollback issue is isolated to MySQL (I suspect
it is).

Rollbacks are caused not by retry logic but by create_or_update logic:
We first try to do INSERT in sub-transaction when it fails we rollback 
this transaction and do update instead.

This is caused by poorly designed schema that requires such hacks.
Cause of this I suspect that we'll have similar results for PostgreSQL.

Tomorrow we'll do the same tests with PostgreSQL and MongoDB to see if 
there is any difference.




Best,
-jay


We do around 5 rollbacks to record a single event!

I guess it means that MySQL backend is currently totally unusable in
production environment.

Please find a full profiling graph attached.

Regards,

On 03/20/2014 10:31 PM, Sean Dague wrote:


On 03/20/2014 01:01 PM, David Kranz wrote:

On 03/20/2014 12:31 PM, Sean Dague wrote:

On 03/20/2014 11:35 AM, David Kranz wrote:

On 03/20/2014 06:15 AM, Sean Dague wrote:

On 03/20/2014 05:49 AM, Nadya Privalova wrote:

Hi all,
First of all, thanks for your suggestions!

To summarize the discussions here:
1. We are not going to install Mongo (because "is's wrong" ?)

We are not going to install Mongo "not from base distribution", because
we don't do that for things that aren't python. Our assumption is
dependent services come from the base OS.

That being said, being an integrated project means you have to be able
to function, sanely, on an sqla backend, as that will always be part of
your gate.

This is a claim I think needs a bit more scrutiny if by "sanely" you
mean "performant". It seems we have an integrated project that no one
would deploy using the sql db driver we have in the gate. Is any one
doing that?  Is having a scalable sql back end a goal of ceilometer?

More generally, if there is functionality that is of great importance to
any cloud deployment (and we would not integrate it if we didn't think
it was) that cannot be deployed at scale using sqla, are we really going
to say it should not be a part of OpenStack because we refuse, for
whatever reason, to run it in our gate using a driver that would
actually be used? And if we do demand an sqla backend, how much time
should we spend trying to optimize it if no one will really use it?
Though the slow heat job is a little different because the slowness
comes directly from running real use cases, perhaps we should just set
up a "slow ceilometer" job if the sql version is too slow for its budget
in the main job.

It seems like there is a similar thread, at least in part, about this
around marconi.

We required a non mongo backend to graduate ceilometer. So I don't think
it's too much to ask that it actually works.

If the answer is that it will never work and it was a checkbox with no
intent to make it work, then it should be deprecated and removed from
the tree in Juno, with a big WARNING that you shouldn't ever use that
backend. Like Nova now does with all the virt drivers that aren't tested
upstream.

Shipping in tree code that you don't want people to use is bad for
users. Either commit to making it work, or deprecate it and remove it.

I don't see this as the same issue as the slow heat job. Heat,
architecturally, is going to be slow. It spins up real OSes and does
real thinks to them. There is no way that's ever going to be fast, and
the dedicated job was a recognition that to support this level of
services in OpenStack we need to give them more breathing room.

Peace. I specifically noted that difference in my original comment. And
for that reason the heat slow job may not be temporar

Re: [openstack-dev] [Ceilometer][QA][Tempest][Infra] Ceilometer tempest testing in gate

[Alexei Kornienko]

On 03/21/2014 12:53 AM, Jay Pipes wrote:

On Fri, 2014-03-21 at 00:32 +0200, Alexei Kornienko wrote:

On 03/21/2014 12:15 AM, Jay Pipes wrote:

On Fri, 2014-03-21 at 00:03 +0200, Alexei Kornienko wrote:

Hello,

We've done some profiling and results are quite interesting:
during 1,5 hour ceilometer inserted 59755 events (59755 calls to
record_metering_data)
this calls resulted in total 2591573 SQL queries.

Yes, this matches my own experience with Ceilo+MySQL. But do not assume
that there are 2591573/59755 or around 43 queries per record meter
event. That is misleading. In fact, the number of queries per record
meter event increases over time, as the number of retries climbs due to
contention between readers and writers.


And the most interesting part is that 291569 queries were ROLLBACK
queries.

Yep, I noted that as well. But, this is not unique to Ceilometer by any
means. Just take a look at any database serving Nova, Cinder, Glance, or
anything that uses the common SQLAlchemy code. You will see a huge
percentage of entire number of queries taken up by ROLLBACK statements.
The problem in Ceilometer is just that the write:read ratio is much
higher than any of the other projects.

I had a suspicion that the rollbacks have to do with the way that the
oslo.db retry logic works, but I never had a chance to investigate it
further. Would be really interested to see similar stats against
PostgreSQL and see if the rollback issue is isolated to MySQL (I suspect
it is).

Rollbacks are caused not by retry logic but by create_or_update logic:
We first try to do INSERT in sub-transaction when it fails we rollback
this transaction and do update instead.

No, that isn't correct, AFAIK. We first do a SELECT into the table and
then if no result, try an insert:

https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/impl_sqlalchemy.py#L286-L292

The problem, IMO, is twofold. There does not need to be nested
transactional containers around these create_or_update lookups -- i.e.
the lookups can be done outside of the main transaction begin here:

https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/impl_sqlalchemy.py#L335

I'm afraid you are wrong here:

nested  =  session.connection().dialect.name  !=  'sqlite' # always True for 
MySQL
if  not  nested  and  session.query(model_class).get(str(_id)): # always False

Short circuit is used and no select is ever performed in MySQL.


Secondly, given the volume of inserts (that also generate selects), a
simple memcache lookup cache would be highly beneficial in cutting down
on writer/reader contention in MySQL.
You are right but I'm afraid that adding memcache will make deployment 
more complicated.


These are things that can be done without changing the schema (which has
other issues that can be looked at of course).

Best,
-jay


This is caused by poorly designed schema that requires such hacks.
Cause of this I suspect that we'll have similar results for PostgreSQL.

Tomorrow we'll do the same tests with PostgreSQL and MongoDB to see if
there is any difference.


Best,
-jay


We do around 5 rollbacks to record a single event!

I guess it means that MySQL backend is currently totally unusable in
production environment.

Please find a full profiling graph attached.

Regards,

On 03/20/2014 10:31 PM, Sean Dague wrote:


On 03/20/2014 01:01 PM, David Kranz wrote:

On 03/20/2014 12:31 PM, Sean Dague wrote:

On 03/20/2014 11:35 AM, David Kranz wrote:

On 03/20/2014 06:15 AM, Sean Dague wrote:

On 03/20/2014 05:49 AM, Nadya Privalova wrote:

Hi all,
First of all, thanks for your suggestions!

To summarize the discussions here:
1. We are not going to install Mongo (because "is's wrong" ?)

We are not going to install Mongo "not from base distribution", because
we don't do that for things that aren't python. Our assumption is
dependent services come from the base OS.

That being said, being an integrated project means you have to be able
to function, sanely, on an sqla backend, as that will always be part of
your gate.

This is a claim I think needs a bit more scrutiny if by "sanely" you
mean "performant". It seems we have an integrated project that no one
would deploy using the sql db driver we have in the gate. Is any one
doing that?  Is having a scalable sql back end a goal of ceilometer?

More generally, if there is functionality that is of great importance to
any cloud deployment (and we would not integrate it if we didn't think
it was) that cannot be deployed at scale using sqla, are we really going
to say it should not be a part of OpenStack because we refuse, for
whatever reason, to run it in our gate using a driver that would
actually be used? And if we do demand an sqla backend, how much time
should we spend trying to optimize it if no one will really use it?
Though the slow heat job is a little different because the slowness
comes di

Re: [openstack-dev] [Ceilometer][QA][Tempest][Infra] Ceilometer tempest testing in gate

[Alexei Kornienko]

Hello,

Please see some comments inline.

Best Regards,
Alexei Kornienko

On 03/21/2014 11:11 PM, Joe Gordon wrote:




On Fri, Mar 21, 2014 at 4:04 AM, Sean Dague <mailto:s...@dague.net>> wrote:


On 03/20/2014 06:18 PM, Joe Gordon wrote:
>
>
>
> On Thu, Mar 20, 2014 at 3:03 PM, Alexei Kornienko
> mailto:alexei.kornie...@gmail.com>
<mailto:alexei.kornie...@gmail.com
<mailto:alexei.kornie...@gmail.com>>> wrote:
>
> Hello,
>
> We've done some profiling and results are quite interesting:
> during 1,5 hour ceilometer inserted 59755 events (59755 calls to
> record_metering_data)
> this calls resulted in total 2591573 SQL queries.
>
> And the most interesting part is that 291569 queries were
ROLLBACK
> queries.
> We do around 5 rollbacks to record a single event!
>
> I guess it means that MySQL backend is currently totally
unusable in
> production environment.
>
>
> It should be noticed that SQLAlchemy is horrible for performance, in
> nova we usually see sqlalchemy overheads of well over 10x (time
> nova.db.api call vs the time MySQL measures when slow log is
recording
> everything).

That's not really a fair assessment. Python object inflation takes
time.
I do get that there is SQLA overhead here, but even if you trimmed it
out you would not get the the mysql query time.


To give an example from nova:

doing a nova list with no servers:

stack@devstack:~/devstack$ nova --timing list

| GET 
http://10.0.0.16:8774/v2/a82ededa9a934b93a7184d06f302d745/servers/detail 
| 0.0817470550537 |


So nova command takes 0.0817470550537 seconds.

Inside the nova logs (when putting a timer around all nova.db.api 
calls [1] ), nova.db.api.instance_get_all_by_filters takes 0.06 seconds:


2014-03-21 20:58:46.760 DEBUG nova.db.api 
[req-91879f86-7665-4943-8953-41c92c42c030 demo demo] 
'instance_get_all_by_filters' 0.06 seconds timed 
/mnt/stack/nova/nova/db/api.py:1940


But the sql slow long reports the same query takes only 0.001006 
seconds with a lock_time of 0.000269 for a total of  0.00127 seconds.


# Query_time: 0.001006  Lock_time: 0.000269 Rows_sent: 0 
 Rows_examined: 0



So in this case only 2% of the time 
that  nova.db.api.instance_get_all_by_filters takes is spent inside of 
mysql. Or to put it differently 
 nova.db.api.instance_get_all_by_filters is 47 times slower then the 
raw DB call underneath.


Yes I agree that that turning raw sql data into python objects should 
take time, but I just don't think it should take 98% of the time.
If you would open actual code of nova.db.api.instance_get_all_by_filters 
- 
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1817

You will find out that python code is actually doing lot's of things:
1) setup join conditions
2) create query filters
3) doing some heavy matching, loops in exact_filter, regex_filter, 
tag_filter
This code won't go away with python objects since it's related to 
busyness logic.
I think that it's quite hypocritical to say that the problem is "turning 
raw sql data into python objects"




[1] 
https://github.com/jogo/nova/commit/7743ee366bbf8746f1c0f634f29ebf73bff16ea1


That being said, having Ceilometer's write path be highly tuned
and not
use SQLA (and written for every back end natively) is probably
appropriate.


While I like this idea, they loose free postgresql support by dropping 
SQLA. But that is a solvable problem.



-Sean

--
Sean Dague
Samsung Research America
s...@dague.net <mailto:s...@dague.net> / sean.da...@samsung.com
<mailto:sean.da...@samsung.com>
http://dague.net


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hello,

Returning to performance issues of oslo.messaging.
I've found 2 biggest issue in existing implementation of rabbit (kombu) 
driver:


1) For almost every message sent/received a new object of 
Consumer/Publisher class is created. And each object of this class tries 
to declare it's queue even if it's already declared.

https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159
This causes a huge slowdown.

2) with issue #1 is fixed (I've applied a small hack to fix it in my 
repo) the next big issue araise. For every rpc message received a reply 
is sent when processing is done (it seems that reply is sent even for 
"cast" calls which it really strange to me). Reply sent is using 
connection pool to "speed up" replies. Due to bad implementation of 
custom connection pool for every message sent underlying connection 
channel is closed and reopened:

https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689

Cause of this major issues oslo.messaging performance is at least 10 
times slower than it could be.


My opinion is that there is no simple and elegant fix for this issues in 
current implementation of oslo.messaging (most because of bad 
architecture of the library and processing flow). My proposal is that we 
should start working on new major release of messaging library with 
architectural issues fixed. This will allow us to avoid mentioned issues 
and provide much more performance and flexibility for end users.
Main goal that we should achieve is separate rpc code from messaging 
code this will allow us to implement both parts in much simpler and 
cleaner way and in the same time it would be much faster.


Please share your thoughts on this topic.

Regards,
Alexei Kornienko

On 06/16/2014 02:47 PM, Gordon Sim wrote:

On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote:

On 10/06/14 15:40, Alexei Kornienko wrote:

On 06/10/2014 03:59 PM, Gordon Sim wrote:

>>>

I think there could be a lot of work required to significantly
improve that driver, and I wonder if that would be better spent
on e.g. the AMQP 1.0 driver which I believe will perform much
better and will offer more choice in deployment.

>>

I agree with you on this. However I'm not sure that we can do such
a decision. If we focus on amqp driver only we should mention it
explicitly and deprecate qpid driver completely. There is no point
in keeping driver that is not really functional.


The driver is functional. It may be not that efficient as
alternatives, but that's not a valid reason to deprecate it.


The question in my view is what the plan is for ongoing development.

Will the driver get better over time, or is it likely to remain as is 
at best (or even deteriorate)?


Choice is good, but every choice adds to the maintenance burden, in 
testing against regressions if nothing else.


I think an explicit decision about the future is beneficial, whatever 
the decision may be.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hello Jay,

Benchmark for oslo.messaging is really simple:
You create a client that sends messages infinitively and a server that 
processes them. After you can use rabbitmq management plugin to see 
average throughput in the queue.
Simple example can be found here - 
https://github.com/andreykurilin/profiler-for-oslo-messaging


I've mentioned some of this already in my previous mails but here it is 
again:


"Huge" is descriptive but not quantitative :) Do you have any numbers 
that pinpoint the amount of time that is being spent reconstructing 
and declaring the queues, say, compared to the time spent doing 
transmission? 
I don't have precise slowdown percentage for each issue. I've just 
identified hotspots using cProfile and strace.


This ten times number... is that an estimate or do you have hard 
numbers that show that? Just curious. 
Using a script that I've mentioned earlier I get average throughput on 
my PC ~700 cast calls per second.
I've written a simple and stupid script that uses kombu directly (in a 
single threaded and synchronous way with single connection) It gave me 
throughput ~8000 messages per second.

Thats why I say that library should work at least 10 times faster.

Regards,
Alexei Kornienko


On 06/26/2014 11:22 PM, Jay Pipes wrote:

Hey Alexei, thanks for sharing your findings. Comments inline.

On 06/26/2014 10:08 AM, Alexei Kornienko wrote:

Hello,

Returning to performance issues of oslo.messaging.
I've found 2 biggest issue in existing implementation of rabbit (kombu)
driver:

1) For almost every message sent/received a new object of
Consumer/Publisher class is created. And each object of this class tries
to declare it's queue even if it's already declared.
https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 



This causes a huge slowdown.


"Huge" is descriptive but not quantitative :) Do you have any numbers 
that pinpoint the amount of time that is being spent reconstructing 
and declaring the queues, say, compared to the time spent doing 
transmission?



2) with issue #1 is fixed (I've applied a small hack to fix it in my
repo) the next big issue araise. For every rpc message received a reply
is sent when processing is done (it seems that reply is sent even for
"cast" calls which it really strange to me). Reply sent is using
connection pool to "speed up" replies. Due to bad implementation of
custom connection pool for every message sent underlying connection
channel is closed and reopened:
https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689 



Cause of this major issues oslo.messaging performance is at least 10
times slower than it could be.


This ten times number... is that an estimate or do you have hard 
numbers that show that? Just curious.



My opinion is that there is no simple and elegant fix for this issues in
current implementation of oslo.messaging (most because of bad
architecture of the library and processing flow). My proposal is that we
should start working on new major release of messaging library with
architectural issues fixed. This will allow us to avoid mentioned issues
and provide much more performance and flexibility for end users.
Main goal that we should achieve is separate rpc code from messaging
code this will allow us to implement both parts in much simpler and
cleaner way and in the same time it would be much faster.


Perhaps actually a better starting point would be to create a 
benchmarking harness that will allow us to see some baseline 
throughput numbers that we can then compare to the iterative 
improvements you will push?


Best,
-jay


Please share your thoughts on this topic.

Regards,
Alexei Kornienko

On 06/16/2014 02:47 PM, Gordon Sim wrote:

On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote:

On 10/06/14 15:40, Alexei Kornienko wrote:

On 06/10/2014 03:59 PM, Gordon Sim wrote:

>>>

I think there could be a lot of work required to significantly
improve that driver, and I wonder if that would be better spent
on e.g. the AMQP 1.0 driver which I believe will perform much
better and will offer more choice in deployment.

>>

I agree with you on this. However I'm not sure that we can do such
a decision. If we focus on amqp driver only we should mention it
explicitly and deprecate qpid driver completely. There is no point
in keeping driver that is not really functional.


The driver is functional. It may be not that efficient as
alternatives, but that's not a valid reason to deprecate it.


The question in my view is what the plan is for ongoing development.

Will the driver get better over time, or is it likely to remain as is
at best (or even deteriorate)?

Choice is good, but every choice adds to the maintenance burden, in
testing against regressions if nothing else.

I think an explicit decision about the future is beneficial

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hello,

Please find some answers inline.

Regards,
Alexei

On 06/27/2014 04:04 PM, Ihar Hrachyshka wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 26/06/14 22:38, Alexei Kornienko wrote:

Hello Jay,

Benchmark for oslo.messaging is really simple: You create a client
that sends messages infinitively and a server that processes them.
After you can use rabbitmq management plugin to see average
throughput in the queue. Simple example can be found here -
https://github.com/andreykurilin/profiler-for-oslo-messaging

I've mentioned some of this already in my previous mails but here
it is again:


"Huge" is descriptive but not quantitative :) Do you have any
numbers that pinpoint the amount of time that is being spent
reconstructing and declaring the queues, say, compared to the
time spent doing transmission?

I don't have precise slowdown percentage for each issue. I've just
identified hotspots using cProfile and strace.


This ten times number... is that an estimate or do you have hard
numbers that show that? Just curious.

Using a script that I've mentioned earlier I get average throughput
on my PC ~700 cast calls per second. I've written a simple and
stupid script that uses kombu directly (in a single threaded and
synchronous way with single connection) It gave me throughput ~8000
messages per second. Thats why I say that library should work at
least 10 times faster.

It doesn't show that those major issues you've pointed out result in
such large message processing speed dropdown though. Maybe there are
other causes of slowdown we observe. Neither it shows that refactoring
of the code will actually help to boost the library significantly.

It doesn't show that those major issues are the *only* reason for slowdown.
Bit it shows that those issues are biggest that are currently visible.
We'll get a major speed boost if we fix them (and possibly discover new 
issues the would prevent us of reaching full speed).


Though having some hard numbers from using kombu directly is still a
good thing. Is it possible that we introduce some performance tests
into oslo.messaging itself?
Some performance tests may be introduced but they would be more like 
functional tests since they require setup of actual messaging server 
(rabbit, etc.).

What do you mean exactly by performance tests?
Just testing overall throughput with some basic scenarios or you mean 
finding hotspots in the code?





Regards, Alexei Kornienko


On 06/26/2014 11:22 PM, Jay Pipes wrote:

Hey Alexei, thanks for sharing your findings. Comments inline.

On 06/26/2014 10:08 AM, Alexei Kornienko wrote:

Hello,

Returning to performance issues of oslo.messaging. I've found 2
biggest issue in existing implementation of rabbit (kombu)
driver:

1) For almost every message sent/received a new object of
Consumer/Publisher class is created. And each object of this
class tries to declare it's queue even if it's already
declared.
https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159





This causes a huge slowdown.

"Huge" is descriptive but not quantitative :) Do you have any
numbers that pinpoint the amount of time that is being spent
reconstructing and declaring the queues, say, compared to the
time spent doing transmission?


2) with issue #1 is fixed (I've applied a small hack to fix it
in my repo) the next big issue araise. For every rpc message
received a reply is sent when processing is done (it seems that
reply is sent even for "cast" calls which it really strange to
me). Reply sent is using connection pool to "speed up" replies.
Due to bad implementation of custom connection pool for every
message sent underlying connection channel is closed and
reopened:
https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689





Cause of this major issues oslo.messaging performance is at least 10

times slower than it could be.

This ten times number... is that an estimate or do you have hard
numbers that show that? Just curious.


My opinion is that there is no simple and elegant fix for this
issues in current implementation of oslo.messaging (most
because of bad architecture of the library and processing
flow). My proposal is that we should start working on new major
release of messaging library with architectural issues fixed.
This will allow us to avoid mentioned issues and provide much
more performance and flexibility for end users. Main goal that
we should achieve is separate rpc code from messaging code this
will allow us to implement both parts in much simpler and
cleaner way and in the same time it would be much faster.

Perhaps actually a better starting point would be to create a
benchmarking harness that will allow us to see some baseline
throughput numbers that we can then compare to the iterative
improvements you will push?

Best, -jay


Please share 

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hi,

Why should we create queue in advance?

Let's consider following use cases:
1)
* listener starts and creates a queue
* publishers connect to exchange and start publishing

No need to create a queue in advance here since listener does it when it 
starts


2)
* publishers create a queue in advance and start publishing


Creation is not correct since there is no guarantee that someone would 
ever use this queue...


IMHO listener should create a queue and publishers should not care about 
it at all.


What do you think?

On 06/27/2014 05:16 PM, Sandy Walsh wrote:
Something to consider is the "create the queue in advance" feature is 
done for notifications, so we don't drop important messages on the 
floor by having an Exchange with no associated Queue.


For RPC operations, this may not be required (we assume the service is 
available). If this check is truly a time-sink we could ignore that 
check for rpc calls.


-S


On 6/10/2014 9:31 AM, Alexei Kornienko wrote:

Hi,

Please find some answers inline.

Regards,
Alexei

On 06/10/2014 03:06 PM, Flavio Percoco wrote:

On 10/06/14 15:03 +0400, Dina Belova wrote:

Hello, stackers!


Oslo.messaging is future of how different OpenStack components 
communicate with
each other, and really I'd love to start discussion about how we 
can make this

library even better then it's now and how can we refactor it make more
production-ready.


As we all remember, oslo.messaging was initially inspired to be 
created as a
logical continuation of nova.rpc - as a separated library, with 
lots of
transports supported, etc. That's why oslo.messaging inherited not 
only
advantages of now did the nova.rpc work (and it were lots of them), 
but also
some architectural decisions that currently sometimes lead to the 
performance
issues (we met some of them while Ceilometer performance testing 
[1] during the

Icehouse).


For instance, simple testing messaging server (with connection pool 
and

eventlet) can process 700 messages per second. The same functionality
implemented using plain kombu (without connection pool and 
eventlet)  driver is

processing ten times more - 7000-8000 messages per second.


So we have the following suggestions about how we may make this 
process better

and quicker (and really I'd love to collect your feedback, folks):


1) Currently we have main loop running in the Executor class, and I 
guess it'll
be much better to move it to the Server class, as it'll make 
relationship
between the classes easier and will leave Executor only one task - 
process the
message and that's it (in blocking or eventlet mode). Moreover, 
this will make

further refactoring much easier.


To some extent, the executors are part of the server class since the
later is the one actually controlling them. If I understood your
proposal, the server class would implement the event loop, which means
we would have an EventletServer / BlockingServer, right?

If what I said is what you meant, then I disagree. Executors keep the
eventloop isolated from other parts of the library and this is really
important for us. One of the reason is to easily support multiple
python versions - by having different event loops.

Is my assumption correct? Could you elaborate more?
No It's not how we plan it. Server will do the loop and pass received 
message to dispatcher and executor. It means that we would still have 
blocking executor and eventlet executor in the same server class. We 
would just change the implementation part to make it more consistent 
and easier to control.






2) Some of the drivers implementations (such as impl_rabbit and 
impl_qpid, for
instance) are full of useless separated classes that in reality 
might be
included to other ones. There are already some changes making the 
whole
structure easier [2], and after the 1st issue will be solved 
Dispatcher and

Listener also will be able to be refactored.


This was done on purpose. The idea was to focus on backwards
compatibility rather than cleaning up/improving the drivers. That
said, sounds like those drivers could user some clean up. However, I
think we should first extend the test suite a bit more before hacking
the existing drivers.




3) If we'll separate RPC functionality and messaging functionality 
it'll make

code base clean and easily reused.


What do you mean with this?
We mean that current drivers are written with RPC code hardcoded 
inside (ReplyWaiter, etc.). Thats not how messaging library is 
supposed to work. We can move RPC to a separate layer and this would 
be beneficial for both rpc (code will become more clean and less 
error-prone) and core messaging part (we'll be able to implement 
messaging in way that will work much faster).




4) Connection pool can be refactored to implement more efficient 
connection

reusage.


Please, elaborate. What changes do you envision?
Currently there is a class that is called ConnectionCont

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hello,


> My understanding is that your analysis is mostly based on running a
> profiler against the code. Network operations can be bottlenecked in
> other places.
>
> You compare 'simple script using kombu' with 'script using
> oslo.messaging'. You don't compare script using oslo.messaging before
> refactoring and 'after that. The latter would show whether refactoring
> was worth the effort. Your test shows that oslo.messaging performance
> sucks, but it's not definite that hotspots you've revealed, once
> fixed, will show huge boost.
>
> My concern is that it may turn out that once all the effort to
> refactor the code is done, we won't see major difference. So we need
> base numbers, and performance tests would be a great helper here.
>

It's really sad for me to see so little faith in what I'm saying.
The test I've done using plain kombu driver was needed exactly to check
that network is not the bottleneck for messaging performance.
If you don't believe in my performance analysis we could ask someone else
to do their own research and provide results.

Problem with refactoring that I'm planning is that it's not a minor
refactoring that can be applied in one patch but it's the whole library
rewritten from scratch.
Existing messaging code was written long long time ago (in a galaxy far far
away maybe?) and it was copy-pasted directly from nova.
It was not built as a library and it was never intended to be used outside
of nova.
Some parts of it cannot even work normally cause it was not designed to
work with drivers like zeromq (matchmaker stuff).

The reason I've raised this question on the mailing list was to get some
agreement about future plans of oslo.messaging development and start
working on it in coordination with community.
For now I don't see any actions plan emerging from it. I would like to see
us bringing more constructive ideas about what should be done.

If you think that first action should be profiling lets discuss how it
should be implemented (cause it works for me just fine on my local PC).
I guess we'll need to define some basic scenarios that would show us
overall performance of the library.
There are a lot of questions that should be answered to implement this:
Where such tests would run (jenking, local PC, devstack VM)?
How such scenarios should look like?
How do we measure performance (cProfile, etc.)?
How do we collect results?
How do we analyze results to find bottlenecks?
etc.

Another option would be to spend some of my free time implementing
mentioned refactoring (as I see it) and show you the results of performance
testing compared with existing code.
The only problem with such approach is that my code won't be oslo.messaging
and it won't be accepted by community. It may be drop in base for v2.0 but
I'm afraid this won't be acceptable either.

Regards,
Alexei Kornienko


2014-06-30 17:51 GMT+03:00 Gordon Sim :

> On 06/30/2014 12:22 PM, Ihar Hrachyshka wrote:
>
>  Alexei Kornienko wrote:
>>>
>>>> Some performance tests may be introduced but they would be more
>>>> like functional tests since they require setup of actual
>>>> messaging server (rabbit, etc.).
>>>>
>>>
>> Yes. I think we already have some. F.e.
>> tests/drivers/test_impl_qpid.py attempts to use local Qpid server
>> (backing up to fake server if it's not available).
>>
>
> I always get failures when there is a real qpidd service listening on the
> expected port. Does anyone else see this?
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hi,

Thanks for detailed answer.
Please see my comments inline.

Regards,

On 07/01/2014 04:28 PM, Ihar Hrachyshka wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 30/06/14 21:34, Alexei Kornienko wrote:

Hello,


My understanding is that your analysis is mostly based on running
a profiler against the code. Network operations can be bottlenecked
in other places.

You compare 'simple script using kombu' with 'script using
oslo.messaging'. You don't compare script using oslo.messaging
before refactoring and 'after that. The latter would show whether
refactoring was worth the effort. Your test shows that
oslo.messaging performance sucks, but it's not definite that
hotspots you've revealed, once fixed, will show huge boost.

My concern is that it may turn out that once all the effort to
refactor the code is done, we won't see major difference. So we
need base numbers, and performance tests would be a great helper
here.


It's really sad for me to see so little faith in what I'm saying.
The test I've done using plain kombu driver was needed exactly to
check that network is not the bottleneck for messaging
performance. If you don't believe in my performance analysis we
could ask someone else to do their own research and provide
results.

Technology is not about faith. :)

First, let me make it clear I'm *not* against refactoring or anything
that will improve performance. I'm just a bit skeptical, but hopefully
you'll be able to show everyone I'm wrong, and then the change will
occur. :)

To add more velocity to your effort, strong arguments should be
present. To facilitate that, I would start from adding performance
tests that would give us some basis for discussion of changes proposed
later.

Please see below for detailed answer about performance tests implementation.
It explains a bit why it's hard to present arguments that would be 
strong enough for you.

I may run performance tests locally but it's not enough for community.

And in addition I've provided some links to existing implementation with 
places that IMHO cause bottlenecks.
From my point of view that code is doing obviously stupid things (like 
closing/opening sockets for each message sent).
That is enough for me to rewrite it even without additional proofs that 
it's wrong.


Then, describing proposed details in a spec will give more exposure to
your ideas. At the moment, I see general will to enhance the library,
but not enough details on how to achieve this. Specification can make
us think not about the burden of change that obviously makes people
skeptic about rewrite-all approach, but about specific technical issues.
I agree that we should start with a spec. However instead of having spec 
of needed changes I would prefer to have a spec describing needed 
functionality of the library (it may differ from existing functionality).
Using such a spec we could decide what it needed and what needs to be 
removed to achieve what we need.



Problem with refactoring that I'm planning is that it's not a
minor refactoring that can be applied in one patch but it's the
whole library rewritten from scratch.

You can still maintain a long sequence of patches, like we did when we
migrated neutron to oslo.messaging (it was like ~25 separate pieces).
Talking into account possible gate issues I would like to avoid long 
series of patches since they won't be able to land at the same time and 
rebasing will become a huge pain.
If we decide to start working on 2.0 API/implementation I think a topic 
branch 2.0a would be much better.



Existing messaging code was written long long time ago (in a galaxy
far far away maybe?) and it was copy-pasted directly from nova. It
was not built as a library and it was never intended to be used
outside of nova. Some parts of it cannot even work normally cause
it was not designed to work with drivers like zeromq (matchmaker
stuff).

oslo.messaging is NOT the code you can find in oslo-incubator rpc
module. It was hugely rewritten to expose a new, cleaner API. This is
btw one of the reasons migration to this new library is so painful. It
was painful to move to oslo.messaging, so we need clear need for a
change before switching to yet another library.
API indeed has changed but general implementation details and processing 
flow goes way back to 2011 and nova code (for example general 
Publisher/Consumer implementation in impl_rabbit)

That's the code I'm talking about.

Refactoring as I see it will do the opposite thing. It will keep intact 
as much API as possible but change internals to make it more efficient 
(that's why I call it refactoring) So 2.0 version might be (partially?) 
backwards compatible and migration won't be such a pain.



The reason I've raised this question on the mailing list was to get
some agreement about future plans of oslo.messaging developme

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Alexei Kornienko]

Hi,

Please see some minor comments inline.
Do you think we can schedule some time to discuss this topic on one of 
the upcoming meetings?
We can come out with some kind of the summary and actions plan to start 
working on.


Regards,

On 07/01/2014 05:52 PM, Ihar Hrachyshka wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 01/07/14 15:55, Alexei Kornienko wrote:

Hi,

Thanks for detailed answer. Please see my comments inline.

Regards,

On 07/01/2014 04:28 PM, Ihar Hrachyshka wrote: On 30/06/14 21:34,
Alexei Kornienko wrote:

Hello,


My understanding is that your analysis is mostly based on
running a profiler against the code. Network operations can
be bottlenecked in other places.

You compare 'simple script using kombu' with 'script using
oslo.messaging'. You don't compare script using
oslo.messaging before refactoring and 'after that. The latter
would show whether refactoring was worth the effort. Your
test shows that oslo.messaging performance sucks, but it's
not definite that hotspots you've revealed, once fixed, will
show huge boost.

My concern is that it may turn out that once all the effort
to refactor the code is done, we won't see major difference.
So we need base numbers, and performance tests would be a
great helper here.


It's really sad for me to see so little faith in what I'm
saying. The test I've done using plain kombu driver was
needed exactly to check that network is not the bottleneck
for messaging performance. If you don't believe in my
performance analysis we could ask someone else to do their
own research and provide results.

Technology is not about faith. :)

First, let me make it clear I'm *not* against refactoring or
anything that will improve performance. I'm just a bit skeptical,
but hopefully you'll be able to show everyone I'm wrong, and then
the change will occur. :)

To add more velocity to your effort, strong arguments should be
present. To facilitate that, I would start from adding performance
tests that would give us some basis for discussion of changes
proposed later.

Please see below for detailed answer about performance tests
implementation. It explains a bit why it's hard to present
arguments that would be strong enough for you. I may run
performance tests locally but it's not enough for community.

Yes, that's why shipping some tests ready to run with oslo.messaging
can help. Science is about reproducility, right? ;)


And in addition I've provided some links to existing
implementation with places that IMHO cause bottlenecks. From my
point of view that code is doing obviously stupid things (like
closing/opening sockets for each message sent).

That indeed sounds bad.


That is enough for me to rewrite it even without additional
proofs that it's wrong.

[Full disclosure: I'm not as involved into oslo.messaging internals as
you probably are, so I may speak out dumb things.]

I wonder whether there are easier ways to fix that particular issue
without rewriting everything from scratch. Like, provide a pool of
connections and make send() functions use it instead of creating new
connections (?)
I've tried to find a way to fix that without big changes but 
unfortunately I've failed to do so.
Problem I see is that connection pool is defined and used on 1 layer of 
library and the problem is on the other.
To fix this issues we need to change several layers of code and it's 
shared between 2 drivers - rabbit, qpid.
Cause of this it seems really hard to make some logically finished and 
working patches that would allow us to move in proper direction without 
big refactoring of the drivers structure.



Then, describing proposed details in a spec will give more exposure
to your ideas. At the moment, I see general will to enhance the
library, but not enough details on how to achieve this.
Specification can make us think not about the burden of change that
obviously makes people skeptic about rewrite-all approach, but
about specific technical issues.

I agree that we should start with a spec. However instead of
having spec of needed changes I would prefer to have a spec
describing needed functionality of the library (it may differ
from existing functionality).

Meaning, breaking API, again?
It's not about breaking the API it's about making it more logical and 
independent. Right now it's not clear to me what API classes are used 
and how they are used.
A lot of driver details leak outside the API and it makes it hard to 
improve driver without changing the API.
What I would like to see is a clear definition of what library should 
provide and API interface that it should implement.
It may be a little bit java like so API should be defined and frozed and 
anyone could propose their driver implementation using kombu/qpid/zeromq 
or pigeons and trained dolphins to deliver messages.


This would allow us to change drivers 

Re: [openstack-dev] [neutron][oslo.messaging]

[Alexei Kornienko]

Hi,

You can use /oslo.messaging._drivers.impl_rabbit/ instead of impl_kombu
It was renamed and slightly change but I think it will work as you expect.

Regards,
Alexei Kornienko

On 07/03/2014 08:47 PM, Nader Lahouti wrote:

Hi All and Ihar,

As part of blueprint oslo-messaging the neutron/openstack/common/rpc 
tree is removed. I was using impl_kombu module to process notification 
from keystone with this following code sample:

...
from neutron.openstack.common.rpc import impl_kombu
   try:
   conf = impl_kombu.cfg.CONF
topicname = self._topic_name
exchange = self._exchange_name
connection = impl_kombu.Connection(conf)
connection.declare_topic_consumer(topic,
self.callback,
topic, exchange)
connection.consume()
except Exception:
connection.close()


Can you please let me what needs to be done to replace the above code 
and make it work with current neutron code?



Thanks in advance,
Nader.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][oslo] Problem installing oslo.config-1.4.0.0a3 from .whl files

[Alexei Kornienko]

Hello Carl,

You should try to update your virtualenv (pip install -U virtualenv).
It fixed this problem for me.

Regards,
Alexei

On 05/08/14 23:00, Carl Baldwin wrote:

Hi,

I noticed this yesterday afternoon.  I tried to run pep8 and unit
tests on a patch I was going to submit.  It failed with an error that
no package satisfying oslo.config could be found [1].  I went to pypi
and saw that the version appears to be available [2] but still
couldn't install it.

I tried to activate the .tox/pep8 virtual environment and install the
version explicitly.  Interestingly, that worked in one gerrit repo for
Neutron [3] but not the other [4].  These two virtual envs are on the
same machine.  I ran "git clean -fdx" to start over and now neither
virtualenv can install it.

Anyone have any idea what is going on?  It seems to be related to the
fact that oslo.config is now uploaded as .whl files, whatever those
are.  Why is it that my system cannot handle these?  I noticed that
oslo.config is now available only as .whl in the 1.4.0.0aN versions
but used to be available as .tar.gz files.

Carl

[1] http://paste.openstack.org/show/90651/
[2] https://pypi.python.org/pypi/oslo.config
[3] http://paste.openstack.org/show/90674/
[4] http://paste.openstack.org/show/90675/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [DB][Migrations] Switching to using of Alembic

[Alexei Kornienko]

Hello,

Please see my comments inline.

Best regards,
Alexei Kornienko

On 07/16/2013 09:16 PM, David Ripton wrote:

On 07/16/2013 01:58 PM, Boris Pavlovic wrote:


There is no "magic" and we have only 2 ways to end up with this problems
and bugs that could be caused by "manually" migrations merging and tons
of bugs in sqlalchemy-migrate.

1) step by step (openstack method)
   There are special tests "test_migrations" that runs migrations on
real data against all backends. So we should:

   a) improve this tests to checks all behaviors // there is a lot of
hidden bugs
   b) replace migration (only one small migration) to alembic
   c) check that in all backends we made the same changes in schema
   d) Merge all old migrations in one using alembic (automatically).
   So it could be done in safe way.

2.a) huge 2 steps
   1. Merge all migrations in one huge manually (drop all tests in test
migrations)
   e.g. In Nova was patch https://review.openstack.org/#/c/35748/
   I don't believe that there are no mistakes in this migration, and
nobody is able to check it. // because of tons of hidden bugs in old
migrations and sqla-migrate.
   2. Replace this migration in Alembic
I don't believe that there will be way to check that there is no
bugs

2.b) suicide mode (1 big step)
   Merge and switch in one step=)


We have compacted migrations before, and there's a test document for 
how to verify that the big migration has exactly the same output as 
the series of small migrations.  See 
https://wiki.openstack.org/wiki/Database_migration_testing  Dan Prince 
is the expert on this.


I think the right process is:

1. Wait until the very beginning of Icehouse cycle.  (But not after we 
have new migrations for Icehouse.)


2. Compact all migrations into 2xx_havana.py (for SQLAlchemy-migrate)
As Boris mentioned: "We are able only to convert only grizzly migrations 
not havana (because our customers should be able to switch from grizzly 
to havana)"

so in the beginning of Icehouse we'll have:
2xx_grizzly.py (sqla-migrate)
2xx_havana.py (sqla-migrate)
...
2xx_havana.py (sqla-migrate)
___
Icehouse migrations...

So my question is: when we should start using alembic for new migrations?
We won't be able to deprecate sqla-migrate and switch to alembic in one 
step since we'll have some migrations kept for backwards compatibility.
I think that to accomplish your plan we should start using alembic for 
new migrations starting from Icehouse.
If you are ok with it I can change ceilometer migrations implementation 
to work accordingly and we'll be able to use it as a reference in other 
projects.




3. Test that it's perfect via above test plan plus whatever 
enhancements we think of.


4. Manually convert 2xx_havana.py (for SQLAlchemy-migrate) into 
Alembic, and verify that it's still perfect.


5. Deprecate the SQLAlchemy-migrate version and announce that new 
migrations should be in Alembic.


#4 is hard work but not impossible.  I have some old code that does 
90% of the work, so we only have to do the other 90%.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Alembic support

[Alexei Kornienko]

Hello,

This conversion is actually quite simple. We are currently working to 
support alembic migrations in ceilometer:

https://blueprints.launchpad.net/ceilometer/+spec/convert-to-alembic

For now we agreed that conversion process will be the following:
1) Create folder for alembic migrations and configure engine correctly
2) Run alembic migrations after sqlalchemy-migrate (alembic creates a 
separate stamp table by default)

3) Create new migrations in alembic
4) Sync model with migrations* - 
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations

5) Remove old migration files after the end of support period (2 releases)

* Is needed to remove the need of base migration so the clean database 
could be created from models directly without the need of migrations.
This allows to simply drop old migrations without the need to compact 
them to one big migration (133_folsom.py in nova for example)


Please share your thoughts about proposed process.

Regards,
Alexei Kornienko

On 07/29/2013 02:13 AM, Jamie Lennox wrote:




- Original Message -

From: "Doug Hellmann" 
To: "OpenStack Development Mailing List" 
Sent: Saturday, 27 July, 2013 4:15:53 AM
Subject: Re: [openstack-dev] [Keystone] Alembic support




On Fri, Jul 26, 2013 at 2:04 PM, Adam Young < ayo...@redhat.com > wrote:



On 07/26/2013 01:55 PM, Doug Hellmann wrote:



It makes sense, though, if the data is going to be (potentially) coming from
separate databases. Is that any different for sqlalchemy-migrate?

As far as tracking the separate migrations for the different schemas, that's
handled by running "alembic init" for each schema to create a different
environment. The environments should then have unique "version_table" values
defined in the alembic.ini that the versions of each schema can be tracked
separately. I suggest alembic_version_${schema_owner} where schema_owner is
the subset of the schema (i.e., policy, tokens, or identity), or the
extension name.

It think that this will require enough of a change that I would like to do it
in Icehouse, and have a detailed blueprint written up for it.


That seems reasonable. Splitting one database into 3 (or more) will take some
consideration.

+1 I think we can leave this till Icehouse, though continue discussing the how.
I'm interested to know how the changeover will work.


Doug








On Fri, Jul 26, 2013 at 1:42 PM, Dolph Mathews < dolph.math...@gmail.com >
wrote:



Based on the docs, it looks like you need to start with separate sqlalchemy
engines with their own metadata instances for each environment you want to
migrate. That sounds like a significant refactor from where we are today
(everything shares keystone.common.sql.core.ModelBase).


On Thu, Jul 25, 2013 at 10:41 PM, Morgan Fainberg < m...@metacloud.com > wrote:


+1 to getting the multiple repos in place. Moving to Alembric later on in H
or even as the first commit of I should meet our goals to be on Alembric in
a reasonable timeframe. This also allows us to ensure we aren't rushing the
work to get our migration repos over to Alembric.

I think that allowing the extensions to have their own repos sooner is
better, and if we end up with an extension that has more than 1 or 2
migrations, we have probably accepted code that is far from fully baked (and
we should evaluate how that code made it in).

I am personally in favor of making the first commit of Icehouse (barring any
major issue) the point in which we move to Alembric. We can be selective in
taking extension modifications that add migration repos if it is a major
concern that moving to Alembric is going to be really painful.

Cheers,
Morgan Fainberg

On Thu, Jul 25, 2013 at 7:35 PM, Adam Young < ayo...@redhat.com > wrote:


I've been looking into Alembic support. It seems that there is one thing
missing that I was counting on: multiple migration repos. It might be
supported, but the docs are thin, and reports vary.

In the current Keystone implementation, we have a table like this:
mysql> desc migrate_version;
+-+--+--+-+-+---+
| Field | Type | Null | Key | Default | Extra |
+-+--+--+-+-+---+
| repository_id | varchar(250) | NO | PRI | NULL | |
| repository_path | text | YES | | NULL | |
| version | int(11) | YES | | NULL | |
+-+--+--+-+-+---+


Right now we only have one row in there:

keystone | /opt/stack/keystone/keystone/common/sql/migrate_repo | 0


However, we have been lumping all of our migrations together into a singel
repo, and we are just now looking to sort them out. For example, Policy,
Tokens, and Identity do not really need to share a database. As such, they
could go into separate migration repos, and it would keep changes to one
from stepping on changes to another, and avoiding the 

Re: [openstack-dev] [Ceilometer] Need help with Alembic...

[Alexei Kornienko]

Hello,

Please see my answers inline and my previous email regarding this topic:

Hello,

This conversion is actually quite simple. We are currently working to 
support alembic migrations in ceilometer:

https://blueprints.launchpad.net/ceilometer/+spec/convert-to-alembic

For now we agreed that conversion process will be the following:
1) Create folder for alembic migrations and configure engine correctly
2) Run alembic migrations after sqlalchemy-migrate (alembic creates a 
separate stamp table by default)

3) Create new migrations in alembic
4) Sync model with migrations* - 
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations 

5) Remove old migration files after the end of support period (2 
releases)


* Is needed to remove the need of base migration so the clean database 
could be created from models directly without the need of migrations.
This allows to simply drop old migrations without the need to compact 
them to one big migration (133_folsom.py in nova for example)


Please share your thoughts about proposed process.

Regards,
Alexei Kornienko 


Regards,
Alexei Kornienko

On 08/27/2013 07:30 PM, Jay Pipes wrote:

On 08/27/2013 04:32 AM, Boris Pavlovic wrote:

Jay,

I should probably share to you about our work around DB.

Migrations should be run only in production and only for production
backends (e.g. psql and mysql)
In tests we should use Schemas created by Models
(BASE.metadata.create_all())


Agree on both.


We are not able to use in this approach in moment  because we don't have
any mechanism to check that MODELS and SCHEMAS are EQUAL.
And actually MODELS and SCHEMAS are DIFFERENT.


Sorry, I don't understand the connection... how does not having a 
codified way of determining the difference between model and schema 
(BTW, this does exist in sqlalchemy-migrate... look at the 
compare_model_to_db method) not allow you to use metadata.create_all() 
in tests or mean that you can't run migrations only in production?
As Boris said we'll use 2 completely different ways to create DB schema 
in production and test environment. Cause of this we won't be able to 
guarantee that code is correct unless we'll have a dedicated test that 
will assure that we work with the same DB schema in both envs.



E.g. in Celiometer we have BP that syncs models and migration
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations 


(in other projects we are doing the same)

And also we are working around (oslo) generic tests that checks that
models and migrations are equal:
https://review.openstack.org/#/c/42307/


OK, cool.


So in our roadmap (in this case is):
1) Soft switch to alembic (with code that allows to have sqla-migrate
and alembic migration in the same time)


I don't see the point in this at all... I would rather see patches 
that just switch to Alembic and get rid of SQLAlchemy-migrate. Create 
an initial Alembic migration that has the last state of the database 
schema under SQLAlchemy-migrate... and then delete SA-Migrate.
It's not that simple as it seems. Please take into account that we have 
to keep SA-migrate + migrations during maintenance period for all 
projects. Cause of this we have to go the long way and keep both engines 
running before we'll be able to completely remove SA-migrate.





2) Sync Models and Migrations (fix DB schemas also)
3) Add from oslo generic test that checks all this stuff
4) Use BASE.create_all() for Schema creation instead of migrations.


This is already done in some projects, IIRC... (Glance used to be this 
way, at least)



But in OpenStack is not so simple to implement such huge changes, so it
take some time=)


Best regards,
Boris Pavlovic
---
Mirantis Inc.










On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes mailto:jaypi...@gmail.com>> wrote:

On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins) 
wrote:


Jay -

It looks there is an error in the migration script that causes
it to abort:

AttributeError: 'ForeignKeyConstraint' object has no attribute
'drop'

My guess is the migration runs on the first test, creates event
types
table fine, but exits with the above error, so migration is not
"complete". Thus every subsequent test tries to migrate the 
db, and

notices that event types already exists.


I'd corrected that particular mistake and pushed an updated
migration script.

Best,
-jay



-john

On 8/26/13 1:15 PM, "Jay Pipes" mailto:jaypi...@gmail.com>> wrote:

I just noticed that every single test case for SQL-driver
storage is
executing every single migration upgrade before every single
test case
run:

https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.p