Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

> On Nov 12, 2014, at 6:23 PM, Mike Bayer wrote: > > > If I may inquire as to the irrelevant complexity, I’m trying to pinpoint > where you see this happening. > > When we talk about updating a ComputeNode, because I’m only slightly familiar > with Nova’s codebase,

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

> On Nov 13, 2014, at 3:52 AM, Nikola Đipanov wrote: > > On 11/13/2014 02:45 AM, Dan Smith wrote: >>> I’m not sure if I’m seeing the second SELECT here either but I’m less >>> familiar with what I’m looking at. compute_node_update() does the >>> one SELECT as we said, then it doesn’t look li

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

> On Nov 13, 2014, at 5:47 AM, Matthew Booth wrote: > > Unfortunately this model doesn't apply to Nova objects, which are > persisted remotely. Unless I've missed something, SQLA doesn't run on > Nova Compute at all. Instead, when Nova Compute calls object.save() this > results in an RPC call to

Re: [openstack-dev] oslo.db 1.1.0 released

ion > 8bc1fb7 Use the six provided iterator mix-in > 436dfdc ModelsMigrationsSync:add correct server_default check for Enum > 2075074 Add history/changelog to docs > c9e5fdf Add run_cross_tests.sh script > > Thanks Andreas Jaeger, Ann Kamyshnikova, Christian Berendt, Davanum

Re: [openstack-dev] [Oslo][DB] oslo.db released

On 6/19/14, 3:56 PM, Doug Hellmann wrote: > On Thu, Jun 19, 2014 at 3:20 PM, Victor Sergeyev > wrote: >> Hello, Folks! >> >> Please, be informed, that the oslo.db library has been released and is >> available on PyPi. >> See [1] for the source code, [2] for the documentation. >> >> There is an e

[openstack-dev] [oslo] Openstack and SQLAlchemy

Hi all - For those who don't know me, I'm Mike Bayer, creator/maintainer of SQLAlchemy, Alembic migrations and Dogpile caching. In the past month I've become a full time Openstack developer working for Red Hat, given the task of carrying Openstack's database integration stor

Re: [openstack-dev] [oslo] Openstack and SQLAlchemy

it's pretty easy to solve). https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad On 6/30/14, 12:56 PM, Mike Bayer wrote: > Hi all - > > For those who don't know me, I'm Mike Bayer, creator/maintainer of > SQLAlchemy, Alembic migrations

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

On 7/4/14, 4:45 AM, Julien Danjou wrote: > On Thu, Jul 03 2014, Mark McLoughlin wrote: > >> We're attempting to take baby-steps towards moving completely from >> eventlet to asyncio/trollius. The thinking is for Ceilometer to be the >> first victim. > Thumbs up for the plan, that sounds like a goo

Re: [openstack-dev] [oslo] Openstack and SQLAlchemy

On 7/7/14, 3:57 PM, Matt Riedemann wrote: > > > > Regarding the eventlet + mysql sadness, I remembered this [1] in the > nova.db.api code. > > I'm not sure if that's just nova-specific right now, I'm a bit too > lazy at the moment to check if it's in other projects, but I'm not > seeing it in neut

Re: [openstack-dev] [nova] fastest way to run individual tests ?

On 7/9/14, 4:51 PM, Matt Riedemann wrote: > > > On 6/12/2014 6:17 AM, Daniel P. Berrange wrote: >> On Thu, Jun 12, 2014 at 07:07:37AM -0400, Sean Dague wrote: >>> On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: Does anyone have any tip on how to actually run individual tests in an effi

Re: [openstack-dev] Where should a test for eventlet and oslo.db interaction go?

On 7/10/14, 7:47 AM, Sean Dague wrote: > Honestly, that seems weird to me. > > oslo.db is built as a common layer for OpenStack services. > > eventlet is used by most OpenStack services. > > There are lots of known issues with eventlet vs. our db access patterns. > > Knowing that the db layer work

Re: [openstack-dev] [nova] fastest way to run individual tests ?

On 7/10/14, 12:08 PM, Chris Dent wrote: > On Thu, 10 Jul 2014, Mike Bayer wrote: > >> I typically never use tox or testtools at the commandline until I'm >> ready to commit and want to see what the jenkins builds will see. I >> start up the whole thing and then it

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

On 7/9/14, 10:59 AM, Roman Podoliaka wrote: > Hi all, > > Not sure what issues you are talking about, but I just replaced > "mysql" with "mysql+mysqlconnector" in my db connection string in > neutron.conf and "neutron-db-manage upgrade head" worked like a charm > for an empty schema. > > Ihar, co

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

On 7/11/14, 7:26 PM, Carl Baldwin wrote: > > > On Jul 11, 2014 5:32 PM, "Vishvananda Ishaya" mailto:vishvana...@gmail.com>> wrote: > > > > I have tried using pymysql in place of mysqldb and in real world concurrency > > tests against cinder and nova it performs slower. I was inspired by the mentio

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

On 7/11/14, 11:26 PM, Jay Pipes wrote: > Yep, couldn't agree more. > > Frankly, the steps you outline in the wiki above are excellent > examples of where we can make significant gains in both performance > and scalability. In addition to those you listed, the underlying > database schemas themselv

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

On Jul 14, 2014, at 9:46 AM, Jay Pipes wrote: > > The point of eventlet, I thought, was to hide the low-level stuff so that > developers could focus on higher-level (and more productive) abstractions. > Introducing asyncio contructs into the higher level code like Nova and > Neutron seems to

Re: [openstack-dev] [all] switch from mysqldb to another eventlet aware mysql client -- status of postgresql drivers?

On Jul 14, 2014, at 12:29 PM, Chris Friesen wrote: > On 07/09/2014 05:17 AM, Ihar Hrachyshka wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA512 >> >> Hi all, >> >> Multiple projects are suffering from db lock timeouts due to deadlocks >> deep in mysqldb library that we use to interac

Re: [openstack-dev] [all] switch from mysqldb to another eventlet aware mysql client -- status of postgresql drivers?

On Jul 14, 2014, at 1:02 PM, Chris Friesen wrote: > On 07/14/2014 10:41 AM, Mike Bayer wrote: >> >> On Jul 14, 2014, at 12:29 PM, Chris Friesen >> wrote: >> >>> On 07/09/2014 05:17 AM, Ihar Hrachyshka wrote: >>>> -BEGIN PGP SIGNED

Re: [openstack-dev] [Infra] MyISAM as a default storage engine for MySQL in the gate

OK, so, we aren’t generally running neutron tests w/ MySQL + InnoDB, right? I happen to be running them locally against a MySQL that defaults to InnoDB. And I’m trying to see if it’s deadlocking or not as I’m not able to get through them. All the eventlet + MySQLdb deadlock issues won’t be

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

On Aug 8, 2014, at 12:03 AM, Li Ma wrote: > > So, I'd like to propose a transparent read/write separation method > for oslo.db that every project may happily takes advantage of it > without any code modification. A single transaction begins, which is to emit a series of INSERT, UPDATE, and

Re: [openstack-dev] [nova] FKs in the DB

On 11/20/2015 11:19 AM, Alexis Lee wrote: > We just had a fun discussion in IRC about whether foreign keys are evil. > Initially I thought this was crazy but mordred made some good points. To > paraphrase, that if you have a scale-out app already it's easier to > manage integrity in your app than

Re: [openstack-dev] [nova] FKs in the DB

On 11/20/2015 02:29 PM, Mike Bayer wrote: > > > On 11/20/2015 11:19 AM, Alexis Lee wrote: >> We just had a fun discussion in IRC about whether foreign keys are evil. >> Initially I thought this was crazy but mordred made some good points. To >> paraphrase, that

Re: [openstack-dev] [nova] FKs in the DB

On 11/20/2015 04:33 PM, Clint Byrum wrote: > Excerpts from Mike Bayer's message of 2015-11-20 11:29:31 -0800: >> >> On 11/20/2015 11:19 AM, Alexis Lee wrote: >>> We just had a fun discussion in IRC about whether foreign keys are evil. >>> Initially I thought this was crazy but mordred made some g

Re: [openstack-dev] [oslo.db][sqlalchemy][mistral] Configuring default transaction isolation level

On 12/08/2015 07:28 AM, Renat Akhmerov wrote: > Hi, > > Moshe, thanks a lot for bringing this up. I remember I tried to find a > way to change isolation level per connection but also was unable to do that. Current SQLAlchemy has a lot of isolation level options.There is a complete guide to

[openstack-dev] [oslo] [keystone] rolling dogpile.core into dogpile.cache, removing namespace packaging (PLEASE REVIEW)

Hi all - Just a heads up what's happening for dogpile.cache, in version 0.6.0 we are rolling the functionality of the dogpile.core package into dogpile.cache itself, and retiring the use of namespace package naming for dogpile.cache. Towards retiring the use of namespace packaging, the magic

Re: [openstack-dev] [oslo] [keystone] rolling dogpile.core into dogpile.cache, removing namespace packaging (PLEASE REVIEW)

Just a reminder, dogpile.cache is doing away with namespace packaging in version 0.6.0, due for the end of this week or sometime next week. dogpile.core is being retired and left as-is. No changes should be needed by anyone using only dopgile.cache. On 05/30/2016 06:17 PM, Mike Bayer

[openstack-dev] [oslo] [keystone] dogpile.cache 0.6.0 released

n using dogpile.core directly (this should be very rare). Changelog for 0.6.0 is at: http://dogpilecache.readthedocs.io/en/latest/changelog.html#change-0.6.0 On 06/01/2016 04:54 PM, Mike Bayer wrote: Just a reminder, dogpile.cache is doing away with namespace packaging in version 0.6.0, due

Re: [openstack-dev] [nova] Distributed Database

On 04/22/2016 04:27 PM, Ed Leafe wrote: OK, so I know that Friday afternoons are usually the worst times to write a blog post and start an email discussion, and that the Friday immediately before a Summit is the absolute worst, but I did it anyway. http://blog.leafe.com/index.php/2016/04/22/di

Re: [openstack-dev] [oslo.config] Encrypt the sensitive options

On 04/26/2016 09:32 AM, Daniel P. Berrange wrote: IMHO encrypting oslo config files is addressing the wrong problem. Rather than having sensitive passwords stored in the main config files, we should have them stored completely separately by a secure password manager of some kind. The config fi

Re: [openstack-dev] [nova] Distributed Database

On 04/28/2016 08:44 AM, Edward Leafe wrote: On Apr 24, 2016, at 3:28 PM, Robert Collins wrote: For instance, the things I think are essential for a distributed database based datastore: - good single-machine developer story. Must not need a physical cluster to hack on OpenStack - deal gracef

Re: [openstack-dev] [nova] Distributed Database

On 04/28/2016 08:25 PM, Edward Leafe wrote: Your own tests showed that a single RDBMS instance doesn’t even break a sweat under your test loads. I don’t see why we need to shard it in the first place, especially if in doing so we add another layer of complexity and another dependency in order

Re: [openstack-dev] [Fuel][MySQL][DLM][Oslo][DB][Trove][Galera][operators] Multi-master writes look OK, OCF RA and more things

On 04/30/2016 02:57 AM, bdobre...@mirantis.com wrote: Hi Roman. That's interesting, although’s hard to believe (there is no slave lag in galera multi master). I can only suggest us to create another jepsen test to verify exactly scenario you describe. As well as other OpenStack specific pattern

Re: [openstack-dev] [Fuel][MySQL][DLM][Oslo][DB][Trove][Galera][operators] Multi-master writes look OK, OCF RA and more things

On 04/30/2016 10:50 AM, Clint Byrum wrote: Excerpts from Roman Podoliaka's message of 2016-04-29 12:04:49 -0700: I'm curious why you think setting wsrep_sync_wait=1 wouldn't help. The exact example appears in the Galera documentation: http://galeracluster.com/documentation-webpages/mysqlw

Re: [openstack-dev] [nova] Distributed Database

On 05/02/2016 07:38 AM, Matthieu Simonin wrote: As far as we understand the idea of an ORM is to hide the relational database with an Object oriented API. I actually disagree with that completely. The reason ORMs are so maligned is because of this misconception; developer attempts to use

Re: [openstack-dev] [nova] Distributed Database

On 05/02/2016 01:48 PM, Clint Byrum wrote: FWIW, I agree with you. If you're going to use SQLAlchemy, use it to take advantage of the relational model. However, how is what you describe a win? Whether you use SELECT .. FOR UPDATE, or a stored procedure, the lock is not distributed, and thus

Re: [openstack-dev] [neutron] Seeing db lockout issues in neutron add_router_interface

On 05/10/2016 04:57 PM, Divya wrote: Hi, I am trying to run this rally test on stable/kilo https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/neutron/create_and_delete_routers.json with concurrency 50 and iterations 2000. This test basically cretaes routers and subnets

Re: [openstack-dev] [Fuel] PostgreSQL 9.3 and JSON operations

On 12/15/2015 07:20 AM, Igor Kalnitsky wrote: > Hey Julien, > >> https://blueprints.launchpad.net/fuel/+spec/openstack-ha-fuel-postgresql > > I believe this blueprint is about DB for OpenStack cloud (we use > Galera now), while here we're talking about DB backend for Fuel > itself. Fuel has a s

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

On 12/16/2015 09:10 AM, Sylvain Bauza wrote: > > > Le 16/12/2015 14:59, Sean Dague a écrit : >> oslo.db test_migrations is using methods for alembic, which changed in >> the 0.8.4 release. This ends up causing a unit test failure (at least in >> the Nova case) that looks like this - >> http://l

Re: [openstack-dev] [Fuel] PostgreSQL 9.3 and JSON operations

is one 24/7 if I don't do that, JSON datatypes are huge (how often does MySQL add a datatype). > > Thanks, > - Igor > > On Tue, Dec 15, 2015 at 5:06 PM, Mike Bayer wrote: >> >> >> On 12/15/2015 07:20 AM, Igor Kalnitsky wrote: >>> Hey Julien, &g

Re: [openstack-dev] [Fuel] PostgreSQL 9.3 and JSON operations

here's already a mostly finished PR for SQLAlchemy support in the > queue. > > Does it mean SQLAlchemy will have one unified interface to make JSON > queries? So we can use different backends if necessary? > > Thanks, > - Igor >

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

On 12/16/2015 11:37 AM, Sean Dague wrote: > On 12/16/2015 11:22 AM, Mike Bayer wrote: >> >> >> On 12/16/2015 09:10 AM, Sylvain Bauza wrote: >>> >>> >>> Le 16/12/2015 14:59, Sean Dague a écrit : >>>> oslo.db test_migrations is using

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

On 12/16/2015 11:53 AM, Sean Dague wrote: > On 12/16/2015 11:37 AM, Sean Dague wrote: >> On 12/16/2015 11:22 AM, Mike Bayer wrote: >>> >>> >>> On 12/16/2015 09:10 AM, Sylvain Bauza wrote: >>>> >>>> >>>> Le 16/12/2015 14:59,

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

On 12/16/2015 01:32 PM, Jeremy Stanley wrote: > On 2015-12-16 11:03:38 -0700 (-0700), Carl Baldwin wrote: > [...] >> We need to vet new package releases before they wreak havoc. We need >> to accept new package releases by proposing a patch to update the >> version and take it through the gate.

Re: [openstack-dev] [gate] any project using olso.db test_migrations is currently blocked

On 12/17/2015 04:00 PM, Thomas Goirand wrote: > On 12/16/2015 06:04 PM, Mike Bayer wrote: >> >> >> On 12/16/2015 11:53 AM, Sean Dague wrote: >>> On 12/16/2015 11:37 AM, Sean Dague wrote: >>>> On 12/16/2015 11:22 AM, Mike Bayer wrote: >>>>>

[openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

Hi all - Let me start out with the assumptions I'm going from for what I want to talk about. 1. I'm looking at Nova right now, but I think similar things are going on in other Openstack apps. 2. Settings that we see in nova.conf, including: #wsgi_default_pool_size = 1000 #max_pool_size = #max_

Re: [openstack-dev] Nova scheduler startup when database is not available

On 12/23/2015 01:32 PM, Jay Pipes wrote: > On 12/23/2015 12:27 PM, Lars Kellogg-Stedman wrote: >> I've been looking into the startup constraints involved when launching >> Nova services with systemd using Type=notify (which causes systemd to >> wait for an explicit notification from the service b

Re: [openstack-dev] The command "neutron-db-manage" of 8.0.0~b1 fails

On 01/03/2016 05:15 PM, Martinx - ジェームズ wrote: > Guys, > > I'm trying to experiment Mitaka on Ubuntu Xenial, which already have > beta version on its repositories, however, "neutron-db-manage" fails. > > Here is the output of it: > > http://paste.openstack.org/show/482920/ > > Any clue?

Re: [openstack-dev] The command "neutron-db-manage" of 8.0.0~b1 fails

On 01/04/2016 06:59 AM, Ihar Hrachyshka wrote: > Martinx - ジェームズ wrote: > >> Guys, >> >> I'm trying to experiment Mitaka on Ubuntu Xenial, which already have >> beta version on its repositories, however, "neutron-db-manage" fails. >> >> Here is the output of it: >> >> http://paste.openstack.

Re: [openstack-dev] The command "neutron-db-manage" of 8.0.0~b1 fails

On 01/03/2016 10:57 PM, Martinx - ジェームズ wrote: > On 4 January 2016 at 01:28, Mike Bayer wrote: >> >> >> On 01/03/2016 05:15 PM, Martinx - ジェームズ wrote: >>> Guys, >>> >>> I'm trying to experiment Mitaka on Ubuntu Xenial, which already have &g

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/06/2016 09:11 AM, Roman Podoliaka wrote: > Hi Mike, > > Thank you for this brilliant analysis! We've been seeing such timeout > errors in downstream periodically and this is the first time someone > has analysed the root cause thoroughly. > > On Fri, Dec 18, 20

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/07/2016 11:02 AM, Sean Dague wrote: > On 01/07/2016 09:56 AM, Brant Knudson wrote: >> >> >> On Thu, Jan 7, 2016 at 6:39 AM, Clayton O'Neill > > wrote: >> >> On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka >> mailto:rpodoly...@mirantis.com>> wrote: >> >

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/08/2016 04:44 AM, Radomir Dopieralski wrote: > On 01/07/2016 05:55 PM, Mike Bayer wrote: > >> but also even if you're under something like >> mod_wsgi, you can spawn a child process or worker thread regardless. >> You always have a Python interpreter runn

Re: [openstack-dev] [neutron] InvalidRequestError: This session is in 'prepared' state; no further SQL can be emitted within this transaction.

On 01/11/2016 03:58 AM, Koteswar wrote: > Hi All, > > > > In my mechanism driver, I am reading/writing into sql db in a fixed > interval looping call. Sometimes I get the following error when I stop > and start neutron server > > InvalidRequestError: This session is in 'prepared' state; no f

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/11/2016 05:39 AM, Radomir Dopieralski wrote: > On 01/08/2016 09:51 PM, Mike Bayer wrote: >> >> >> On 01/08/2016 04:44 AM, Radomir Dopieralski wrote: >>> On 01/07/2016 05:55 PM, Mike Bayer wrote: >>> >>>> but also even if you're u

Re: [openstack-dev] [oslo][oslo.versionedobjects] Is it possible to make changes to oslo repos?

On 01/28/2016 01:52 PM, Doug Hellmann wrote: > Excerpts from Hayes, Graham's message of 2016-01-28 17:01:09 +: >> Recently I tried started to use oslo.versionedobjects for a project. >> >> After playing around with it for a while, I noticed I could set "this is >> not a uuid" as the value of

Re: [openstack-dev] [neutron] Plethora of dbase migration questions...

On 7/7/15 1:28 PM, Paul Michali wrote: HEAD, head, 24f28869838b (my new file) all say the same thing. :( On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando > wrote: possibly I was wrong in mixing up git & alembic. It should be "upgrade head" - lowercase.

Re: [openstack-dev] [testing] moving testrepository *outside* the tox venv

On 7/11/15 6:04 AM, Robert Collins wrote: Wearing my upstream hat, testr is *still* intended to be used differently than OpenStack is doing. Running all the tests for all python versions at once in parallel is the sort of thing testr is aimed at, and thats fairly fundamentally incompatible wi

Re: [openstack-dev] [neutron][db] online-schema-migrations patch landed

On 7/15/15 9:26 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi all, since it's a high impact change in the migration tree, I wanted to drop an email to everyone affected (basically, anyone who wants to introduce a new migration from now on). So there was a prop

[openstack-dev] [nova] [oslo] troubling passing of unit tests on broken code

Just a heads up that this recently merged code is wrong: https://review.openstack.org/#/c/192760/14/nova/tests/unit/db/test_migrations.py,cm and here it is failing tests on my local env, as it does on my CI, as would be expected, there's a lot more if I keep it running: http://paste.openstack

Re: [openstack-dev] [nova] [oslo] troubling passing of unit tests on broken code

Column() in the remove operation has no Table associated with it. But the code is still wrong and should be fixed. -- dims On Fri, Aug 7, 2015 at 6:42 PM, Mike Bayer <mailto:mba...@redhat.com>> wrote: Just a heads up that this recently merged code is wrong:

Re: [openstack-dev] In memory joins in Nova

On 8/11/15 7:14 PM, Sachin Manpathak wrote: I am struggling with python code profiling in general. It has its own caveats like 100% plus overhead. However, on a host with only nova services (DB on a different host), I see cpu utilization spike up quickly with scale. The DB server is relativel

Re: [openstack-dev] In memory joins in Nova

On 8/12/15 1:49 PM, Sachin Manpathak wrote: Thanks, This feedback was helpful. Perhaps my paraphrasing was misleading. I am not running openstack at scale in order to see how much the DB can sustain. My observation was that the host running nova services saturates on CPU much earlier than th

Re: [openstack-dev] In memory joins in Nova

On 8/12/15 10:29 PM, Clint Byrum wrote: Excerpts from Dan Smith's message of 2015-08-12 23:12:23 +0800: If OTOH we are referring to the width of the columns and the join is such that you're going to get the same A identity over and over again, if you join A and B you get a "wide" row with all

Re: [openstack-dev] [magnum] versioned objects changes

On 8/26/15 4:47 AM, Grasza, Grzegorz wrote: Hi, I noticed that right now, when we make changes (adding/removing fields) in https://github.com/openstack/magnum/tree/master/magnum/objects , we don't change object versions. The idea of objects is that each change in their fields should be

Re: [openstack-dev] [oslo.db][sqlalchemy] rollback after commit

On 9/16/15 10:16 AM, Roman Podoliaka wrote: Hi Gareth, Right, 'SELECT 1' issued at the beginning of every transaction is a pessimistic check to detect disconnects early. oslo.db will create a new DB connection (as well as invalidate all the existing connections to the same DB in the pool) and

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/14/15 11:58 AM, Doug Hellmann wrote: At one point we were exploring having both sqlalchemy-migrate and alembic run, one after the other, so that we only need to create new migrations with alembic and do not need to change any of the existing migrations. Was that idea dropped? to my know

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/14/15 7:12 PM, Angus Salkeld wrote: On Fri, May 15, 2015 at 4:46 AM, Mike Bayer <mailto:mba...@redhat.com>> wrote: On 5/14/15 11:58 AM, Doug Hellmann wrote: At one point we were exploring having both sqlalchemy-migrate and alembic run, one after the

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/15/15 9:31 AM, Doug Hellmann wrote: This seems more complicated than needed. If we just stop writing the sqlalchemy-migrate scripts and don't change them, then for 1 cycle we have to run both sets of migrations and after that we can just run alembic. Then we have a forever-in-perpetuity de

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/14/15 10:34 PM, Angus Salkeld wrote: Thanks. Would you suggest we hold off moving to alembic (in Heat) until you have this ironed out? I just want to make sure we don't do this prematurely. If you need to support running an Alembic-version of Heat against an existing database in mid-

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/15/15 1:13 PM, Mike Bayer wrote: On 5/15/15 9:31 AM, Doug Hellmann wrote: This seems more complicated than needed. If we just stop writing the sqlalchemy-migrate scripts and don't change them, then for 1 cycle we have to run both sets of migrations and after that we can jus

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/16/15 10:05 AM, Doug Hellmann wrote: Excerpts from Mike Bayer's message of 2015-05-15 13:13:39 -0400: On 5/15/15 9:31 AM, Doug Hellmann wrote: This seems more complicated than needed. If we just stop writing the sqlalchemy-migrate scripts and don't change them, then for 1 cycle we have t

[openstack-dev] [oslo] needed, driver for oslo.db thursday session

Hello - It is my extreme displeasure and frustration to announce that due to an incredibly unfortunate choice of airline, I had to cancel my entire trip to the Openstack summit after spending 26 hours in my home airport waiting for my airline to produce a working airplane (which they did not).

Re: [openstack-dev] [oslo] needed, driver for oslo.db thursday session

from Oslo team can replace Mike as a session driver. Thanks, Roman On Tue, May 19, 2015 at 3:53 AM, Mike Bayer wrote: Hello - It is my extreme displeasure and frustration to announce that due to an incredibly unfortunate choice of airline, I had to cancel my entire trip to the Openstack su

Re: [openstack-dev] [oslo] needed, driver for oslo.db thursday session

On 5/20/15 9:31 AM, Davanum Srinivas wrote: Thanks Jeremy, Mike, Roman, Victor, Please see remote connection details in: https://etherpad.openstack.org/p/YVR-oslo-db-plans The schedule time for the session is in: https://libertydesignsummit.sched.org/event/3571aa54b364c62e097da8cd32d97258 Ho

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/14/15 11:40 AM, Mike Bayer wrote: The "online schema changes" patch has been abandoned.I regret that I was not able to review the full nature of this spec in time to note some concerns I have, namely that Alembic does not plan on ever acheiving 100% "automatio

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/21/15 12:28 PM, Jay Pipes wrote: Hi Mike, I had some initial concerns around the online db schema migration work as you do below. However, once I realized one big thing, those concerns were allayed. Here's the big thing that you're missing: no data migrations are allowed any more in

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/21/15 2:50 PM, Mike Bayer wrote: Put another way, if I want to do a migration that someday, "over time", eventually, etc., will drop the "widget_status" table and merge it into the "widget" table, how do I write the "get_widget()" DB API ca

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/21/15 2:57 PM, Mike Bayer wrote: On 5/21/15 2:50 PM, Mike Bayer wrote: Put another way, if I want to do a migration that someday, "over time", eventually, etc., will drop the "widget_status" table and merge it into the "widget" table, how do I write t

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

On 5/21/15 3:36 PM, Mike Bayer wrote: On 5/21/15 2:57 PM, Mike Bayer wrote: On 5/21/15 2:50 PM, Mike Bayer wrote: Put another way, if I want to do a migration that someday, "over time", eventually, etc., will drop the "widget_status" table and merge it into the &q

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

OK, have just gotten off a chat with the folks at summit. I am glad that I've managed to get my concerns about this approach out there. For people reading my notes here, I've gotten the answer to my question about how database access code is written for a system that is moving from some part

Re: [openstack-dev] [Neutron][DB] neutron DB migration scripts

On 5/25/15 10:24 AM, Henry Gessau wrote: Yes, unfortunately the autogenerate currently generates commands to drop all the FWaaS, LBaaS and VPNaaS tables since their models are not in the neutron tree. You can and should delete all these commands that are not related to your new models. We hav

Re: [openstack-dev] [Neutron][DB] neutron DB migration scripts

On 5/25/15 12:34 PM, Armando M. wrote: One thing I would like to point out is that in this cycle we'll be working extensively in this area to make the very task you are working on easier to deal with, and better documented. This will fall under the umbrella of the blueprint [1]. HTH Arm

Re: [openstack-dev] [Neutron][DB] neutron DB migration scripts

On 5/25/15 10:35 PM, Armando M. wrote: On 25 May 2015 at 09:46, Mike Bayer <mailto:mba...@redhat.com>> wrote: On 5/25/15 12:34 PM, Armando M. wrote: One thing I would like to point out is that in this cycle we'll be working extensively in this area to make

Re: [openstack-dev] [all] cross project communication: Return request-id to caller

On 5/27/15 3:06 AM, Kekane, Abhishek wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each s

Re: [openstack-dev] [all] Switching to SQLAlchemy 1.0.x

On 6/9/15 9:26 AM, Thomas Goirand wrote: Hi, The python-sqlalchemy package has been uploaded to Debian Experimental, and is about to be uploaded to Debian Unstable. So I wonder what's the state of the project regarding upgrading SQLA. Maybe Mike can tell what kind of issue we may run into? Ho

Re: [openstack-dev] [all] Switching to SQLAlchemy 1.0.x

On 6/9/15 1:08 PM, Mike Bayer wrote: So for Openstack, we would mostly worry about code that is doing things oddly in some unexpected way, however all the Openstack code I've seen tends to be very ORM centric and uses the ORM very conservatively so I don't anticipate any pro

Re: [openstack-dev] [all] Switching to SQLAlchemy 1.0.x

On 6/10/15 3:26 AM, Thomas Goirand wrote: Hi Mike, Thanks a lot for your quick reply which is very useful to me. On 06/09/2015 07:08 PM, Mike Bayer wrote: On 6/9/15 9:26 AM, Thomas Goirand wrote: Hi, The python-sqlalchemy package has been uploaded to Debian Experimental, and is about to

Re: [openstack-dev] Online Migrations.

On 6/10/15 4:34 PM, Dan Smith wrote: The remaining work is to have a way of preventing database contracts Foreign Keys. I'm not sure if SQLAlchemy will balk at this or not, but we could do something like: class NameRemovedCondition(object): def satisfied(self): # Check t

Re: [openstack-dev] [all] Switching to SQLAlchemy 1.0.x

On 6/10/15 5:18 PM, Thomas Goirand wrote: On 06/10/2015 04:52 PM, Mike Bayer wrote: whew! :) Do you know why it fails with Kilo? https://review.openstack.org/#/c/190062/ I noticed this, I don't know very well how the requirements get distributed around but seems like it is not happ

Re: [openstack-dev] [all][python3] use of six.iteritems()

On 6/10/15 11:48 PM, Dolph Mathews wrote: tl;dr *.iteritems() is faster and more memory efficient than .items() in python2* Using xrange() in python2 instead of range() because it's more memory efficient and consistent between python 2 and 3... # xrange() + .items() python -m timeit -n 20

Re: [openstack-dev] [Neutron] Proposing Ann Kamyshnikova for the API & DB core reviewer team

+1 to Ann ! On 6/11/15 10:34 AM, Henry Gessau wrote: As one of the Lieutenants [1] for the API and DB areas under the PTL, I would like to propose Ann Kamyshnikova as a member of the Neutron API and DB core reviewer team. Ann has been a long time contributor in Neutron showing expertise partic

Re: [openstack-dev] [all][python3] use of six.iteritems()

On 6/11/15 1:39 PM, Dolph Mathews wrote: On Thu, Jun 11, 2015 at 12:34 AM, Robert Collins mailto:robe...@robertcollins.net>> wrote: On 11 June 2015 at 17:16, Robert Collins mailto:robe...@robertcollins.net>> wrote: > This test conflates setup and execution. Better like my examp

Re: [openstack-dev] [Neutron] Issue with pymysql

On 6/11/15 9:32 PM, Eugene Nikanorov wrote: Hi neutrons, I'd like to draw your attention to an issue discovered by rally gate job: http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=TRACE I don't have bandwidth to take a deep look

Re: [openstack-dev] [Neutron] Issue with pymysql

On 6/12/15 6:42 AM, Sean Dague wrote: On 06/12/2015 06:31 AM, Joe Gordon wrote: On Fri, Jun 12, 2015 at 7:13 PM, Sean Dague mailto:s...@dague.net>> wrote: On 06/12/2015 01:17 AM, Salvatore Orlando wrote: > It is however interesting that both "lock wait timeouts" and "missing >

Re: [openstack-dev] [Neutron] Issue with pymysql

On 6/12/15 11:37 AM, Mike Bayer wrote: On 6/11/15 9:32 PM, Eugene Nikanorov wrote: Hi neutrons, I'd like to draw your attention to an issue discovered by rally gate job: http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=

Re: [openstack-dev] Online Migrations.

On 6/15/15 9:21 AM, Philip Schwartz wrote: This weekend, I discussed the requested change at length with Mike. I think before moving forward, we need a better understanding of what is trying to be achieved. Request: Add the ability to verify migrations are completed prior to contract. As

Re: [openstack-dev] Online Migrations.

On 6/15/15 2:21 PM, Dan Smith wrote: Tying this to the releases is less desirable from my perspective. It means that landing a thing requires more than six months of developer and reviewer context. We have that right now, and we get along, but it's much harder to plan, execute, and cleanup t

Re: [openstack-dev] Online Migrations.

On 6/15/15 4:21 PM, Andrew Laski wrote: On 06/15/15 at 03:23pm, Mike Bayer wrote: 1. at runtime? e.g. your nova service is running, it's doing "SELECT x, y FROM thing", then some magic thing happens somewhere and the app suddenly sees, hey "y" is gone! change

Re: [openstack-dev] Online Migrations.

On 6/15/15 6:37 PM, Mike Bayer wrote: On 6/15/15 4:21 PM, Andrew Laski wrote: If I had to visualize what an approach looks like that does this somewhat cleanly, other than just putting off contract until the API has naturally moved beyond it, it would involve a fixed and structured

Re: [openstack-dev] [nova] Online Migrations.

ure of tables. -Ph On Jun 15, 2015, at 8:06 PM, Mike Bayer wrote: On 6/15/15 6:37 PM, Mike Bayer wrote: On 6/15/15 4:21 PM, Andrew Laski wrote: If I had to visualize what an approach looks like that does this somewhat cleanly, other than just putting off contract until the API

<    1   2   3   4   >