[openstack-dev] [oslo.db] Stepping down from core

2017-06-11 Thread Roman Podoliaka
Hi all, I recently changed job and hasn't been able to devote as much time to oslo.db as it is expected from a core reviewer. I'm no longer working on OpenStack, so you won't see me around much. Anyway, it's been an amazing experience to work with all of you! Best of luck! And see ya at various P

Re: [openstack-dev] [oslo][requirements][all] requesting assistance to unblock SQLAlchemy 1.1 from requirements

2017-03-15 Thread Roman Podoliaka
Isn't the purpose of that specific job - gate-tempest-dsvm-neutron-src-oslo.db-ubuntu-xenial-ocata - to test a change to the library master branch with stable releases (i.e. Ocata) - of all other components? On Wed, Mar 15, 2017 at 5:20 PM, Sean Dague wrote: > On 03/15/2017 10:38 AM, Mike Bayer w

Re: [openstack-dev] [nova] [placement] experimenting with extracting placement

2017-03-14 Thread Roman Podoliaka
Hi Matt, On Tue, Mar 14, 2017 at 5:27 PM, Matt Riedemann wrote: > We did agree to provide an openstackclient plugin purely for CLI > convenience. That would be in a separate repo, not part of nova or > novaclient. I've started a blueprint [1] for tracking that work. *The > placement osc plugin bl

Re: [openstack-dev] [oslo_reports]Does GMR work for uwsgi mode service?

2017-03-07 Thread Roman Podoliaka
Hi, My understanding is that it's not recommended for WSGI apps to set up custom signal handlers. The reason for that is that a WSGI server (i.e. uwsgi in your case or Apache+mod_wsgi) will most likely have its own handlers for the very same set of signals [1]. There is an alternative way to trig

Re: [openstack-dev] [all][requirements] Eventlet verion bump coming?

2017-03-01 Thread Roman Podoliaka
Hi Tony, I'm ready to help with this! The version we use now (0.19.0) has (at least) 2 known issues: - recv_into() >8kb from an SSL wrapped socket hangs [1] - adjusting of system clock backwards makes periodic tasks hang [2] so it'd be great to allow for newer releases in upper-constraints. Th

Re: [openstack-dev] [oslo][oslo.db] MySQL Cluster support

2017-02-03 Thread Roman Podoliaka
On Fri, Feb 3, 2017 at 4:41 PM, Mike Bayer wrote: > Not sure if people on the list are seeing that we are simultaneously talking > about getting rid of Postgresql in the efforts to support only "one > database", while at the same time adding one that is in many ways a new > database. ++ and, FWI

Re: [openstack-dev] [nova] Different length limit for tags in object definition and db model definition

2017-01-17 Thread Roman Podoliaka
Hi all, Changing the type of column from VARCHAR(80) to VARCHAR(60) would also require a data migration (i.e. a schema migration to add a new column with the "correct" type, changes to the object, data migration logic) as it is not an "online" DDL operation according to [1]. Adding a new API micr

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

2016-10-03 Thread Roman Podoliaka
Timofei, On Mon, Oct 3, 2016 at 10:11 AM, Timofei Durakov wrote: > Hi team, > Taking that into account, the > question here would be: why not to store all required information(e.g. boot > order) in DB instead? I think, we definitely could do that, just like we currently preserve the order of NI

Re: [openstack-dev] [cinder][db] lazy loading of an attribute impossible

2016-09-30 Thread Roman Podoliaka
Michał, You are absolutely right: this exception is raised when you try to lazy-load instance attributes outside a Session scope. There is an obvious problem with that - instances do not communicate with a DB on their own - it's left up to Session [1]. Unfortunately, it does not play nicely with

Re: [openstack-dev] [oslo.db] [release] opportunistic tests breaking randomly

2016-09-21 Thread Roman Podoliaka
FWIW, there was no new failures in Nova jobs since then. I'm confused as well why these tests would sporadically take much longer time to execute. Perhaps we could install something like atop on our nodes to answer that question. On Wed, Sep 21, 2016 at 5:46 PM, Ihar Hrachyshka wrote: > I just h

Re: [openstack-dev] [oslo.db] [release] opportunistic tests breaking randomly

2016-09-15 Thread Roman Podoliaka
wal.html). Similar settings must be available for MySQL as well. Thanks, Roman On Thu, Sep 15, 2016 at 3:07 PM, Sean Dague wrote: > On 09/15/2016 05:52 AM, Roman Podoliaka wrote: >> Mike, >> >> On Thu, Sep 15, 2016 at 5:48 AM, Mike Bayer wrote: >> >>> * Prior to os

Re: [openstack-dev] [oslo.db] [release] opportunistic tests breaking randomly

2016-09-15 Thread Roman Podoliaka
Mike, I think the exact error (InterfaceError vs TimeoutException) varies depending on what code is being executed at the very moment when a process receives SIGALRM. I tried to run the tests against PostgreSQL passing very small timeout values (OS_TEST_TIMEOUT=5 python -m testtools.run nova.test

Re: [openstack-dev] [oslo.db] [release] opportunistic tests breaking randomly

2016-09-15 Thread Roman Podoliaka
Mike, On Thu, Sep 15, 2016 at 5:48 AM, Mike Bayer wrote: > * Prior to oslo.db 4.13.3, did we ever see this "timeout" condition occur? > If so, was it also accompanied by the same "resource closed" condition or > did this second part of the condition only appear at 4.13.3? > * Did we see a simila

Re: [openstack-dev] [oslo.db] [release] opportunistic tests breaking randomly

2016-09-14 Thread Roman Podoliaka
Hmm, looks like we now run more testr workers in parallel (8 instead of 4): http://logs.openstack.org/76/335676/7/check/gate-nova-python34-db/6841fce/console.html.gz http://logs.openstack.org/62/369862/3/check/gate-nova-python27-db-ubuntu-xenial/2784de9/console.html On my machine running Nova mig

Re: [openstack-dev] [oslo.db] [release] opportunistic tests breaking randomly

2016-09-14 Thread Roman Podoliaka
Sean, I'll take a closer look, but test execution times and errors look suspicious: ironic.tests.unit.db.sqlalchemy.test_migrations.TestMigrationsPostgreSQL.test_walk_versions 60.002 2016-09-14 14:21:38.756421 | File "/home/jenkins/workspace/gate-ironic-python27-db-ubuntu-xenial/.tox/py27/

Re: [openstack-dev] [nova] using the qemu memory overhead in memory resource tracking or scheduling

2016-09-06 Thread Roman Podoliaka
Looks like we already have something like that in the virt drivers interface: https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L205-L216 which is used in the resource tracker. On Tue, Sep 6, 2016 at 10:40 AM, Balázs Gibizer wrote: > Hi, > > In our cloud we use 1G huge pages for

Re: [openstack-dev] [nova] resignation from bug czar role

2016-09-05 Thread Roman Podoliaka
+1 I'll be happy to help with triaging of new bugs and reviews of bug fixes. On Mon, Sep 5, 2016 at 7:33 PM, Timofei Durakov wrote: > Hi, folks, > > Thanks, Markus, for doing this job! I'm interested in this activity. > > Timofey > > On Mon, Sep 5, 2016 at 7:20 PM, Sylvain Bauza wrote: >> >> >>

Re: [openstack-dev] [nova] [placement] unresolved topics in resource providers/placement api

2016-07-28 Thread Roman Podoliaka
Hi Chris, A really good summary, thank you! On Thu, Jul 28, 2016 at 4:57 PM, Chris Dent wrote: > It's pretty clear that were going to need at least an interim and > maybe permanent endpoint that returns a list of candidate target > resource providers. This is because, at least initially, the > p

Re: [openstack-dev] [oslo.db] [CC neutron] CIDR overlap functionality and constraints

2016-07-19 Thread Roman Podoliaka
Mike, On Tue, Jul 19, 2016 at 7:40 AM, Mike Bayer wrote: > Note that I use the term "function" and not "procedure" to stress that this > is not a "stored procedure" in the traditional sense of performing complex > business logic and persistence operations - this CIDR function performs a > calcula

Re: [openstack-dev] [all][Python 3.4-3.5] Async python clients

2016-07-04 Thread Roman Podoliaka
That's exactly what https://github.com/koder-ua/os_api is for: it polls status changes in a separate thread and then updates the futures, so that you can wait on multiple futures at once. On Mon, Jul 4, 2016 at 2:19 PM, Denis Makogon wrote: > > > 2016-07-04 13:22 GMT+03:00 R

Re: [openstack-dev] [Openstack-operators] [nova] Fail build request if we can't inject files?

2016-07-04 Thread Roman Podoliaka
Hi all, > Won't the user provided files also get made available by the config drive / > metadata service ? I believe, they should. Not sure it's the same problem, so just FYI: we recently encountered an issue with VFAT formatted config drives when nova-compute is deployed on CentOS or RHEL: htt

Re: [openstack-dev] [all][Python 3.4-3.5] Async python clients

2016-07-04 Thread Roman Podoliaka
Denis, > Major problem > appears when you trying to provision resource that requires to have some > time to reach ACTIVE/COMPLETED state (like, nova instance, stack, trove > database, etc.) and you have to use polling for status changes and in > general polling requires to send HTTP requests with

Re: [openstack-dev] [Neutron] Getting rid of lazy init for engine facade

2016-05-11 Thread Roman Podoliaka
Hi Anna, Thank you for working on this in Neutron! EngineFacade is initialized lazily internally - you don't have to do anything for that in Neutron (you *had to* with "old" EngineFacade - this is the boiler plate your patch removes). I believe, you should be able to call configure(...) uncondit

Re: [openstack-dev] [Fuel][MySQL][DLM][Oslo][DB][Trove][Galera][operators] Multi-master writes look OK, OCF RA and more things

2016-04-29 Thread Roman Podoliaka
Hi Bogdan, Thank you for sharing this! I'll need to familiarize myself with this Jepsen thing, but overall it looks interesting. As it turns out, we already run Galera in multi-writer mode in Fuel unintentionally in the case, when the active MySQL node goes down, HAProxy starts opening connection

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-02-23 Thread Roman Podoliaka
That's what I tried first :) For some reason load distribution was still uneven. I'll check this again, maybe I missed something. On Tue, Feb 23, 2016 at 5:37 PM, Chris Friesen wrote: > On 02/23/2016 05:25 AM, Roman Podoliaka wrote: > >> So looks like it's two rel

Re: [openstack-dev] [oslo] oslo.db reset session?

2016-02-23 Thread Roman Podoliaka
On Tue, Feb 23, 2016 at 7:23 PM, Mike Bayer wrote: > Also I'm not > sure how the enginefacade integration with nova didn't already cover this, I > guess it doesn't yet impact all of those existing MySQLOpportunisticTest > classes it has. Yeah, I guess it's the first test case that actually tries

Re: [openstack-dev] [oslo] oslo.db reset session?

2016-02-23 Thread Roman Podoliaka
t; On 02/23/2016 12:06 PM, Roman Podoliaka wrote: >> >> Mike, >> >> I think that won't work as Nova creates its own instance of >> _TransactionContextManager: >> >> >> https://github.com/openstack/nova/blob/d8ddecf6e3ed1e8193e5f6dba910eb29

Re: [openstack-dev] [oslo] oslo.db reset session?

2016-02-23 Thread Roman Podoliaka
Mike, I think that won't work as Nova creates its own instance of _TransactionContextManager: https://github.com/openstack/nova/blob/d8ddecf6e3ed1e8193e5f6dba910eb29bbe6dac6/nova/db/sqlalchemy/api.py#L134-L135 Maybe we could change _TestTransactionFactory a bit, so that it takes a context manage

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-02-23 Thread Roman Podoliaka
Hi all, I've taken another look at this in order to propose patches to oslo.service/oslo.db, so that we have better defaults for WSGI greenlets number / max DB connections overflow [1] [2], which would be more suitable for DB oriented services like our APIs are. I used the Mike's snippet [3] for

Re: [openstack-dev] [ceilometer]ceilometer-collector high CPU usage

2016-02-17 Thread Roman Podoliaka
Hi all, Based on my investigation [1], I believe this is a combined effect of using eventlet and condition variables on Python 2.x. When heartbeats are enabled in oslo.messaging, you'll see polling with very small timeout values. This must not waste a lot of CPU time, still it is kind of annoying.

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-01-06 Thread Roman Podoliaka
Hi Mike, Thank you for this brilliant analysis! We've been seeing such timeout errors in downstream periodically and this is the first time someone has analysed the root cause thoroughly. On Fri, Dec 18, 2015 at 10:33 PM, Mike Bayer wrote: > Hi all - > > Let me start out with the assumptions I'm

Re: [openstack-dev] [oslo.db][sqlalchemy][mistral] Configuring default transaction isolation level

2015-12-08 Thread Roman Podoliaka
Hi Moshe, Feel free to submit a patch! This seems to be something we want to be able to configure. Thanks, Roman On Tue, Dec 8, 2015 at 9:41 AM, ELISHA, Moshe (Moshe) wrote: > Hi, > > > > We at Mistral want to move from the default transaction isolation level of > REPEATABLE READ to READ COMMIT

Re: [openstack-dev] [nova][ironic] do we really need websockify with numpy speedups?

2015-11-26 Thread Roman Podoliaka
Hi Pavlo, Can we just use a wheel package for numpy instead? Thanks, Roman On Thu, Nov 26, 2015 at 3:00 PM, Pavlo Shchelokovskyy wrote: > Hi again, > > I've went on and created a proper pull request to websockify [0], comment > there if you think we need it :) > > I also realized that there is

Re: [openstack-dev] [oslo.db][sqlalchemy] rollback after commit

2015-09-16 Thread Roman Podoliaka
Hi Gareth, Right, 'SELECT 1' issued at the beginning of every transaction is a pessimistic check to detect disconnects early. oslo.db will create a new DB connection (as well as invalidate all the existing connections to the same DB in the pool) and retry the transaction once [1] ROLLBACK you are

Re: [openstack-dev] [Infra][CI] gate-{name}-requirements job fails on stable/juno

2015-07-24 Thread Roman Podoliaka
Oh, I missed that one! Thank you, Ihar! On Fri, Jul 24, 2015 at 2:45 PM, Ihar Hrachyshka wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On 07/24/2015 12:37 PM, Roman Podoliaka wrote: >> Hi all, >> >> In oslo.db we recently hit a requirements che

[openstack-dev] [Infra][CI] gate-{name}-requirements job fails on stable/juno

2015-07-24 Thread Roman Podoliaka
Hi all, In oslo.db we recently hit a requirements checking job failure [0]. It's caused by the fact the job tries to import a module from openstack_requirements package, which is missing in stable/juno branch of requirements project [1]. The job must have been broken for stable/juno since [2] was

Re: [openstack-dev] [openstack][nova] Streamlining of config options in nova

2015-07-23 Thread Roman Podoliaka
Hi all, FWIW, this is exactly what we have in oslo libs, e.g. in oslo.db [0] Putting all Nova options into one big file is probably not a good idea, still we could consider storing those per-package (per backend, per driver, etc), rather than per Python module to reduce the number of possible cir

Re: [openstack-dev] [nova] Rebuilding instances booted from volume

2015-07-02 Thread Roman Podoliaka
Hi ZhengZhenyu, I'd say, it's more like a new feature and rebuild of volume-backed instances is simply not implemented yet. Though, I agree, that existing behaviour is rather confusing, as such rebuild will be a no-op, and Nova won't report any errors either. AFAIK, Eugeniya K. (CC'ed) started t

Re: [openstack-dev] [oslo] needed, driver for oslo.db thursday session

2015-05-21 Thread Roman Podoliaka
Will try to make it, but will probably be in 'read-only' mode, sorry :( On Wed, May 20, 2015 at 5:59 PM, Mike Bayer wrote: > > > On 5/20/15 9:31 AM, Davanum Srinivas wrote: >> >> Thanks Jeremy, >> >> Mike, Roman, Victor, Please see remote connection details in: >> https://etherpad.openstack.org/p

Re: [openstack-dev] [oslo] needed, driver for oslo.db thursday session

2015-05-19 Thread Roman Podoliaka
Hi all, Just FYI, due to problems with obtaining a Canadian visa, neither Victor Sergeyev, nor I made it to Vancouver. I hope someone else from Oslo team can replace Mike as a session driver. Thanks, Roman On Tue, May 19, 2015 at 3:53 AM, Mike Bayer wrote: > Hello - > > It is my extreme disple

Re: [openstack-dev] [Fuel] Testing DB migrations

2015-03-06 Thread Roman Podoliaka
Hi all, You could take a look at how this is done in OpenStack projects [1][2] Most important parts: 1) use the same RDBMS you use in production 2) test migration scripts on data, not on empty schema 3) test corner cases (adding a NOT NULL column without a server side default value, etc) 4) do a

Re: [openstack-dev] [api][nova] Openstack HTTP error codes

2015-01-29 Thread Roman Podoliaka
Hi Anne, I think Eugeniya refers to a problem, that we can't really distinguish between two different badRequest (400) errors (e.g. wrong security group name vs wrong key pair name when starting an instance), unless we parse the error description, which might be error prone. Thanks, Roman On Th

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-29 Thread Roman Podoliaka
l still be able to use it as a general WSGI app. Thanks, Roman On Thu, Jan 29, 2015 at 6:55 PM, Mike Bayer wrote: > > > Roman Podoliaka wrote: > >> >> On native threads vs green threads: I very much like the Keystone >> approach, which allows to run the service us

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-29 Thread Roman Podoliaka
r tempest/rally workloads, rather than just running synthetic tests. I'm going to give it a try on my devstack installation. Thanks, Roman On Thu, Jan 29, 2015 at 6:42 PM, Jeremy Stanley wrote: > On 2015-01-29 18:35:20 +0200 (+0200), Roman Podoliaka wrote: > [...] >> Otherwise, PyMySQL

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-29 Thread Roman Podoliaka
Hi all, Mike, thanks for summarizing this in https://wiki.openstack.org/wiki/PyMySQL_evaluation ! On PyMySQL: this is something we need to enable testing of oslo.db on Python 3.x and PyPy. Though, I doubt we want to make PyMySQL the default DB API driver for OpenStack services for Python 2.x. At

Re: [openstack-dev] [nova] Integration with Ceph

2014-12-01 Thread Roman Podoliaka
Hi Sergey, AFAIU, the problem is that when Nova was designed initially, it had no notion of shared storage (e.g. Ceph), so all the resources were considered to be local to compute nodes. In that case each total value was a sum of values per node. But as we see now, that doesn't work well with Ceph

Re: [openstack-dev] oslo.db 1.1.0 released

2014-11-18 Thread Roman Podoliaka
rovided iterator mix-in >> 436dfdc ModelsMigrationsSync:add correct server_default check for Enum >> 2075074 Add history/changelog to docs >> c9e5fdf Add run_cross_tests.sh script >> >> Thanks Andreas Jaeger, Ann Kamyshnikova, Christian Berend

Re: [openstack-dev] [oslo.db] Marker based paging

2014-11-03 Thread Roman Podoliaka
Hi Mike, I think that code was taken from Nova (or maybe some other project) as is and we haven't touched it since then. Please speak up - we want to know about all possible problems with current approach. Thanks, Roman On Fri, Oct 31, 2014 at 2:58 PM, Heald, Mike wrote: > Hi all, > > I'm impl

Re: [openstack-dev] [novaclient] E12* rules

2014-10-20 Thread Roman Podoliaka
Hi Andrey, Generally I'm opposed to such changes enabling random PEP8 checks, but in this particular case I kind of like the fact you fix the mess with indents in the code. python-novaclient code base is fairly small, CI nodes are not overloaded at this point of the release cycle, code looks bett

Re: [openstack-dev] [all] [oslo] Proposed database connectivity patterns

2014-10-09 Thread Roman Podoliaka
Hi Mike, Great stuff! Fixing the mess with transactions and their scope is probably one of the most important tasks for us, IMO. I look forward for this to be implemented in oslo.db and consuming projects! Thanks, Roman On Thu, Oct 9, 2014 at 12:07 AM, Mike Bayer wrote: > Hi all - > > I’ve draf

[openstack-dev] [oslo] Meeting time

2014-10-07 Thread Roman Podoliaka
Hi all, Last Friday we decided to find a better time for the weekly team meeting. Keeping in mind that DST ends soon (October the 26th in Europe, November the 2nd in the US), I think, we can choose from: - Mondays at 1600 UTC [1]: #openstack-meeting-alt, #openstack-meeting-3 - Thursdays at 1600 U

Re: [openstack-dev] [nova] Create an instance with a custom uuid

2014-09-24 Thread Roman Podoliaka
Hi Joe, Tools like Pumphouse [1] (migrates workloads, e.g. instances, between two OpenStack clouds) would benefit from supporting this (Pumphouse would be able to replicate user instances in a new cloud up to their UUIDs). Are there any known gotchas with support of this feature in REST APIs (in

Re: [openstack-dev] [nova] 2 weeks in the bug tracker

2014-09-19 Thread Roman Podoliaka
Hi all, FWIW, a quick and dirty solution is here: http://xsnippet.org/360188/ :) Thanks, Roman On Fri, Sep 19, 2014 at 2:03 PM, Ben Nemec wrote: > On 09/19/2014 08:13 AM, Sean Dague wrote: >> I've spent the better part of the last 2 weeks in the Nova bug tracker >> to try to turn it into someth

[openstack-dev] [oslo][db] Nominating Mike Bayer for the oslo.db core reviewers team

2014-08-15 Thread Roman Podoliaka
Hi Oslo team, I propose that we add Mike Bayer (zzzeek) to the oslo.db core reviewers team. Mike is an author of SQLAlchemy, Alembic, Mako Templates and some other stuff we use in OpenStack. Mike has been working on OpenStack for a few months contributing a lot of good patches and code reviews to

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-08 Thread Roman Podoliaka
Hi Li, How are you going to make this separation transparent? I mean, generally, in a function code, you can't know in advance if the transaction will be read-only or it will contain an INSERT/UPDATE/DELETE statement. On the other hand, as a developer, you could analyze the DB queries that can be

Re: [openstack-dev] [Infra] MyISAM as a default storage engine for MySQL in the gate

2014-07-21 Thread Roman Podoliaka
Aha, makes sense. Yeah, this means we miss such a check at least in Neutron and should add one to the test suite. Thanks! On Mon, Jul 21, 2014 at 6:34 PM, Clark Boylan wrote: > > On Jul 21, 2014 8:28 AM, "Roman Podoliaka" wrote: >> >> Hi all, >> >> T

[openstack-dev] [Infra] MyISAM as a default storage engine for MySQL in the gate

2014-07-21 Thread Roman Podoliaka
Hi all, To my surprise I found that we default to using MyISAM in the gate [1], while InnoDB would be a much more suitable choice, which people use in production deployments (== we should test it in the gate). This means, that every table, for which we haven't explicitly specified to use InnoDB, w

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-07-09 Thread Roman Podoliaka
changes to oslo.db are needed? (as an oslo.db developer I'm very interested in this part :) ) Thanks, Roman On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 09/07/14 15:40, Sean Dague wrote: >> On 07

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-07-09 Thread Roman Podoliaka
Hi Ihar, AFAIU, the switch is a matter of pip install + specifying the correct db URI in the config files. I'm not sure why you are filing a spec in Neutron project. IMHO, this has nothing to do with projects, but rather a purely deployment question. E.g. don't we have PostgreSQL+psycopg2 or MySQL

Re: [openstack-dev] Moving neutron to oslo.db

2014-07-04 Thread Roman Podoliaka
work. >> >> Thanks for doings this work. >> >> Salvatore >> >> >> On 3 July 2014 14:15, Roman Podoliaka wrote: >> >>> Hi Salvatore, >>> >>> I must be missing something. Hasn't it been done in >>> ht

Re: [openstack-dev] Moving neutron to oslo.db

2014-07-03 Thread Roman Podoliaka
Hi Salvatore, I must be missing something. Hasn't it been done in https://review.openstack.org/#/c/103519/? :) Thanks, Roman On Thu, Jul 3, 2014 at 2:51 PM, Salvatore Orlando wrote: > Hi, > > As you surely now, in Juno oslo.db will graduate [1] > I am currently working on the port. It's been al

Re: [openstack-dev] [Fuel] Bug squashing day on June, 17

2014-06-18 Thread Roman Podoliaka
> separately while maintaing visibility of both on the same page. The only >>> downside is spamming the bug with comments related to different projects, >>> but I think it's a reasonable trade off, you can't have too much information >>> about a bug :) >>

Re: [openstack-dev] [Fuel] Bug squashing day on June, 17

2014-06-18 Thread Roman Podoliaka
Hi Fuelers, Not directly related to bug squashing day, but something to keep in mind. AFAIU, both MOS and Fuel bugs are currently tracked under https://bugs.launchpad.net/fuel/ Launchpad project page. Most bugs filed there are probably deployment-specific, but still I bet there is a lot of bugs i

Re: [openstack-dev] [Oslo] [Ironic] DB migration woes

2014-06-08 Thread Roman Podoliaka
Hi Mike, >>> However, when testing an application that uses a fixed set of tables, as >>> should be the case for the majority if not all Openstack apps, there’s no >>> reason that these tables need to be recreated for every test. This is a very good point. I tried to use the recipe from SQLAlch

Re: [openstack-dev] [Oslo] [Ironic] DB migration woes

2014-06-08 Thread Roman Podoliaka
Hi Deva, I haven't actually touched Ironic db migrations tests code yet, but your feedback is very valuable for oslo.db maintainers, thank you! So currently, there are two ways to run migrations tests: 1. Opportunistically (using openstack_citest user credentials; this is how we test migrations i

Re: [openstack-dev] [oslo][db] oslo.db repository review request

2014-05-30 Thread Roman Podoliaka
on't break people projects every time we make some API change). Thanks, Roman On Fri, May 30, 2014 at 6:06 PM, Sergey Lukjanov wrote: > Hey Roman, > > will it be the alpha version that should not be used by other projects > or it'll be ready to use? > > Thanks.

Re: [openstack-dev] [oslo][db] oslo.db repository review request

2014-05-30 Thread Roman Podoliaka
Hi Matt, We're waiting for a few important fixes to be merged (usage of oslo.config, eventlet tpool support). Once those are merged, we'll cut the initial release. Thanks, Roman On Fri, May 30, 2014 at 5:19 PM, Matt Riedemann wrote: > > > On 4/25/2014 7:46 AM, Doug Hellmann wrote: >> >> On Fri,

Re: [openstack-dev] [nova] plan for moving to using oslo.db

2014-05-12 Thread Roman Podoliaka
Hi all, Yes, once the oslo.db initial release is cut, we expect the migration from using of its oslo-incubator version to a library one to be as simple as following the steps you've mentioned. Though, we still need to finish the setup of oslo.db repo (AFAIK, this is currently blocked by the fact w

[openstack-dev] [TripleO][Summit] Neutron etherpad

2014-05-01 Thread Roman Podoliaka
Hi all, Following the mailing list thread started by Marios I've put some initial questions to discuss into this etherpad document: https://etherpad.openstack.org/p/juno-summit-tripleo-neutron You are encouraged to take a look at it and add your thoughts and/or questions :) Thanks, Roman _

Re: [openstack-dev] Gerrit downtime and upgrade on 2014-04-28

2014-04-25 Thread Roman Podoliaka
Hi all, >>> Wouldn't it be better to make this label more persistent? +1. It's really annoying to press "Work in Progress" button every time you upload a new patch set. Thanks, Roman On Fri, Apr 25, 2014 at 11:02 AM, Yuriy Taraday wrote: > Hello. > > On Wed, Apr 23, 2014 at 2:40 AM, James E. B

Re: [openstack-dev] oslo removal of use_tpool conf option

2014-04-18 Thread Roman Podoliaka
Hi all, >>> I objected to this and asked (more demanded) for this to be added back into >>> oslo. It was not. What I did not realize when I was reviewing this nova >>> patch, was that nova had already synced oslo’s change. And now we’ve >>> released Icehouse with a conf option missing that exis

Re: [openstack-dev] [fuel] oddness with sqlalchemy db().refresh(object)

2014-04-13 Thread Roman Podoliaka
Hi Andrew, I believe, it's just the way SQLAlchemy Session works: all the changes you've made within a session aren't propagated to the db (read: no actual queries are issued) until you explicitly do: - flush(), or - commit() (as commit() calls flush() first), or - Query - one(), first(), all(),

Re: [openstack-dev] [oslo] Split Oslo Incubator?

2014-04-08 Thread Roman Podoliaka
Hi Victor, >>> The openstack.common module also known as "Oslo Incubator" or "OpenStack >>> Common Libraries" has 44 dependencies. IMO we reach a point where it became >>> too huge. Would it be possible to split it into smaller parts and >>> distribute it on PyPI with a stable API? I don't know

Re: [openstack-dev] [Tripleo][Neutron] Tripleo & Neutron

2014-04-07 Thread Roman Podoliaka
Hi all, Perhaps, we should file a design session for Neutron-specific questions? >>> 1. Define a neutron node (tripleo-image-elements/disk-image-builder) and >>> make sure it deploys and scales ok (tripleo-heat-templates/tuskar). This >>> comes under by lifeless blueprint at >>> https://bluepr

Re: [openstack-dev] Duplicate code for processing REST APIs

2014-03-14 Thread Roman Podoliaka
s. Hopefully, we'll no longer need to sync code from incubator soon. Thanks, Roman On Fri, Mar 14, 2014 at 9:48 AM, Duncan Thomas wrote: > On 13 March 2014 21:13, Roman Podoliaka wrote: >> Hi Steven, >> >> Code from openstack/common/ dir is 'synced' from oslo

Re: [openstack-dev] OpenStack vs. SQLA 0.9

2014-03-13 Thread Roman Podoliaka
Hi all, I think it's actually not that hard to fix the errors we have when using SQLAlchemy 0.9.x releases. I uploaded two changes two Nova to fix unit tests: - https://review.openstack.org/#/c/80431/ (this one should also fix the Tempest test run error) - https://review.openstack.org/#/c/80432/

Re: [openstack-dev] Duplicate code for processing REST APIs

2014-03-13 Thread Roman Podoliaka
Hi Steven, Code from openstack/common/ dir is 'synced' from oslo-incubator. The 'sync' is effectively a copy of oslo-incubator subtree into a project source tree. As syncs are not done at the same time, the code of synced modules may indeed by different for each project depending on which commit o

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

2014-03-10 Thread Roman Podoliaka
Hi all, >>> I've never understood why we treat the DB as a LOG (keeping deleted == 0 >>> records around) when we should just use a LOG (or similar system) to begin >>> with instead. I can't agree more with you! Storing deleted records in tables is hardly usable, bad for performance (as it makes

Re: [openstack-dev] UTF-8 required charset/encoding for openstack database?

2014-03-10 Thread Roman Podoliaka
Hi all, >>> It sounds like for the near future my best bet would be to just set the >>> install scripts to configure postgres (which is used only for openstack) to >>> default to utf-8. Is that a fair summation? Yes. UTF-8 is a reasonable default value. Thanks, Roman On Mon, Mar 10, 2014 at

Re: [openstack-dev] UTF-8 required charset/encoding for openstack database?

2014-03-10 Thread Roman Podoliaka
Hi Chris, AFAIK, most OpenStack projects enforce tables to be created with the encoding set to UTF-8 because MySQL has horrible defaults and would use latin1 otherwise. PostgreSQL must default to the locale of a system on which it's running. And, I think, most systems default to UTF-8 nowadays. A

Re: [openstack-dev] [oslo-incubator] removal of slave_connection from db.sqlalchemy.session

2014-03-05 Thread Roman Podoliaka
Hi all, So yeah, we could restore the option and put creation of a slave engine instance to EngineFacade class, but I don't think we want this. The only reason why slave connections aren't implemented e.g. in SQLAlchemy is that, SQLAlchemy, as a library can't decide for you how those engines sho

Re: [openstack-dev] WARNING: ... This application has not enabled MySQL traditional mode, which means silent data corruption may occur - real issue?

2014-03-03 Thread Roman Podoliaka
Hi all, This is just one another example of MySQL not having production ready defaults. The original idea was to force setting the SQL mode to TRADITIONAL in code in projects using oslo.db code when "they are ready" (unit and functional tests pass). So the warning was actually for developers rathe

Re: [openstack-dev] [Infra] openstack_citest MySQL user privileges to create databases on CI nodes

2014-02-28 Thread Roman Podoliaka
stack_citest@localhost/openstack_citest]/privileges: > privileges changed '' to 'all' [0m > > On Fri, Feb 28, 2014 at 12:28 PM, Roman Podoliaka > wrote: >> Hi Clark, all, >> >> https://review.openstack.org/#/c/76634/ has been merged, but I still &g

Re: [openstack-dev] [Neutron][TripleO] Neutron DB migrations best practice

2014-02-28 Thread Roman Podoliaka
Hi Robert, all, >>> But what are we meant to do? Nova etc are dead easy: nova-manage db sync >>> every time the code changes, done. I believe, it's not different from Nova: run db sync every time the code changes. It's the only way to guarantee the most recent DB schema version is used. Interest

Re: [openstack-dev] [Infra] openstack_citest MySQL user privileges to create databases on CI nodes

2014-02-28 Thread Roman Podoliaka
/gate-oslo-incubator-python27/e115a5f/console.html On Wed, Feb 26, 2014 at 11:54 AM, Roman Podoliaka wrote: > Hi Clark, > >>>> I think we can safely GRANT ALL on *.* to openstack_citest@localhost and >>>> call that good enough > Works for me. > > Thanks, &

Re: [openstack-dev] [Infra] openstack_citest MySQL user privileges to create databases on CI nodes

2014-02-26 Thread Roman Podoliaka
Hi Clark, >>> I think we can safely GRANT ALL on *.* to openstack_citest@localhost and >>> call that good enough Works for me. Thanks, Roman On Tue, Feb 25, 2014 at 8:29 PM, Clark Boylan wrote: > On Tue, Feb 25, 2014 at 2:33 AM, Roman Podoliaka > wrote: >> Hi a

[openstack-dev] [Infra] openstack_citest MySQL user privileges to create databases on CI nodes

2014-02-25 Thread Roman Podoliaka
Hi all, [1] made it possible for openstack_citest MySQL user to create new databases in tests on demand (which is very useful for parallel running of tests on MySQL and PostgreSQL, thank you, guys!). Unfortunately, openstack_citest user can only create tables in the created databases, but not to

Re: [openstack-dev] [nova][oslo] Changes to oslo-incubator sync workflow

2014-02-20 Thread Roman Podoliaka
Hi all, I'm ready to help with syncing of DB code. But we'll need reviewers attention in both oslo-incubator in nova :) Thanks, Roman On Thu, Feb 20, 2014 at 5:37 AM, Lance D Bragstad wrote: > Shed a little bit of light on Matt's comment about Keystone removing > oslo-incubator code and the iss

Re: [openstack-dev] [savanna] Alembic migrations and absence of DROP column in sqlite

2014-02-01 Thread Roman Podoliaka
Hi all, My two cents. > 2) Extend alembic so that op.drop_column() does the right thing We could, but should we? The only reason alembic doesn't support these operations for SQLite yet is that SQLite lacks proper support of ALTER statement. For sqlalchemy-migrate we've been providing a work-arou

Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-17 Thread Roman Podoliaka
Hi all, Huge +1 for periodic syncs for two reasons: 1) it makes syncs smaller and thus easier 2) code in oslo-incubator often contains important bug fixes (e.g. incorrect usage of eventlet TLS we found in Nova a few months ago) Thanks, Roman On Fri, Jan 17, 2014 at 10:15 AM, Flavio Percoco wrot

Re: [openstack-dev] [Ironic] Let's move to Alembic

2014-01-16 Thread Roman Podoliaka
Hi all, I'm glad you've decided to drop sqlalchemy-migrate support :) As for porting Ironic to using of alembic migrations, I believe, Dmitriy Shulyak already uploaded a proof-of-concept patch to Ironic before, but it abandoned. Adding Dmitriy to this thread, so he is notified he can restore his

Re: [openstack-dev] [infra] Unit tests, gating, and real databases

2014-01-08 Thread Roman Podoliaka
Hi Ivan, Indeed, nodepool nodes have MySQL and PostgreSQL installed and running. There are databases you can access from your tests (mysql://openstack_citest:openstack_citest@localhost/openstack_citest and postgresql://openstack_citest:openstack_citest@localhost/openstack_citest). [1] is a great e

Re: [openstack-dev] [Neutron] Alembic migrations

2013-12-22 Thread Roman Podoliaka
Hi Gary, It's a known bug (the migration script creating 'agents' table is mistakenly not applied when running schema migrations with ML2 core plugin selected). There is a patch on review https://review.openstack.org/#/c/61677 fixing this error. Thanks, Roman On Sun, Dec 22, 2013 at 4:02 PM, Gar

Re: [openstack-dev] [TripleO] Gerrit review refs now supported by diskimage-builder's source-repositories element

2013-12-20 Thread Roman Podoliaka
Hi Chris, This is super useful for testing patches on review! Thank you! Roman On Fri, Dec 20, 2013 at 7:35 PM, Chris Jones wrote: > Hi > > As of just now (review 63021) the source-repositories element in > diskimage-builder can fetch git repos from gerrit reviews. > > I figured it'd be worth m

Re: [openstack-dev] [Neutron] Fix to agents race condition creates another issue

2013-12-12 Thread Roman Podoliaka
Hi, So actually these two are different issues: 1) https://review.openstack.org/#/c/61677 fixes the issue with a few mandatory migrations not applying when ML2 plugin is used (the only reason we hadn't seen it before is that those migrations were applied on Jenkins CI nodes because LBaaS service p

Re: [openstack-dev] [Neutron] Fix to agents race condition creates another issue

2013-12-11 Thread Roman Podoliaka
Hi Edgar, Nice catch! I proposed a fix here https://review.openstack.org/#/c/61663/ Could you please test it and and tell us if it fixes your problem? Thanks, Roman On Thu, Dec 12, 2013 at 5:45 AM, Edgar Magana wrote: > In commit: > https://review.openstack.org/#/c/58814/ > > There is an assum

Re: [openstack-dev] [Olso][DB] Remove eventlet from oslo.db

2013-12-03 Thread Roman Podoliaka
dback! Roman On Wed, Dec 4, 2013 at 1:45 AM, Mark McLoughlin wrote: > On Mon, 2013-12-02 at 16:02 +0200, Victor Sergeyev wrote: >> Hi folks! >> >> At the moment I and Roman Podoliaka are working on splitting of >> openstack.common.db code into a separate library. And it wou

Re: [openstack-dev] Top Gate Bugs

2013-11-20 Thread Roman Podoliaka
Hey all, I think I found a serious bug in our usage of eventlet thread local storage. Please check out this snippet [1]. This is how we use eventlet TLS in Nova and common Oslo code [2]. This could explain how [3] actually breaks TripleO devtest story and our gates. Am I right? Or I am missing s

Re: [openstack-dev] sqlalchemy-migrate needs a new release

2013-11-15 Thread Roman Podoliaka
Hey, Awesome! Thank you, guys! I'm going to give it a try early next week (we basically need to run/fix tests for every project that uses sqlalchemy-migrate before we can bump the version in global-requirements, but at least we can start doing this now). Roman On Fri, Nov 15, 2013 at 5:41 PM, D

Re: [openstack-dev] Alembic or SA Migrate (again)

2013-11-13 Thread Roman Podoliaka
Hey, I don't quite know the current state of affairs in Ceilometer, but we had a design session at the summit at which we discussed how we were going to proceed with Alembic not supporting migrations on SQLite. Basically, we don't care about supporting of SQLite for production, but we still need t

  1   2   >