Well, I can run the upgrade command, but it doesn't seem to be processing my new migration file. I have tried using upgrade with 'head', and with the HEAD file set to the previous version and to the new version. In both cases, I get these info messages twice: "Context impl MySWLImpl." and "Will assume non-transactional DDL." I put a "import pdb; pdb.set_trace() in my migration file, but it never reaches that.
What am I possibly missing? Regards, PCM On Tue, Jul 7, 2015 at 4:04 PM Paul Michali <p...@michali.net> wrote: > I found the issue. The upgrade is looking for config to have [database] > section and connection definition. If I use /etc/neutron/neutron.conf, then > the neutron-db-manage runs. > > > > On Tue, Jul 7, 2015 at 3:38 PM Paul Michali <p...@michali.net> wrote: > >> I have that change in the neutron repo that is being used with by this >> neutron-vpnaas repo. >> >> On Tue, Jul 7, 2015 at 3:12 PM Mike Bayer <mba...@redhat.com> wrote: >> >>> >>> >>> On 7/7/15 1:28 PM, Paul Michali wrote: >>> >>> HEAD, head, 24f28869838b (my new file) all say the same thing. :( >>> >>> >>> On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <sorla...@nicira.com> >>> wrote: >>> >>>> possibly I was wrong in mixing up git & alembic. >>>> It should be "upgrade head" - lowercase. >>>> >>>> If that doesn't work there might some other issue lurking. >>>> >>>> Salvatore >>>> >>>> On 7 July 2015 at 17:44, Paul Michali <p...@michali.net> wrote: >>>> >>>>> Salvatore, >>>>> >>>>> I changed head to the version before my new one, and then tried to >>>>> upgrade and I see this: >>>>> neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf >>>>> --service vpnaas upgrade HEAD >>>>> Traceback (most recent call last): >>>>> File "/usr/local/bin/neutron-db-manage", line 10, in <module> >>>>> sys.exit(main()) >>>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in >>>>> main >>>>> CONF.command.func(config, CONF.command.name) >>>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in >>>>> do_upgrade >>>>> run_sanity_checks(config, revision) >>>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in >>>>> run_sanity_checks >>>>> script_dir.run_env() >>>>> File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", >>>>> line 390, in run_env >>>>> util.load_python_file(self.dir, 'env.py') >>>>> File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line >>>>> 243, in load_python_file >>>>> module = load_module_py(module_id, path) >>>>> File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", >>>>> line 79, in load_module_py >>>>> mod = imp.load_source(module_id, path, fp) >>>>> File >>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >>>>> line 86, in <module> >>>>> run_migrations_online() >>>>> File >>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >>>>> line 67, in run_migrations_online >>>>> engine = session.create_engine(neutron_config.database.connection) >>>>> File >>>>> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", >>>>> line 112, in create_engine >>>>> url = sqlalchemy.engine.url.make_url(sql_connection) >>>>> File >>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line >>>>> 186, in make_url >>>>> return _parse_rfc1738_args(name_or_url) >>>>> File >>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line >>>>> 235, in _parse_rfc1738_args >>>>> "Could not parse rfc1738 URL from string '%s'" % name) >>>>> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string >>>>> '' >>>>> >>>>> Any ideas what is wrong here? >>>>> >>>> >>> I'm going to guess this is the issue fixed by >>> https://review.openstack.org/#/c/194197/ >>> >>> >>> >>> >>> >>> >>>>> On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <p...@michali.net> wrote: >>>>> >>>>>> >>>>>> Yes, I wasn't using the --service option, so I suspect that is why >>>>>> my down_version was wrong. In talking with Akihiro, I added a check to >>>>>> PEP8 and made sure that it fails if head is wrong. It is: >>>>>> https://review.openstack.org/#/c/199082/ (of course that failed py27 >>>>>> - I've got to see if there was some recent breakage in vpn repo, again). >>>>>> >>>>>> Regarding the migration, one of the new columns may be None, but >>>>>> there must be at least one IP version entry (there is an existing test in >>>>>> VPN for using a router w/o an external IP set). Since the new code will >>>>>> rely on these new fields, I'd like to populate them as part of the >>>>>> migration. I think it would be more complicated to handle during >>>>>> operation. >>>>>> >>>>>> Does anyone have examples of how to do queries of objects, from the >>>>>> migration upgrade() code? >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> PCM >>>>>> >>>>>> On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amot...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> 2015-07-07 21:39 GMT+09:00 Henry Gessau <ges...@cisco.com>: >>>>>>> >>>>>>>> On Tue, Jul 07, 2015, Paul Michali <p...@michali.net> >>>>>>>> <p...@michali.net> <p...@michali.net> wrote: >>>>>>>> >>>>>>>> Thanks Salvatore for the responses. See @PCM in-line... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando < >>>>>>>> sorla...@nicira.com> wrote: >>>>>>>> >>>>>>>>> Some comments inline. >>>>>>>>> >>>>>>>>> Salvatore >>>>>>>>> >>>>>>>>> On 6 July 2015 at 20:00, Paul Michali <p...@michali.net> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have some urgent requests about migration that I'm hoping to >>>>>>>>>> get some info on. I'm working on a bug where I need to add two >>>>>>>>>> (related) >>>>>>>>>> fields to a table for VPNaaS. Here's the objectives related to >>>>>>>>>> migration... >>>>>>>>>> >>>>>>>>>> 1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice >>>>>>>>>> table >>>>>>>>>> 2) for each entry in the vpnservice table: >>>>>>>>>> 2.1) Get the router.gw_port.fixed_ips list >>>>>>>>>> 2.2) Determine the version of each fixed IP and store the >>>>>>>>>> first of each version (if any) into the appropriate new field. >>>>>>>>>> >>>>>>>>>> I have created a migration file, and I changed the >>>>>>>>>> down_revision to be the number of the revision that is the first in >>>>>>>>>> the >>>>>>>>>> migration chain in the VPN repo. >>>>>>>>>> >>>>>>>>>> Here are the many questions I have... >>>>>>>>>> >>>>>>>>>> When I look in the VPN repo, the HEAD file has the version >>>>>>>>>> 'kilo', which is not the current head. >>>>>>>>>> >>>>>>>>> >>>>>>>>>> Shouldn't it the version number of the first file in the >>>>>>>>>> migration chain? >>>>>>>>>> >>>>>>>>> >>>>>>>>> It should indeed. How are you generating the revision >>>>>>>>> script? Using neutron-db-manage it should be updated automatically [1] >>>>>>>>> >>>>>>>> >>>>>>>> @PCM I ran neutron-db-manage, when in the neutron repo, and it >>>>>>>> assigned some version, but it was not the latest in the neutron-vpnaas >>>>>>>> repo. >>>>>>>> >>>>>>>> neutron-db-manage does not handle alembic branches in separate >>>>>>>> repos very well at all yet. I am working on updating it with >>>>>>>> https://review.openstack.org/198524 but I have quite a lot left to >>>>>>>> do. >>>>>>>> >>>>>>> >>>>>>> Yes, at now we have implicit order of running alembic >>>>>>> migrations. >>>>>>> First run neutron db migration and then advanced service migrations. >>>>>>> >>>>>>> I do not fully understand how alembic branch mechanism works, but >>>>>>> I think we can have a common ancestor and have multiple branches, >>>>>>> and each branch can evolve independently. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> I checked the VPN repo and there were a chain of versions, which >>>>>>>> I used to determine what the head should be and have set the version >>>>>>>> accordingly. However, in the current repo, head is set to "kilo", which >>>>>>>> appears to be incorrect. The versions are: >>>>>>>> >>>>>>>> 56893333aa52 >>>>>>>> kilo <<< HEAD >>>>>>>> 3ea02b2a773e >>>>>>>> start_neutron_vpnaas >>>>>>>> None >>>>>>>> >>>>>>>> Ouch. That is an error, because >>>>>>>> https://review.openstack.org/190569 should have updated HEAD but >>>>>>>> didn't. >>>>>>>> >>>>>>>> The version sequence (you can see it in any devstack run) is: >>>>>>>> >>>>>>>> INFO [alembic.migration] Running upgrade -> start_neutron_vpnaas, >>>>>>>> start neutron-vpnaas chain >>>>>>>> INFO [alembic.migration] Running upgrade start_neutron_vpnaas -> >>>>>>>> 3ea02b2a773e, add_index_tenant_id >>>>>>>> INFO [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo >>>>>>>> INFO [alembic.migration] Running upgrade kilo -> 56893333aa52, fix >>>>>>>> identifier map fk >>>>>>>> >>>>>>> >>>>>>> It seems we don't have an appropriate check for HEAD revision >>>>>>> in at least VPNaaS repo. >>>>>>> Paul and I just discussed it. We need to improve the check too. >>>>>>> >>>>>>> >>>>>>>> Should I do a separate commit that fixes the HEAD file, or just >>>>>>>> fix it as part of the bug fix I'm working on. >>>>>>>> >>>>>>>> Yes, you should immediately submit a patch to change HEAD to >>>>>>>> 56893333aa52. >>>>>>>> >>>>>>>> >>>>>>>> BTW, at one point, after having correctly set the HEAD and >>>>>>>> versions in my new migration file, I think I ran neutron-db-manage >>>>>>>> check_migration, and I think it set the HEAD to my version, but it did >>>>>>>> that >>>>>>>> in the neutron repo, and not the VPN repo. I might have been running >>>>>>>> from >>>>>>>> the wrong repo? >>>>>>>> >>>>>>>> I working on updating the devref docs for this process. Things >>>>>>>> have changed quite a bit with the alembic branches in separate repos. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> For my commit, I'm assuming I change the HEAD file to use my >>>>>>>>>> migration file's version? >>>>>>>>>> >>>>>>>>> >>>>>>>>> You can do that manually too, yes. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I set HEAD to my migration file, and my file has a down >>>>>>>>>> revision of the previous head's revision. If I run 'neutron-db-manage >>>>>>>>>> --config-file ../neutron/etc/neutron.conf --config-file >>>>>>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' >>>>>>>>>> there is >>>>>>>>>> no output so I guess that is OK. >>>>>>>>>> >>>>>>>>>> As I develop my new migration file, is there a way that I can >>>>>>>>>> test it (running neutron-db-migration, maybe)? >>>>>>>>>> >>>>>>>>> >>>>>>>>> When I test migrations I usually dump the database, run the >>>>>>>>> migration with neutron-db-manage upgrade HEAD (I think it's not >>>>>>>>> necessary >>>>>>>>> to specify HEAD), and restore the db from the dump if the migration >>>>>>>>> fails. >>>>>>>>> >>>>>>>>> >>>>>>>>>> Is there a way to run the migration file under the debugger, as >>>>>>>>>> well (importing pdb, for example)? >>>>>>>>>> >>>>>>>>> >>>>>>>>> The migration process is just like any python application, >>>>>>>>> so I guess you can debug it with pdb. >>>>>>>>> >>>>>>>> >>>>>>>> @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the >>>>>>>> piece that was missing. I take it there are no specific unit tests of >>>>>>>> the >>>>>>>> migration files? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> In the migration, I can add the columns needed. What's the best >>>>>>>>>> way to fill out those fields - using raw SQL queries or create a >>>>>>>>>> Session >>>>>>>>>> object and access the VpnService object's router object? >>>>>>>>>> >>>>>>>>> >>>>>>>>> If the default value for the column is not enough, and you >>>>>>>>> need to specify a value which depends on other values in the same row >>>>>>>>> I >>>>>>>>> would prefer plain SQL statements, but if that become cumbersome I >>>>>>>>> guess >>>>>>>>> it's ok to use sqlalchemy's session. >>>>>>>>> >>>>>>>>> >>>>>>>>>> I see there is some op.bind() call and then engine.execute(), >>>>>>>>>> but could use some help on the best way to extract the needed >>>>>>>>>> queries (I >>>>>>>>>> need to access the vpnservice's router, and then access the (Port) >>>>>>>>>> gw_port >>>>>>>>>> relationship, and from that access the (IPAllocation) fixed_ips >>>>>>>>>> list). >>>>>>>>>> >>>>>>>>> >>>>>>>>> Perhaps you can point us to the review pages on gerrit, and >>>>>>>>> we can provide detailed comments there. >>>>>>>>> >>>>>>>> >>>>>>>> @PCM Yeah, I haven't pushed it up yet. I have a few more changes >>>>>>>> to make, and should be able to get it up in a few days. The LP bug is >>>>>>>> 1464387. >>>>>>>> >>>>>>>> Essentially, in the vpnservices table, I'm adding an IPv4 and/or >>>>>>>> IPv6 addresses for the "local" end of VPN connections that will be >>>>>>>> established. This is to allow alternative VPN implementation >>>>>>>> (appliances, >>>>>>>> separate S/W, H/W, VM based VPN, etc) to specify addresses different >>>>>>>> than >>>>>>>> what is available on the Neutron router. >>>>>>>> >>>>>>>> However, for the reference implementation, we'll use the Neutron >>>>>>>> router's fixed_ips list (as is done today), and to handle the >>>>>>>> migration, >>>>>>>> I'm thinking the following is needed: >>>>>>>> >>>>>>>> 1) create the new columns. >>>>>>>> 2) Identify the router for that service and obtain it's GW >>>>>>>> fixed_ips list. >>>>>>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and >>>>>>>> store in new columns. >>>>>>>> >>>>>>>> So I need to form a query and code to do this. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Appreciate any advise here on how to debug the migration >>>>>>>>>> stuff... >>>>>>>>>> >>>>>>>>>> Paul Michali (pc_m) >>>>>>>>>> >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124 >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> __________________________________________________________________________ >>>>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>>>> Unsubscribe: >>>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>> >>>>>>>> >>>>>>> __________________________________________________________________________ >>>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>>> Unsubscribe: >>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>> >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
