HEAD, head, 24f28869838b (my new file) all say the same thing. :(
On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <sorla...@nicira.com> wrote: > possibly I was wrong in mixing up git & alembic. > It should be "upgrade head" - lowercase. > > If that doesn't work there might some other issue lurking. > > Salvatore > > On 7 July 2015 at 17:44, Paul Michali <p...@michali.net> wrote: > >> Salvatore, >> >> I changed head to the version before my new one, and then tried to >> upgrade and I see this: >> neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf >> --service vpnaas upgrade HEAD >> Traceback (most recent call last): >> File "/usr/local/bin/neutron-db-manage", line 10, in <module> >> sys.exit(main()) >> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in main >> CONF.command.func(config, CONF.command.name) >> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in >> do_upgrade >> run_sanity_checks(config, revision) >> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in >> run_sanity_checks >> script_dir.run_env() >> File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line >> 390, in run_env >> util.load_python_file(self.dir, 'env.py') >> File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line >> 243, in load_python_file >> module = load_module_py(module_id, path) >> File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line >> 79, in load_module_py >> mod = imp.load_source(module_id, path, fp) >> File >> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >> line 86, in <module> >> run_migrations_online() >> File >> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >> line 67, in run_migrations_online >> engine = session.create_engine(neutron_config.database.connection) >> File >> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", >> line 112, in create_engine >> url = sqlalchemy.engine.url.make_url(sql_connection) >> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", >> line 186, in make_url >> return _parse_rfc1738_args(name_or_url) >> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", >> line 235, in _parse_rfc1738_args >> "Could not parse rfc1738 URL from string '%s'" % name) >> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string '' >> >> Any ideas what is wrong here? >> >> On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <p...@michali.net> wrote: >> >>> >>> Yes, I wasn't using the --service option, so I suspect that is why my >>> down_version was wrong. In talking with Akihiro, I added a check to PEP8 >>> and made sure that it fails if head is wrong. It is: >>> https://review.openstack.org/#/c/199082/ (of course that failed py27 - >>> I've got to see if there was some recent breakage in vpn repo, again). >>> >>> Regarding the migration, one of the new columns may be None, but there >>> must be at least one IP version entry (there is an existing test in VPN for >>> using a router w/o an external IP set). Since the new code will rely on >>> these new fields, I'd like to populate them as part of the migration. I >>> think it would be more complicated to handle during operation. >>> >>> Does anyone have examples of how to do queries of objects, from the >>> migration upgrade() code? >>> >>> >>> Regards, >>> >>> PCM >>> >>> On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amot...@gmail.com> wrote: >>> >>>> 2015-07-07 21:39 GMT+09:00 Henry Gessau <ges...@cisco.com>: >>>> >>>>> On Tue, Jul 07, 2015, Paul Michali <p...@michali.net> <p...@michali.net> >>>>> wrote: >>>>> >>>>> Thanks Salvatore for the responses. See @PCM in-line... >>>>> >>>>> >>>>> >>>>> On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando <sorla...@nicira.com> >>>>> wrote: >>>>> >>>>>> Some comments inline. >>>>>> >>>>>> Salvatore >>>>>> >>>>>> On 6 July 2015 at 20:00, Paul Michali < <p...@michali.net> >>>>>> p...@michali.net> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have some urgent requests about migration that I'm hoping to get >>>>>>> some info on. I'm working on a bug where I need to add two (related) >>>>>>> fields >>>>>>> to a table for VPNaaS. Here's the objectives related to migration... >>>>>>> >>>>>>> 1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice >>>>>>> table >>>>>>> 2) for each entry in the vpnservice table: >>>>>>> 2.1) Get the router.gw_port.fixed_ips list >>>>>>> 2.2) Determine the version of each fixed IP and store the first >>>>>>> of each version (if any) into the appropriate new field. >>>>>>> >>>>>>> I have created a migration file, and I changed the down_revision >>>>>>> to be the number of the revision that is the first in the migration >>>>>>> chain >>>>>>> in the VPN repo. >>>>>>> >>>>>>> Here are the many questions I have... >>>>>>> >>>>>>> When I look in the VPN repo, the HEAD file has the version 'kilo', >>>>>>> which is not the current head. >>>>>>> >>>>>> >>>>>>> Shouldn't it the version number of the first file in the migration >>>>>>> chain? >>>>>>> >>>>>> >>>>>> It should indeed. How are you generating the revision script? >>>>>> Using neutron-db-manage it should be updated automatically [1] >>>>>> >>>>> >>>>> @PCM I ran neutron-db-manage, when in the neutron repo, and it >>>>> assigned some version, but it was not the latest in the neutron-vpnaas >>>>> repo. >>>>> >>>>> neutron-db-manage does not handle alembic branches in separate repos >>>>> very well at all yet. I am working on updating it with >>>>> <https://review.openstack.org/198524> >>>>> https://review.openstack.org/198524 but I have quite a lot left to do. >>>>> >>>> >>>> Yes, at now we have implicit order of running alembic migrations. >>>> First run neutron db migration and then advanced service migrations. >>>> >>>> I do not fully understand how alembic branch mechanism works, but >>>> I think we can have a common ancestor and have multiple branches, >>>> and each branch can evolve independently. >>>> >>>> >>>>> >>>>> I checked the VPN repo and there were a chain of versions, which I >>>>> used to determine what the head should be and have set the version >>>>> accordingly. However, in the current repo, head is set to "kilo", which >>>>> appears to be incorrect. The versions are: >>>>> >>>>> 56893333aa52 >>>>> kilo <<< HEAD >>>>> 3ea02b2a773e >>>>> start_neutron_vpnaas >>>>> None >>>>> >>>>> Ouch. That is an error, because <https://review.openstack.org/190569> >>>>> https://review.openstack.org/190569 should have updated HEAD but >>>>> didn't. >>>>> >>>>> The version sequence (you can see it in any devstack run) is: >>>>> >>>>> INFO [alembic.migration] Running upgrade -> start_neutron_vpnaas, >>>>> start neutron-vpnaas chain >>>>> INFO [alembic.migration] Running upgrade start_neutron_vpnaas -> >>>>> 3ea02b2a773e, add_index_tenant_id >>>>> INFO [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo >>>>> INFO [alembic.migration] Running upgrade kilo -> 56893333aa52, fix >>>>> identifier map fk >>>>> >>>> >>>> It seems we don't have an appropriate check for HEAD revision in at >>>> least VPNaaS repo. >>>> Paul and I just discussed it. We need to improve the check too. >>>> >>>> >>>>> Should I do a separate commit that fixes the HEAD file, or just fix it >>>>> as part of the bug fix I'm working on. >>>>> >>>>> Yes, you should immediately submit a patch to change HEAD to >>>>> 56893333aa52. >>>>> >>>>> >>>>> BTW, at one point, after having correctly set the HEAD and versions >>>>> in my new migration file, I think I ran neutron-db-manage check_migration, >>>>> and I think it set the HEAD to my version, but it did that in the neutron >>>>> repo, and not the VPN repo. I might have been running from the wrong >>>>> repo? >>>>> >>>>> I working on updating the devref docs for this process. Things have >>>>> changed quite a bit with the alembic branches in separate repos. >>>>> >>>>> >>>>> >>>>> >>>>>> For my commit, I'm assuming I change the HEAD file to use my >>>>>>> migration file's version? >>>>>>> >>>>>> >>>>>> You can do that manually too, yes. >>>>>> >>>>>> >>>>>>> >>>>>>> I set HEAD to my migration file, and my file has a down revision >>>>>>> of the previous head's revision. If I run 'neutron-db-manage >>>>>>> --config-file >>>>>>> ../neutron/etc/neutron.conf --config-file >>>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' there >>>>>>> is >>>>>>> no output so I guess that is OK. >>>>>>> >>>>>>> As I develop my new migration file, is there a way that I can test >>>>>>> it (running neutron-db-migration, maybe)? >>>>>>> >>>>>> >>>>>> When I test migrations I usually dump the database, run the >>>>>> migration with neutron-db-manage upgrade HEAD (I think it's not necessary >>>>>> to specify HEAD), and restore the db from the dump if the migration >>>>>> fails. >>>>>> >>>>>> >>>>>>> Is there a way to run the migration file under the debugger, as >>>>>>> well (importing pdb, for example)? >>>>>>> >>>>>> >>>>>> The migration process is just like any python application, so I >>>>>> guess you can debug it with pdb. >>>>>> >>>>> >>>>> @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the piece >>>>> that was missing. I take it there are no specific unit tests of the >>>>> migration files? >>>>> >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> In the migration, I can add the columns needed. What's the best >>>>>>> way to fill out those fields - using raw SQL queries or create a Session >>>>>>> object and access the VpnService object's router object? >>>>>>> >>>>>> >>>>>> If the default value for the column is not enough, and you need >>>>>> to specify a value which depends on other values in the same row I would >>>>>> prefer plain SQL statements, but if that become cumbersome I guess it's >>>>>> ok >>>>>> to use sqlalchemy's session. >>>>>> >>>>>> >>>>>>> I see there is some op.bind() call and then engine.execute(), but >>>>>>> could use some help on the best way to extract the needed queries (I >>>>>>> need >>>>>>> to access the vpnservice's router, and then access the (Port) gw_port >>>>>>> relationship, and from that access the (IPAllocation) fixed_ips list). >>>>>>> >>>>>> >>>>>> Perhaps you can point us to the review pages on gerrit, and we >>>>>> can provide detailed comments there. >>>>>> >>>>> >>>>> @PCM Yeah, I haven't pushed it up yet. I have a few more changes to >>>>> make, and should be able to get it up in a few days. The LP bug is >>>>> 1464387. >>>>> >>>>> Essentially, in the vpnservices table, I'm adding an IPv4 and/or >>>>> IPv6 addresses for the "local" end of VPN connections that will be >>>>> established. This is to allow alternative VPN implementation (appliances, >>>>> separate S/W, H/W, VM based VPN, etc) to specify addresses different than >>>>> what is available on the Neutron router. >>>>> >>>>> However, for the reference implementation, we'll use the Neutron >>>>> router's fixed_ips list (as is done today), and to handle the migration, >>>>> I'm thinking the following is needed: >>>>> >>>>> 1) create the new columns. >>>>> 2) Identify the router for that service and obtain it's GW fixed_ips >>>>> list. >>>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and >>>>> store in new columns. >>>>> >>>>> So I need to form a query and code to do this. >>>>> >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> Appreciate any advise here on how to debug the migration stuff... >>>>>>> >>>>>>> Paul Michali (pc_m) >>>>>>> >>>>>> >>>>>> [1] >>>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124 >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
