Johnathan, this would be easier to debug with a nova-compute log. Are you willing to post one somewhere that people could take a look at?
Thanks, Michael On Thu, Aug 8, 2013 at 7:35 AM, Jonathan Proulx <j...@jonproulx.com> wrote: > Hi All, > > Apologies to those who saw this on the operators list earlier, there is a > bit of new info here & having gotten no response there thought I'd take it > to a wider audience... > > > I'm almost through my grizzly upgrade. I'd upgraded everything except > nova-compute before upgrading that (ubuntu 12.04 cloud archieve pkgs). > > On most nodes the nova-compute service upgraded and restarted properly, but > on some it imediately exits with: > > CRITICAL nova [-] 'instance_type_memory_mb' > > It would seem like this is https://code.launchpad.net/bugs/1161022 but the > fix for that was released in March and I've verified is in the packaged > version I'm using. > > The referenced bug involves the DB migration only updating non-deleted > instances in the instance-system-metatata table and the patch skips the > lookups that are broken (and irrelevant) for deleted instances. > > Tracing the DB calls from the host shows it is trying to do lookups for > instances that were deleted last October, which is a bit surprising as it's > run thousands of instances since & it's not looking those up. > > It is note worthy that that is around the time I upgraded from Essex -> > Folsom so it's possible their state is weirder than most having run through > that update. > > There were directories for the instances in question in > /var/lib/nova/instances, so I thought "Aha!" and moved them, but on restart > I still get the same failure and same DB query for the old instances. Where > is nova getting the idea it should look these up & how can I stop it? > > I've go so far as to generate instance_type_<foo> entries in the > instance_system_metadata table for all instances ever on my deployment > (about 500k) but I still only have the cryptic "CRITICAL nova [-] > 'instance_type_memory_mb'" error and a failure to start, so clearly I'm > casing the wrong problem some how. > > Help? > -Jon > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > -- Rackspace Australia _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
