Jon,
The value in quota_usages was wrong. Fixed. Thanks for the help.
JR
On 12/11/2013 2:36 PM, Jonathan Proulx wrote:
> I've seen issues where the quota_use table in the nova database gets
> out of sync with the resources actually used.
>
> try this query and see if the fixed_ips matches what is in use:
>
> mysql nova -e 'select * from quota_usages where
> project_id="60d776fe573f44a4810cb294b95e09d6" and
> resource="fixed_ips";
>
> if it doesn't you'll need to manually update it and check the other
> resources for this tenant (if one is wrong usually they all are)
>
> I was just ranting privately after running into this issue today (not
> for the first time) about how having this in its own table is bad for
> these reasons & that I'd rather have the right answer slowly than the
> wrong answer quickly. Though this may not be your problem & I haven't
> yet reproduced the issue to see if I can file a proper bug (a previous
> bug related to this was fixed a long time ago)
>
> -Jon
>
> On Wed, Dec 11, 2013 at 2:11 PM, JR <botem...@gmail.com> wrote:
>> Hi all,
>>
>> I'm having an odd issue where I can't spin up new VMs; the error is below.
>> I've confirmed that the tenant starting the instance has sufficient
>> capacity.
>> I have another tenant who is running a similar number of VMs and it's not
>> having the problem.
>>
>> This is ubuntu 12.04 running grizzly.
>>
>> Any thoughts? I assume that something in the DB is wedged ...
>>
>> Thanks
>> JR
>>
>> root@nebula01:/mnt/gdm_source# su - dd_test -c 'nova quota-show'
>> +-----------------------------+--------+
>> | Property | Value |
>> +-----------------------------+--------+
>> | metadata_items | 128 |
>> | injected_file_content_bytes | 10240 |
>> | ram | 256000 |
>> | floating_ips | 128 |
>> | key_pairs | 100 |
>> | instances | 128 |
>> | security_group_rules | 20 |
>> | injected_files | 5 |
>> | cores | 128 |
>> | fixed_ips | 128 |
>> | injected_file_path_bytes | 255 |
>> | security_groups | 50 |
>> +-----------------------------+--------+
>>
>> root@nebula01:/mnt/gdm_source# su - dd_test -c 'nova show gdm-worker17-t'
>> +-----------------------------+---------------------------------------------------------------------------------+
>> | Property | Value
>> |
>> +-----------------------------+---------------------------------------------------------------------------------+
>> | status | ERROR
>> |
>> | updated | 2013-12-11T19:05:15Z
>> |
>> | OS-EXT-STS:task_state | None
>> |
>> | key_name | dd_test
>> |
>> | image | gdm-worker
>> (fe0147b8-d355-4d7e-826f-195ba97d7594) |
>> | hostId |
>> 5f1a4bd11ee027f10f8fc37b0e2ce091904b94befdf5abeb196c7f67
>> |
>> | OS-EXT-STS:vm_state | error
>> |
>> | public network |
>> |
>> | flavor | i3.small_d10g
>> (7d1037f1-dfd0-4a0e-8da7-097258c9934d) |
>> | id | d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85
>> |
>> | security_groups | [{u'name': u'dd_test'}]
>> |
>> | user_id | dcc3374a58984a92981d09f27c76a869
>> |
>> | name | gdm-worker17-t
>> |
>> | created | 2013-12-11T19:05:10Z
>> |
>> | fault | {u'message': u'NoValidHost', u'code': 500,
>> u'created': u'2013-12-11T19:05:15Z'} |
>> | OS-DCF:diskConfig | MANUAL
>> |
>> | metadata | {}
>> |
>> | accessIPv4 |
>> |
>> | accessIPv6 |
>> |
>> | tenant_id | 60d776fe573f44a4810cb294b95e09d6
>> |
>> | OS-EXT-STS:power_state | 0
>> |
>> | OS-EXT-AZ:availability_zone | nova
>> |
>> | config_drive |
>> |
>> +-----------------------------+---------------------------------------------------------------------------------+
>> root@nebula01:/mnt/gdm_source# keystone tenant-list|grep 60d
>> | 60d776fe573f44a4810cb294b95e09d6 | dd_test | True |
>>
>>
>> Error:
>>
>> 2013-12-11 12:05:15.153 ERROR nova.compute.manager
>> [req-2363d68e-a947-41af-9359-2a6e6720a0ee dcc3374a58984a92981d09f27c76a869
>> 60d776fe573f44a4810cb294b95e09d6] [instance:
>> d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85] Error: ['Traceback (most recent call
>> last):\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 847, in
>> _run_instance\n requested_networks, macs, security_groups)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1091, in
>> _allocate_network\n instance=instance)\n', ' File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>> self.gen.next()\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1087, in
>> _allocate_network\n security_groups=security_groups)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 102, in
>> wrapped\n return func(self, context, *args, **kwargs)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 47, in
>> wrapper\n res = f(self, context, *args, **kwargs)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 283, in
>> allocate_for_instance\n nw_info =
>> self.network_rpcapi.allocate_for_instance(context, **args)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 171, in
>> allocate_for_instance\n topic=topic, version=\'1.9\')\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line
>> 80, in call\n return rpc.call(context, self._get_topic(topic), msg,
>> timeout)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
>> line 140, in call\n return _get_impl().call(CONF, context, topic, msg,
>> timeout)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 798, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', '
>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>> line 612, in call\n rv = list(rv)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 561, in __iter__\n raise result\n', 'FixedIpLimitExceeded_Remote: Maximum
>> number of fixed ips exceeded\nTraceback (most recent call last):\n\n File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version, method,
>> **args)\n\n File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>> line 133, in dispatch\n return getattr(proxyobj, method)(ctxt,
>> **kwargs)\n\n File
>> "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 116,
>> in allocate_for_instance\n **kwargs)\n\n File
>> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 513, in
>> allocate_for_instance\n requested_networks=requested_networks)\n\n File
>> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 217, in
>> _allocate_fixed_ips\n vpn=vpn, address=address)\n\n File
>> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 836, in
>> allocate_fixed_ip\n raise
>> exception.FixedIpLimitExceeded()\n\nFixedIpLimitExceeded: Maximum number of
>> fixed ips exceeded\n\n']
>>
>> --
>> Your electronic communications are being monitored; strong encryption is an
>> answer. My public key
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
--
Your electronic communications are being monitored; strong encryption is
an answer. My public key
<http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4F08C504BD634953>
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
