On Fri, Nov 2, 2018 at 9:05 AM Bernd Bausch <berndbau...@gmail.com> wrote:
> Welcome to OpenStack! I think the joy of troubleshooting it is one of its
> main selling points.
>
> The conductor log says "*libxenlight failed to create new domain*" on
> various nodes. You should check the nova-compute logs on those nodes for
> relevant messages. I suspect an ill-configured interface between Nova
> Compute and libvirt/Xen. In other words, double-check the libvirt and xen
> sections of nova.conf on those nodes.
>
> Then there is this message "*Invalid **input for field
> 'identity/password/user/password': None is not of type 'string' ....*".
> This is often caused by incompatible software. I.e. you installed Openstack
> components that don't fit together. Might also be caused by a configuration
> error.
> Or it's an ordinary bug, but it seems to occur on node 2 only.
> On 11/1/2018 11:01 PM, Minjun Hong wrote:
>
> Hi everyone.
> I'm really new to OpenStack.
> After I install essential components of OpenStack, such as Nova, Keystone,
> etc, I attempted creating an instance through Openstack command in the
> terminal.
> But a trouble has occurred. It was logged in "nova-conductor.log":
>
> 2018-11-01 22:18:59.831 2570 ERROR nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node2 (node
>> node2): [u'Traceback (most recent call last):\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in
>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u'
>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120,
>> in _build_and_run_instance\n instance_uuid=instance.uuid,
>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: Invalid input for
>> field 'identity/password/user/password': None is not of type
>> 'string'\n\nFailed validating 'type' in
>> schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n
>> {'type': 'string'}\n\nOn
>> instance['identity']['password']['user']['password']:\n None (HTTP 400)
>> (Request-ID: req-350beee4-2fed-4645-9e21-79806d7ebfe7)\n"]
>> 2018-11-01 22:18:59.833 2570 WARNING oslo_config.cfg
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Option "os_region_name"
>> from group "placement" is deprecated. Use option "region-name" from group
>> "placement".
>> 2018-11-01 22:19:10.936 2571 ERROR nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node4 (node
>> node4): [u'Traceback (most recent call last):\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in
>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u'
>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120,
>> in _build_and_run_instance\n instance_uuid=instance.uuid,
>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: internal error:
>> libxenlight failed to create new domain 'instance-00000005'\n"]
>> 2018-11-01 22:19:21.783 2567 ERROR nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Error from last host: node5 (node
>> node5): [u'Traceback (most recent call last):\n', u' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1840, in
>> _do_build_and_run_instance\n filter_properties, request_spec)\n', u'
>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120,
>> in _build_and_run_instance\n instance_uuid=instance.uuid,
>> reason=six.text_type(e))\n', u"RescheduledException: Build of instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee was re-scheduled: internal error:
>> libxenlight failed to create new domain 'instance-00000005'\n"]
>> 2018-11-01 22:19:21.783 2567 WARNING nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] Failed to
>> compute_task_build_instances: Exceeded maximum number of retries. Exceeded
>> max scheduling attempts 3 for instance
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee. Last exception: internal error:
>> libxenlight failed to create new domain 'instance-00000005':
>> MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max
>> scheduling attempts 3 for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee.
>> Last exception: internal error: libxenlight failed to create new domain
>> 'instance-00000005'
>> 2018-11-01 22:19:21.784 2567 WARNING nova.scheduler.utils
>> [req-12aae2ff-4186-4ab0-964c-35b335c3188a cc22ec575cb44e53aced9ddf58d9e8d7
>> 965ff1c2002d4c278b5f838dbdbbb780 - default default] [instance:
>> 684b0a7d-22b9-4c87-88f8-b1474d3f9cee] Setting instance to ERROR state.:
>> MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max
>> scheduling attempts 3 for instance 684b0a7d-22b9-4c87-88f8-b1474d3f9cee.
>> Last exception: internal error: libxenlight failed to create new domain
>> 'instance-00000005'
>
>
> I'm not sure which component is involved in this trouble.
> And libvirt and Xen have been successfully installed on all compute node I
> have without any problem.
>
> nickeys@node2:~$ virsh create ./test.xml
>> Domain guest1 created from ./test.xml
>> nickeys@node2:~$ virsh list
>> Id Name State
>> --------------------------
>> 0 Domain-0 running
>> 1 guest1 running
>> nickeys@node2:~$
>
>
> What should I check first for that issue ?
> Your hint would be big help for me.
>
> Thanks!
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
Thanks for your kind answer, Bernd.
I checked the configure files which you mentioned like it can cause the
trouble. following files are my nova.conf and nova-compute.conf of the
compute node ('node2').
# /etc/nova/nova.conf
[DEFAULT]
> dhcpbridge_flagfile=/etc/nova/nova.conf
> dhcpbridge=/usr/bin/nova-dhcpbridge
> state_path=/var/lib/nova
> lock_path=/var/lock/nova
> force_dhcp_release=True
> libvirt_use_virtio_for_bridges=True
> verbose=True
> ec2_private_dns_show_ip=True
> api_paste_config=/etc/nova/api-paste.ini
> enabled_apis=osapi_compute,metadata
> transport_url = rabbit://admin:1234@10.150.3.88
> my_ip = 10.150.21.182
> use_neutron = True
> firewall_driver = nova.virt.firewall.NoopFirewallDriver
> [api]
> auth_strategy = keystone
> [vnc]
> enabled = True
> server_listen = 0.0.0.0
> vncserver_proxyclient_address = 10.150.21.182
> novncproxy_base_url = http://10.150.3.88:6080/vnc_auth.html
> [glance]
> api_servers = http://10.150.3.88:9292
> [oslo_concurrency]
> lock_path = /var/lib/nova/tmp
> [keystone_authtoken]
> auth_url = http://10.150.3.88:5000/v3
> memcached_servers = 10.150.3.88:11211
> auth_type = password
> project_domain_name = default
> user_domain_name = default
> project_name = service
> username = nova
> password = <KEYSTONE_PASS>
> [placement]
> os_region_name = RegionOne
> project_domain_name = Default
> project_name = service
> auth_type = password
> user_domain_name = Default
> auth_url = http://10.150.3.88:5000/v3
> username = placement
> password = <PLACEMENT_PASS>
> [database]
> connection=sqlite:////var/lib/nova/nova.sqlite
> [api_database]
> connection=sqlite:////var/lib/nova/nova.sqlite
> [neutron]
> url = http://10.150.3.88:9696
> auth_url = http://10.150.3.88:5000
> auth_type = password
> project_domain_name = default
> user_domain_name = default
> region_name = RegionOne
> project_name = service
> username = neutron
> passsword = <NEUTRON_PASS>
# /etc/nova/nova-compute.conf
> [DEFAULT]
> compute_driver=libvirt.LibvirtDriver
> [libvirt]
> virt_type=xen
And I set all the compute nodes I'm using with the same configure file
above.
Also, I installed the OpenStack components, which belong to Queens version,
following the official installation guide (
https://docs.openstack.org/install-guide/).
By the way, I found something weird in '/var/log/nova/nova-compute.log' of
the 'node2'.
After I manually created a VM by 'virsh' command (i.g. $ virsh create
test.xml), nova-compute of 'node2' started to complain.
And on an other node which have not created a virtual machine by the
'virsh' command, there is not below error.
I guess that the nova-compute component was configured to qemu as
virtualization type, but I have no idea where the setting value is.
2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> [req-ec324126-6dfc-4011-9f45-41d014e6f900 - - - - -] Error updating
> resources for node node2.: InvalidDiskInfo: Disk info file is invalid:
> qemu-img failed to execute on
> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while
> running command.
> Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824
> --cpu=30 -- env LC_ALL=C LANG=C qemu-img info
> /home/caslab/xenguest1/domains/guest1/disk.img
> Exit code: 1
> Stdout: u''
> Stderr: u"qemu-img: Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Traceback (most
> recent call last):
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7284, in
> update_available_resource_for_node
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> rt.update_available_resource(context, nodename)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line
> 664, in update_available_resource
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager resources =
> self.driver.get_available_resource(nodename)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6440,
> in get_available_resource
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
> disk_over_committed = self._get_disk_over_committed_size_total()
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8019,
> in _get_disk_over_committed_size_total
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager config,
> block_device_info)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7918,
> in _get_instance_disk_info_from_config
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager dk_size =
> disk_api.get_allocated_disk_size(path)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 147, in
> get_allocated_disk_size
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager return
> images.qemu_img_info(path).disk_size
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager File
> "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in
> qemu_img_info
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager raise
> exception.InvalidDiskInfo(reason=msg)
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager InvalidDiskInfo:
> Disk info file is invalid: qemu-img failed to execute on
> /home/caslab/xenguest1/domains/guest1/disk.img : Unexpected error while
> running command.
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Command:
> /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 --
> env LC_ALL=C LANG=C qemu-img info
> /home/caslab/xenguest1/domains/guest1/disk.img
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Exit code: 1
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stdout: u''
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager Stderr:
> u"qemu-img: Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Could not open
> '/home/caslab/xenguest1/domains/guest1/disk.img': Permission denied\n"
> 2018-11-02 18:19:13.048 16305 ERROR nova.compute.manager
Thanks!
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
