However, the workaround that Paul Angus noted in the ticket worked for me just fine.
From: "Tutkowski, Mike" <mike.tutkow...@netapp.com> Date: Wednesday, August 2, 2017 at 4:26 PM To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org> Subject: Re: VMware VR stuck in Starting state (master branch) I knew this issue sounded familiar. :) I logged a bug for it a couple years ago: https://issues.apache.org/jira/browse/CLOUDSTACK-9144 From: "Tutkowski, Mike" <mike.tutkow...@netapp.com> Date: Tuesday, August 1, 2017 at 10:22 PM To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org> Subject: VMware VR stuck in Starting state (master branch) Hi, I’m running some tests on master with VMware 5.5 in a Basic Zone. My SS and CP VMs come up fine, but the VR is stuck in the Starting state (even when I manually reboot the VM hosting the VR). Anyone have thoughts on this based on the cloud.log from the VR VM: Wed Aug 2 04:04:14 UTC 2017 Executing cloud-early-config Wed Aug 2 04:04:14 UTC 2017 Detected that we are running inside vmware guest Wed Aug 2 04:04:14 UTC 2017 Patching scripts oldmd5=3bd71d0ff4892529f127a46cb76d9aee newmd5=bfdc7753040c29810cfb707d21cb1cc8 Wed Aug 2 04:04:14 UTC 2017 Patching cloud service Wed Aug 2 04:04:15 UTC 2017 Rebooting system since we patched init scripts Wed Aug 2 04:04:17 UTC 2017 Updating log4j-cloud.xml Wed Aug 2 04:04:17 UTC 2017 Setting up dhcp server system vm Wed Aug 2 04:04:17 UTC 2017 checking that eth0 has IP Wed Aug 2 04:04:17 UTC 2017 checking that eth1 has IP Wed Aug 2 04:04:18 UTC 2017 waiting for eth1 interface setup with ip timer=0 Wed Aug 2 04:04:38 UTC 2017 Executing cloud-early-config Wed Aug 2 04:04:38 UTC 2017 Detected that we are running inside vmware guest Wed Aug 2 04:04:39 UTC 2017 Patching cloud service Wed Aug 2 04:04:39 UTC 2017 Updating log4j-cloud.xml Wed Aug 2 04:04:39 UTC 2017 Setting up dhcp server system vm Wed Aug 2 04:04:39 UTC 2017 checking that eth0 has IP Wed Aug 2 04:04:40 UTC 2017 waiting for eth0 interface setup with ip timer=0 Wed Aug 2 04:04:40 UTC 2017 checking that eth1 has IP Wed Aug 2 04:04:42 UTC 2017 waiting for eth1 interface setup with ip timer=0 Wed Aug 2 04:04:43 UTC 2017 waiting for eth1 interface setup with ip timer=1 Wed Aug 2 04:04:44 UTC 2017 waiting for eth1 interface setup with ip timer=2 Wed Aug 2 04:04:45 UTC 2017 waiting for eth1 interface setup with ip timer=3 Wed Aug 2 04:04:46 UTC 2017 waiting for eth1 interface setup with ip timer=4 Wed Aug 2 04:04:47 UTC 2017 waiting for eth1 interface setup with ip timer=5 Wed Aug 2 04:04:48 UTC 2017 waiting for eth1 interface setup with ip timer=6 Wed Aug 2 04:04:49 UTC 2017 waiting for eth1 interface setup with ip timer=7 Wed Aug 2 04:04:50 UTC 2017 waiting for eth1 interface setup with ip timer=8 Wed Aug 2 04:04:51 UTC 2017 waiting for eth1 interface setup with ip timer=9 Wed Aug 2 04:04:52 UTC 2017 waiting for eth1 interface setup with ip timer=10 Wed Aug 2 04:04:53 UTC 2017 waiting for eth1 interface setup with ip timer=11 Wed Aug 2 04:04:54 UTC 2017 waiting for eth1 interface setup with ip timer=12 Wed Aug 2 04:04:55 UTC 2017 waiting for eth1 interface setup with ip timer=13 Wed Aug 2 04:04:56 UTC 2017 waiting for eth1 interface setup with ip timer=14 Wed Aug 2 04:04:57 UTC 2017 waiting for eth1 interface setup with ip timer=15 Wed Aug 2 04:04:58 UTC 2017 waiting for eth1 interface setup with ip timer=16 Wed Aug 2 04:04:58 UTC 2017 interface eth1 is not set up with ip... exiting Wed Aug 2 04:05:07 UTC 2017 Setting up dnsmasq Wed Aug 2 04:05:07 UTC 2017 Setting up apache web server Wed Aug 2 04:05:07 UTC 2017 Enable service dnsmasq = 1 Wed Aug 2 04:05:07 UTC 2017 Enable service haproxy = 0 Wed Aug 2 04:05:07 UTC 2017 Processors = 1 Enable service = 0 Wed Aug 2 04:05:07 UTC 2017 Enable service cloud-passwd-srvr = 1 Wed Aug 2 04:05:07 UTC 2017 Enable service cloud = 0 Wed Aug 2 04:05:08 UTC 2017 cloud: enable_fwding = 0 Wed Aug 2 04:05:08 UTC 2017 enable_fwding = 0 2017-08-02 04:05:08,447 merge.py process:101 Command of type cmdline received 2017-08-02 04:05:08,447 merge.py process:101 Command of type ips received 2017-08-02 04:05:08,448 merge.py process:101 Command of type ips received 2017-08-02 04:05:08,451 merge.py load:291 Could not open /var/cache/cloud/cmd_line.json 2017-08-02 04:05:08,465 CsAddress.py process:107 Address found in DataBag ==> {u'public_ip': u'0.0.0.0', u'one_to_one_nat': False, u'nic_dev_id': u'1', u'network': u'0.0.0.0/0', u'netmask': u'0.0.0.0', u'source_nat': False, u'broadcast': u'255.255.255.255', u'add': True, u'nw_type': u'control', u'device': u'eth1', u'cidr': u'0.0.0.0/0', u'gateway': u'10.117.40.126', u'size': u'0'} 2017-08-02 04:05:08,465 CsAddress.py process:120 Address 0.0.0.0/0 on device eth1 not configured 2017-08-02 04:05:08,466 CsAddress.py configure:278 Configuring address 0.0.0.0/0 on device eth1 2017-08-02 04:05:08,470 CsRoute.py defaultroute_exists:104 Checking if default ipv4 route is present 2017-08-02 04:05:08,475 CsRoute.py defaultroute_exists:108 Default route found: default via 10.117.40.126 dev eth0 2017-08-02 04:05:08,479 CsAddress.py process:107 Address found in DataBag ==> {u'public_ip': u'10.117.40.29', u'one_to_one_nat': False, u'nic_dev_id': u'0', u'network': u'10.117.40.0/25', u'netmask': u'255.255.255.128', u'source_nat': False, u'broadcast': u'10.117.40.127', u'add': True, u'nw_type': u'guest', u'device': u'eth0', u'cidr': u'10.117.40.29/25', u'gateway': u'10.117.40.126', u'size': u'25'} 2017-08-02 04:05:08,480 CsAddress.py process:115 Address 10.117.40.29/25 on device eth0 already configured 2017-08-02 04:05:08,480 CsRoute.py add_table:38 Adding route table: 0 Table_eth0 to /etc/iproute2/rt_tables if not present 2017-08-02 04:05:08,548 CsRule.py addMark:49 Added fwmark rule for Table_eth0 2017-08-02 04:05:08,562 CsRoute.py add_route:55 Adding route: dev eth0 table: Table_eth0 network: 10.117.40.126 if not present 2017-08-02 04:05:08,568 CsRoute.py set_route:75 Add default via 10.117.40.126 table Table_eth0 proto static 2017-08-02 04:05:08,573 CsRoute.py add_network_route:65 Adding route: dev eth0 table: Table_eth0 network: 10.117.40.0/25 if not present 2017-08-02 04:05:08,589 CsHelper.py copy:254 Copied /etc/apache2/vhost.template to /etc/apache2/sites-enabled/vhost-10.117.40.29.conf 2017-08-02 04:05:08,592 CsFile.py commit:66 Wrote edited file /etc/apache2/sites-enabled/vhost-10.117.40.29.conf 2017-08-02 04:05:08,592 CsFile.py commit:68 Updated file in-cache configuration 2017-08-02 04:05:08,860 CsHelper.py service:216 Service apache2 restart 2017-08-02 04:05:08,876 CsProcess.py start:33 Started dummy 2017-08-02 04:05:08,878 CsRoute.py defaultroute_exists:104 Checking if default ipv4 route is present 2017-08-02 04:05:08,885 CsRoute.py defaultroute_exists:108 Default route found: default via 10.117.40.126 dev eth0 2017-08-02 04:05:08,900 CsFile.py commit:66 Wrote edited file /etc/hosts 2017-08-02 04:05:08,900 CsFile.py commit:68 Updated file in-cache configuration 2017-08-02 04:05:08,900 CsDhcp.py write_hosts:128 Updated hosts file 2017-08-02 04:05:08,900 CsFile.py commit:60 Nothing to commit. The /etc/dnsmasq.d/cloud.conf file did not change 2017-08-02 04:05:08,900 CsFile.py commit:60 Nothing to commit. The /etc/dhcphosts.txt file did not change 2017-08-02 04:05:08,900 CsFile.py commit:60 Nothing to commit. The /etc/dhcpopts.txt file did not change 2017-08-02 04:05:10,995 CsHelper.py service:216 Service dnsmasq restart 2017-08-02 04:05:11,156 CsNetfilter.py compare:163 Add: rule=-A FW_OUTBOUND -m state --state RELATED,ESTABLISHED -j ACCEPT table=filter 2017-08-02 04:05:11,170 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth1 -p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j ACCEPT table=filter 2017-08-02 04:05:11,183 CsNetfilter.py compare:163 Add: rule=-P INPUT DROP table=filter 2017-08-02 04:05:11,188 CsNetfilter.py compare:163 Add: rule=-P FORWARD DROP table=filter 2017-08-02 04:05:11,193 CsNetfilter.py compare:163 Add: rule=-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff table=mangle 2017-08-02 04:05:11,202 CsNetfilter.py compare:163 Add: rule=-A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill table=mangle 2017-08-02 04:05:11,213 CsNetfilter.py compare:163 Add: rule=-A INPUT -d 224.0.0.18/32 -j ACCEPT table=filter 2017-08-02 04:05:11,218 CsNetfilter.py compare:163 Add: rule=-A INPUT -d 225.0.0.50/32 -j ACCEPT table=filter 2017-08-02 04:05:11,223 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT table=filter 2017-08-02 04:05:11,228 CsNetfilter.py compare:163 Add: rule=-A INPUT -p icmp -j ACCEPT table=filter 2017-08-02 04:05:11,233 CsNetfilter.py compare:163 Add: rule=-A INPUT -i lo -j ACCEPT table=filter 2017-08-02 04:05:11,238 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -p udp -m udp --dport 67 -j ACCEPT table=filter 2017-08-02 04:05:11,243 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -p udp -m udp --dport 53 -s 10.117.40.0/25 -j ACCEPT table=filter 2017-08-02 04:05:11,248 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -p tcp -m tcp --dport 53 -s 10.117.40.0/25 -j ACCEPT table=filter 2017-08-02 04:05:11,253 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT table=filter 2017-08-02 04:05:11,259 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT table=filter 2017-08-02 04:05:11,264 CsNetfilter.py compare:163 Add: rule=-A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT table=filter 2017-08-02 04:05:11,270 CsNetfilter.py compare:163 Add: rule=-A FORWARD -i eth0 -o eth0 -m state --state NEW -j ACCEPT table=filter 2017-08-02 04:05:11,275 CsNetfilter.py compare:163 Add: rule=-A FORWARD -i eth0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT table=filter 2017-08-02 04:05:11,280 CsNetfilter.py compare:163 Add: rule=-A FORWARD -j NETWORK_STATS table=filter 2017-08-02 04:05:11,285 CsNetfilter.py compare:163 Add: rule=-A INPUT -j NETWORK_STATS table=filter 2017-08-02 04:05:11,290 CsNetfilter.py compare:163 Add: rule=-A OUTPUT -j NETWORK_STATS table=filter 2017-08-02 04:05:11,296 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS -i eth0 -o eth2 table=filter 2017-08-02 04:05:11,301 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS -i eth2 -o eth0 table=filter 2017-08-02 04:05:11,306 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS -o eth2 ! -i eth0 -p tcp table=filter 2017-08-02 04:05:11,311 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS -i eth2 ! -o eth0 -p tcp table=filter 2017-08-02 04:05:11,317 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -p udp -m udp --dport 67 -j ACCEPT table=filter 2017-08-02 04:05:11,322 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -d 10.117.40.29/32 -p udp -m udp --dport 53 -j ACCEPT table=filter 2017-08-02 04:05:11,327 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -d 10.117.40.29/32 -p tcp -m tcp --dport 53 -j ACCEPT table=filter 2017-08-02 04:05:11,332 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -d 10.117.40.29/32 -p tcp -m tcp -m state --state NEW --dport 80 -j ACCEPT table=filter 2017-08-02 04:05:11,338 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0 -d 10.117.40.29/32 -p tcp -m tcp -m state --state NEW --dport 443 -j ACCEPT table=filter 2017-08-02 04:05:11,396 CsHelper.py service:216 Service conntrackd stop 2017-08-02 04:05:11,411 CsHelper.py service:216 Service keepalived stop Thanks! Mike
