Hi,
I’m running some tests on master with VMware 5.5 in a Basic Zone. My SS and CP
VMs come up fine, but the VR is stuck in the Starting state (even when I
manually reboot the VM hosting the VR).
Anyone have thoughts on this based on the cloud.log from the VR VM:
Wed Aug 2 04:04:14 UTC 2017 Executing cloud-early-config
Wed Aug 2 04:04:14 UTC 2017 Detected that we are running inside vmware guest
Wed Aug 2 04:04:14 UTC 2017 Patching scripts
oldmd5=3bd71d0ff4892529f127a46cb76d9aee newmd5=bfdc7753040c29810cfb707d21cb1cc8
Wed Aug 2 04:04:14 UTC 2017 Patching cloud service
Wed Aug 2 04:04:15 UTC 2017 Rebooting system since we patched init scripts
Wed Aug 2 04:04:17 UTC 2017 Updating log4j-cloud.xml
Wed Aug 2 04:04:17 UTC 2017 Setting up dhcp server system vm
Wed Aug 2 04:04:17 UTC 2017 checking that eth0 has IP
Wed Aug 2 04:04:17 UTC 2017 checking that eth1 has IP
Wed Aug 2 04:04:18 UTC 2017 waiting for eth1 interface setup with ip timer=0
Wed Aug 2 04:04:38 UTC 2017 Executing cloud-early-config
Wed Aug 2 04:04:38 UTC 2017 Detected that we are running inside vmware guest
Wed Aug 2 04:04:39 UTC 2017 Patching cloud service
Wed Aug 2 04:04:39 UTC 2017 Updating log4j-cloud.xml
Wed Aug 2 04:04:39 UTC 2017 Setting up dhcp server system vm
Wed Aug 2 04:04:39 UTC 2017 checking that eth0 has IP
Wed Aug 2 04:04:40 UTC 2017 waiting for eth0 interface setup with ip timer=0
Wed Aug 2 04:04:40 UTC 2017 checking that eth1 has IP
Wed Aug 2 04:04:42 UTC 2017 waiting for eth1 interface setup with ip timer=0
Wed Aug 2 04:04:43 UTC 2017 waiting for eth1 interface setup with ip timer=1
Wed Aug 2 04:04:44 UTC 2017 waiting for eth1 interface setup with ip timer=2
Wed Aug 2 04:04:45 UTC 2017 waiting for eth1 interface setup with ip timer=3
Wed Aug 2 04:04:46 UTC 2017 waiting for eth1 interface setup with ip timer=4
Wed Aug 2 04:04:47 UTC 2017 waiting for eth1 interface setup with ip timer=5
Wed Aug 2 04:04:48 UTC 2017 waiting for eth1 interface setup with ip timer=6
Wed Aug 2 04:04:49 UTC 2017 waiting for eth1 interface setup with ip timer=7
Wed Aug 2 04:04:50 UTC 2017 waiting for eth1 interface setup with ip timer=8
Wed Aug 2 04:04:51 UTC 2017 waiting for eth1 interface setup with ip timer=9
Wed Aug 2 04:04:52 UTC 2017 waiting for eth1 interface setup with ip timer=10
Wed Aug 2 04:04:53 UTC 2017 waiting for eth1 interface setup with ip timer=11
Wed Aug 2 04:04:54 UTC 2017 waiting for eth1 interface setup with ip timer=12
Wed Aug 2 04:04:55 UTC 2017 waiting for eth1 interface setup with ip timer=13
Wed Aug 2 04:04:56 UTC 2017 waiting for eth1 interface setup with ip timer=14
Wed Aug 2 04:04:57 UTC 2017 waiting for eth1 interface setup with ip timer=15
Wed Aug 2 04:04:58 UTC 2017 waiting for eth1 interface setup with ip timer=16
Wed Aug 2 04:04:58 UTC 2017 interface eth1 is not set up with ip... exiting
Wed Aug 2 04:05:07 UTC 2017 Setting up dnsmasq
Wed Aug 2 04:05:07 UTC 2017 Setting up apache web server
Wed Aug 2 04:05:07 UTC 2017 Enable service dnsmasq = 1
Wed Aug 2 04:05:07 UTC 2017 Enable service haproxy = 0
Wed Aug 2 04:05:07 UTC 2017 Processors = 1 Enable service = 0
Wed Aug 2 04:05:07 UTC 2017 Enable service cloud-passwd-srvr = 1
Wed Aug 2 04:05:07 UTC 2017 Enable service cloud = 0
Wed Aug 2 04:05:08 UTC 2017 cloud: enable_fwding = 0
Wed Aug 2 04:05:08 UTC 2017 enable_fwding = 0
2017-08-02 04:05:08,447 merge.py process:101 Command of type cmdline received
2017-08-02 04:05:08,447 merge.py process:101 Command of type ips received
2017-08-02 04:05:08,448 merge.py process:101 Command of type ips received
2017-08-02 04:05:08,451 merge.py load:291 Could not open
/var/cache/cloud/cmd_line.json
2017-08-02 04:05:08,465 CsAddress.py process:107 Address found in DataBag ==>
{u'public_ip': u'0.0.0.0', u'one_to_one_nat': False, u'nic_dev_id': u'1',
u'network': u'0.0.0.0/0', u'netmask': u'0.0.0.0', u'source_nat': False,
u'broadcast': u'255.255.255.255', u'add': True, u'nw_type': u'control',
u'device': u'eth1', u'cidr': u'0.0.0.0/0', u'gateway': u'10.117.40.126',
u'size': u'0'}
2017-08-02 04:05:08,465 CsAddress.py process:120 Address 0.0.0.0/0 on device
eth1 not configured
2017-08-02 04:05:08,466 CsAddress.py configure:278 Configuring address
0.0.0.0/0 on device eth1
2017-08-02 04:05:08,470 CsRoute.py defaultroute_exists:104 Checking if default
ipv4 route is present
2017-08-02 04:05:08,475 CsRoute.py defaultroute_exists:108 Default route
found: default via 10.117.40.126 dev eth0
2017-08-02 04:05:08,479 CsAddress.py process:107 Address found in DataBag ==>
{u'public_ip': u'10.117.40.29', u'one_to_one_nat': False, u'nic_dev_id': u'0',
u'network': u'10.117.40.0/25', u'netmask': u'255.255.255.128', u'source_nat':
False, u'broadcast': u'10.117.40.127', u'add': True, u'nw_type': u'guest',
u'device': u'eth0', u'cidr': u'10.117.40.29/25', u'gateway': u'10.117.40.126',
u'size': u'25'}
2017-08-02 04:05:08,480 CsAddress.py process:115 Address 10.117.40.29/25 on
device eth0 already configured
2017-08-02 04:05:08,480 CsRoute.py add_table:38 Adding route table: 0
Table_eth0 to /etc/iproute2/rt_tables if not present
2017-08-02 04:05:08,548 CsRule.py addMark:49 Added fwmark rule for Table_eth0
2017-08-02 04:05:08,562 CsRoute.py add_route:55 Adding route: dev eth0 table:
Table_eth0 network: 10.117.40.126 if not present
2017-08-02 04:05:08,568 CsRoute.py set_route:75 Add default via 10.117.40.126
table Table_eth0 proto static
2017-08-02 04:05:08,573 CsRoute.py add_network_route:65 Adding route: dev eth0
table: Table_eth0 network: 10.117.40.0/25 if not present
2017-08-02 04:05:08,589 CsHelper.py copy:254 Copied
/etc/apache2/vhost.template to
/etc/apache2/sites-enabled/vhost-10.117.40.29.conf
2017-08-02 04:05:08,592 CsFile.py commit:66 Wrote edited file
/etc/apache2/sites-enabled/vhost-10.117.40.29.conf
2017-08-02 04:05:08,592 CsFile.py commit:68 Updated file in-cache configuration
2017-08-02 04:05:08,860 CsHelper.py service:216 Service apache2 restart
2017-08-02 04:05:08,876 CsProcess.py start:33 Started dummy
2017-08-02 04:05:08,878 CsRoute.py defaultroute_exists:104 Checking if default
ipv4 route is present
2017-08-02 04:05:08,885 CsRoute.py defaultroute_exists:108 Default route
found: default via 10.117.40.126 dev eth0
2017-08-02 04:05:08,900 CsFile.py commit:66 Wrote edited file /etc/hosts
2017-08-02 04:05:08,900 CsFile.py commit:68 Updated file in-cache configuration
2017-08-02 04:05:08,900 CsDhcp.py write_hosts:128 Updated hosts file
2017-08-02 04:05:08,900 CsFile.py commit:60 Nothing to commit. The
/etc/dnsmasq.d/cloud.conf file did not change
2017-08-02 04:05:08,900 CsFile.py commit:60 Nothing to commit. The
/etc/dhcphosts.txt file did not change
2017-08-02 04:05:08,900 CsFile.py commit:60 Nothing to commit. The
/etc/dhcpopts.txt file did not change
2017-08-02 04:05:10,995 CsHelper.py service:216 Service dnsmasq restart
2017-08-02 04:05:11,156 CsNetfilter.py compare:163 Add: rule=-A FW_OUTBOUND -m
state --state RELATED,ESTABLISHED -j ACCEPT table=filter
2017-08-02 04:05:11,170 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth1
-p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j ACCEPT
table=filter
2017-08-02 04:05:11,183 CsNetfilter.py compare:163 Add: rule=-P INPUT DROP
table=filter
2017-08-02 04:05:11,188 CsNetfilter.py compare:163 Add: rule=-P FORWARD DROP
table=filter
2017-08-02 04:05:11,193 CsNetfilter.py compare:163 Add: rule=-A PREROUTING -m
state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask
0xffffffff --ctmask 0xffffffff table=mangle
2017-08-02 04:05:11,202 CsNetfilter.py compare:163 Add: rule=-A POSTROUTING -p
udp -m udp --dport 68 -j CHECKSUM --checksum-fill table=mangle
2017-08-02 04:05:11,213 CsNetfilter.py compare:163 Add: rule=-A INPUT -d
224.0.0.18/32 -j ACCEPT table=filter
2017-08-02 04:05:11,218 CsNetfilter.py compare:163 Add: rule=-A INPUT -d
225.0.0.50/32 -j ACCEPT table=filter
2017-08-02 04:05:11,223 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-m state --state RELATED,ESTABLISHED -j ACCEPT table=filter
2017-08-02 04:05:11,228 CsNetfilter.py compare:163 Add: rule=-A INPUT -p icmp
-j ACCEPT table=filter
2017-08-02 04:05:11,233 CsNetfilter.py compare:163 Add: rule=-A INPUT -i lo -j
ACCEPT table=filter
2017-08-02 04:05:11,238 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-p udp -m udp --dport 67 -j ACCEPT table=filter
2017-08-02 04:05:11,243 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-p udp -m udp --dport 53 -s 10.117.40.0/25 -j ACCEPT table=filter
2017-08-02 04:05:11,248 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-p tcp -m tcp --dport 53 -s 10.117.40.0/25 -j ACCEPT table=filter
2017-08-02 04:05:11,253 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT table=filter
2017-08-02 04:05:11,259 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT table=filter
2017-08-02 04:05:11,264 CsNetfilter.py compare:163 Add: rule=-A FORWARD -i
eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT table=filter
2017-08-02 04:05:11,270 CsNetfilter.py compare:163 Add: rule=-A FORWARD -i
eth0 -o eth0 -m state --state NEW -j ACCEPT table=filter
2017-08-02 04:05:11,275 CsNetfilter.py compare:163 Add: rule=-A FORWARD -i
eth0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT table=filter
2017-08-02 04:05:11,280 CsNetfilter.py compare:163 Add: rule=-A FORWARD -j
NETWORK_STATS table=filter
2017-08-02 04:05:11,285 CsNetfilter.py compare:163 Add: rule=-A INPUT -j
NETWORK_STATS table=filter
2017-08-02 04:05:11,290 CsNetfilter.py compare:163 Add: rule=-A OUTPUT -j
NETWORK_STATS table=filter
2017-08-02 04:05:11,296 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS
-i eth0 -o eth2 table=filter
2017-08-02 04:05:11,301 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS
-i eth2 -o eth0 table=filter
2017-08-02 04:05:11,306 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS
-o eth2 ! -i eth0 -p tcp table=filter
2017-08-02 04:05:11,311 CsNetfilter.py compare:163 Add: rule=-A NETWORK_STATS
-i eth2 ! -o eth0 -p tcp table=filter
2017-08-02 04:05:11,317 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-p udp -m udp --dport 67 -j ACCEPT table=filter
2017-08-02 04:05:11,322 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-d 10.117.40.29/32 -p udp -m udp --dport 53 -j ACCEPT table=filter
2017-08-02 04:05:11,327 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-d 10.117.40.29/32 -p tcp -m tcp --dport 53 -j ACCEPT table=filter
2017-08-02 04:05:11,332 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-d 10.117.40.29/32 -p tcp -m tcp -m state --state NEW --dport 80 -j ACCEPT
table=filter
2017-08-02 04:05:11,338 CsNetfilter.py compare:163 Add: rule=-A INPUT -i eth0
-d 10.117.40.29/32 -p tcp -m tcp -m state --state NEW --dport 443 -j ACCEPT
table=filter
2017-08-02 04:05:11,396 CsHelper.py service:216 Service conntrackd stop
2017-08-02 04:05:11,411 CsHelper.py service:216 Service keepalived stop
Thanks!
Mike
