I feel pretty dumb about this, but it was fixed by adding a rule to my security groups. I'm still very confused about some of the other behavior that I saw, but at least the problem is fixed now.
Torin Woltjer Grand Dial Communications - A ZK Tech Inc. Company 616.776.1066 ext. 2006 www.granddial.com ---------------------------------------- From: Brian Haley <haleyb....@gmail.com> Sent: 7/16/18 4:39 PM To: torin.wolt...@granddial.com, thangam.ar...@gmail.com, jpetr...@coredial.com Cc: openstack-operat...@lists.openstack.org, openstack@lists.openstack.org Subject: Re: [Openstack] [Openstack-operators] Recovering from full outage On 07/16/2018 08:41 AM, Torin Woltjer wrote: > $ip netns exec qdhcp-87a5200d-057f-475d-953d-17e873a47454 curl > http://169.254.169.254 > > > 404 Not Found > > > 404 Not Found > The resource could not be found. > > Strange, don't know where the reply came from for that. > $ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e curl > http://169.254.169.254 > curl: (7) Couldn't connect to server Based on your iptables output below, I would think the metadata proxy is running in the qrouter namespace. However, a curl from there will not work since it is restricted to only work for incoming packets from the qr- device(s). You would have to try curl from a running instance. Is there an haproxy process running? And is it listening on port 9697 in the qrouter namespace? -Brian > ------------------------------------------------------------------------ > *From*: "Torin Woltjer" > *Sent*: 7/12/18 11:16 AM > *To*: , , > "jpetr...@coredial.com" > *Cc*: openstack-operat...@lists.openstack.org, openstack@lists.openstack.org > *Subject*: Re: [Openstack] [Openstack-operators] Recovering from full outage > Checking iptables for the metadata-proxy inside of qrouter provides the > following: > $ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e > iptables-save -c | grep 169 > [0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p > tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 > [0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p > tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff > Packets:Bytes are both 0, so no traffic is touching this rule? > > Interestingly the command: > $ip netns exec qrouter-80c3bc40-b49c-446a-926f-99811adc0c5e netstat > -anep | grep 9697 > returns nothing, so there isn't actually anything running on 9697 in the > network namespace... > > This is the output without grep: > Active Internet connections (servers and established) > Proto Recv-Q Send-Q Local Address Foreign Address > State User Inode PID/Program name > raw 0 0 0.0.0.0:112 0.0.0.0:* 7 > 0 76154 8404/keepalived > raw 0 0 0.0.0.0:112 0.0.0.0:* 7 > 0 76153 8404/keepalived > Active UNIX domain sockets (servers and established) > Proto RefCnt Flags Type State I-Node PID/Program > name Path > unix 2 [ ] DGRAM 64501 7567/python2 > unix 2 [ ] DGRAM 79953 8403/keepalived > > Could the reason no traffic touching the rule be that nothing is > listening on that port, or is there a second issue down the chain? > > Curl fails even after restarting the neutron-dhcp-agent & > neutron-metadata agent. > > Thank you for this, and any future help.
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
