Hi Openstackers, We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone can help ;)
When our hosts boots up, the ARP cache population loop of L3 Agent is delaying the start of neutron-ns-metadata-proxy for around a minute -- see logs below; then, when nova-compute launches VMs, all of cloud-init runs fail with timeout when reading metadata To workaround this, we've made a systemd unit on which nova-compute is dependent; this unit waits for ns-metadata-proxy process to appear, and only then nova-compute starts Curiously, in dvr_local_router.py, in _update_arp_entry function, there is a comment saying "# TODO(mrsmith): optimize the calls below for bulk calls"... By now we have a single virtual router with 170 VMs, but the number of VMs will grow, so my questions are Should this be issue of concern? Is there a better / faster / bulk way to execute those "ip neigh" commands? Or simply, metadata proxy should launch before ARP cache population? PD: I've also seen (obviously) this ARP cache population in the L3 agent of Neutron Nodes, and I hope it does not affect / delay the HA failover mechanism... (didn't test yet) # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ; PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf,,g' | head -25 Dec 13 13:33:43 e71-host15 sudo[20157]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid --metadata_proxy_socket=/var/ Dec 13 13:33:55 e71-host15 sudo[20309]: ip -o netns list Dec 13 13:33:55 e71-host15 sudo[20315]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.ip_forward=1 Dec 13 13:33:55 e71-host15 sudo[20322]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv6.conf.all.forwarding=1 Dec 13 13:33:56 e71-host15 sudo[20331]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show rfp-6149559f-f Dec 13 13:33:56 e71-host15 sudo[20336]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:56 e71-host15 sudo[20342]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:56 e71-host15 sudo[20345]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip addr show qr-24f3070a-d4 permanent Dec 13 13:33:56 e71-host15 sudo[20348]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route list dev qr-24f3070a-d4 scope link Dec 13 13:33:56 e71-host15 sudo[20354]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -6 route list dev qr-24f3070a-d4 scope link Dec 13 13:33:56 e71-host15 sudo[20357]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 arping -A -I qr-24f3070a-d4 -c 3 -w 4.5 10.96.0.1 Dec 13 13:33:57 e71-host15 sudo[20368]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:57 e71-host15 sudo[20372]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.100 lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4 Dec 13 13:33:57 e71-host15 sudo[20375]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:57 e71-host15 sudo[20378]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.101 lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20384]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20387]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.102 lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20390]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20393]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.103 lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20399]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20402]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.104 lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4 Dec 13 13:33:58 e71-host15 sudo[20405]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:33:59 e71-host15 sudo[20411]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.105 lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4 ... ... ... # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ; PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf,,g' | tail -25 Dec 13 13:34:51 e71-host15 sudo[21771]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.1.83 lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4 Dec 13 13:34:51 e71-host15 sudo[21777]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default via 10.96.0.2 dev qr-24f3070a-d4 table 174063617 Dec 13 13:34:51 e71-host15 sudo[21780]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show Dec 13 13:34:52 e71-host15 sudo[21783]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.conf.qr-24f3070a-d4.send_redirects=0 Dec 13 13:34:52 e71-host15 sudo[21786]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth 1 -type l -printf %f Dec 13 13:34:52 e71-host15 sudo[21789]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.conf.all.send_redirects=0 Dec 13 13:34:52 e71-host15 sudo[21792]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default via 10.96.0.2 dev qr-24f3070a-d4 table 174063617 Dec 13 13:34:52 e71-host15 sudo[21795]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show Dec 13 13:34:52 e71-host15 sudo[21801]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.conf.qr-24f3070a-d4.send_redirects=0 Dec 13 13:34:53 e71-host15 sudo[21804]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4 Dec 13 13:34:53 e71-host15 sudo[21807]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.2 lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4 Dec 13 13:34:53 e71-host15 sudo[21817]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth 1 -type l -printf %f Dec 13 13:34:53 e71-host15 sudo[21822]: ip netns exec fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f Dec 13 13:34:53 e71-host15 sudo[21827]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save Dec 13 13:34:53 e71-host15 sudo[21833]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n Dec 13 13:34:54 e71-host15 sudo[21836]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save Dec 13 13:34:54 e71-host15 sudo[21839]: ip netns exec fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f Dec 13 13:34:54 e71-host15 sudo[21844]: ip netns exec fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f Dec 13 13:34:54 e71-host15 sudo[21849]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save Dec 13 13:34:54 e71-host15 sudo[21852]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n Dec 13 13:34:54 e71-host15 sudo[21855]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save Dec 13 13:34:54 e71-host15 sudo[21861]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save Dec 13 13:34:55 e71-host15 sudo[21864]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n Dec 13 13:34:55 e71-host15 sudo[21869]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save Dec 13 13:34:55 e71-host15 sudo[21877]: ip netns exec qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=6149559f-fa54-493c-bf37-7d1827181228 --state_path=/var/lib/neutron --metadata_port=9697 --metadata_proxy_user=118 --metadata_proxy_group=120 --debug --verbose --use-syslog --syslog-log-facility=LOG_LOCAL4
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
