Thanks Brian for the link, it's exactly what I was looking for.
On Thu, Dec 15, 2016 at 5:29 PM, Brian Haley <brian.ha...@hpe.com> wrote: > On 12/13/2016 02:45 PM, Gustavo Randich wrote: > >> Hi Openstackers, >> >> We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone >> can >> help ;) >> >> When our hosts boots up, the ARP cache population loop of L3 Agent is >> delaying >> the start of neutron-ns-metadata-proxy for around a minute -- see logs >> below; >> then, when nova-compute launches VMs, all of cloud-init runs fail with >> timeout >> when reading metadata >> > > Hi Gustavo, > > We had seen a similar slowdown with DVR and the ARP cache, see: > > https://bugs.launchpad.net/neutron/+bug/1511134 > https://review.openstack.org/#/c/239543/ > > We decided against that approach in favor of using privsep and the > pyroute2 library. That adoption has not been as fast as we hoped, so it is > probably time to re-visit this decision and possibly resurrect that change. > > -Brian > > > To workaround this, we've made a systemd unit on which nova-compute is >> dependent; this unit waits for ns-metadata-proxy process to appear, and >> only >> then nova-compute starts >> >> Curiously, in dvr_local_router.py, in _update_arp_entry function, there >> is a >> comment saying "# TODO(mrsmith): optimize the calls below for bulk >> calls"... >> >> By now we have a single virtual router with 170 VMs, but the number of >> VMs will >> grow, so my questions are >> >> Should this be issue of concern? >> >> Is there a better / faster / bulk way to execute those "ip neigh" >> commands? >> >> Or simply, metadata proxy should launch before ARP cache population? >> >> >> >> >> PD: I've also seen (obviously) this ARP cache population in the L3 agent >> of >> Neutron Nodes, and I hope it does not affect / delay the HA failover >> mechanism... (didn't test yet) >> >> >> >> >> # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-root >> wrap >> /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ; >> PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap >> /etc/neutron/rootwrap.conf,,g' | head -25 >> >> Dec 13 13:33:43 e71-host15 sudo[20157]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy >> --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c >> -bf37-7d1827181228.pid >> --metadata_proxy_socket=/var/ >> Dec 13 13:33:55 e71-host15 sudo[20309]: ip -o netns list >> Dec 13 13:33:55 e71-host15 sudo[20315]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w >> net.ipv4.ip_forward=1 >> Dec 13 13:33:55 e71-host15 sudo[20322]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w >> net.ipv6.conf.all.forwarding=1 >> Dec 13 13:33:56 e71-host15 sudo[20331]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> rfp-6149559f-f >> Dec 13 13:33:56 e71-host15 sudo[20336]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:56 e71-host15 sudo[20342]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:56 e71-host15 sudo[20345]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip addr show qr-24f3070a-d4 >> permanent >> Dec 13 13:33:56 e71-host15 sudo[20348]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route list dev >> qr-24f3070a-d4 >> scope link >> Dec 13 13:33:56 e71-host15 sudo[20354]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -6 route list dev >> qr-24f3070a-d4 >> scope link >> Dec 13 13:33:56 e71-host15 sudo[20357]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 arping -A -I qr-24f3070a-d4 >> -c 3 -w >> 4.5 10.96.0.1 >> Dec 13 13:33:57 e71-host15 sudo[20368]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:57 e71-host15 sudo[20372]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.100 >> lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4 >> Dec 13 13:33:57 e71-host15 sudo[20375]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:57 e71-host15 sudo[20378]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.101 >> lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20384]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20387]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.102 >> lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20390]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20393]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.103 >> lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20399]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20402]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.104 >> lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4 >> Dec 13 13:33:58 e71-host15 sudo[20405]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:33:59 e71-host15 sudo[20411]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.105 >> lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4 >> ... >> ... >> ... >> # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-root >> wrap >> /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ; >> PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap >> /etc/neutron/rootwrap.conf,,g' | tail -25 >> >> Dec 13 13:34:51 e71-host15 sudo[21771]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.1.83 >> lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4 >> Dec 13 13:34:51 e71-host15 sudo[21777]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default >> via >> 10.96.0.2 dev qr-24f3070a-d4 table 174063617 >> Dec 13 13:34:51 e71-host15 sudo[21780]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show >> Dec 13 13:34:52 e71-host15 sudo[21783]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w >> net.ipv4.conf.qr-24f3070a-d4.send_redirects=0 >> Dec 13 13:34:52 e71-host15 sudo[21786]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net >> -maxdepth 1 >> -type l -printf %f >> Dec 13 13:34:52 e71-host15 sudo[21789]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w >> net.ipv4.conf.all.send_redirects=0 >> Dec 13 13:34:52 e71-host15 sudo[21792]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default >> via >> 10.96.0.2 dev qr-24f3070a-d4 table 174063617 >> Dec 13 13:34:52 e71-host15 sudo[21795]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show >> Dec 13 13:34:52 e71-host15 sudo[21801]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w >> net.ipv4.conf.qr-24f3070a-d4.send_redirects=0 >> Dec 13 13:34:53 e71-host15 sudo[21804]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show >> qr-24f3070a-d4 >> Dec 13 13:34:53 e71-host15 sudo[21807]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace >> 10.96.0.2 >> lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4 >> Dec 13 13:34:53 e71-host15 sudo[21817]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net >> -maxdepth 1 >> -type l -printf %f >> Dec 13 13:34:53 e71-host15 sudo[21822]: ip netns exec >> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f >> Dec 13 13:34:53 e71-host15 sudo[21827]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save >> Dec 13 13:34:53 e71-host15 sudo[21833]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n >> Dec 13 13:34:54 e71-host15 sudo[21836]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save >> Dec 13 13:34:54 e71-host15 sudo[21839]: ip netns exec >> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f >> Dec 13 13:34:54 e71-host15 sudo[21844]: ip netns exec >> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f >> Dec 13 13:34:54 e71-host15 sudo[21849]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save >> Dec 13 13:34:54 e71-host15 sudo[21852]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n >> Dec 13 13:34:54 e71-host15 sudo[21855]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save >> Dec 13 13:34:54 e71-host15 sudo[21861]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save >> Dec 13 13:34:55 e71-host15 sudo[21864]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n >> Dec 13 13:34:55 e71-host15 sudo[21869]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save >> Dec 13 13:34:55 e71-host15 sudo[21877]: ip netns exec >> qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy >> --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c >> -bf37-7d1827181228.pid >> --metadata_proxy_socket=/var/lib/neutron/metadata_proxy >> --router_id=6149559f-fa54-493c-bf37-7d1827181228 >> --state_path=/var/lib/neutron >> --metadata_port=9697 --metadata_proxy_user=118 --metadata_proxy_group=120 >> --debug --verbose --use-syslog --syslog-log-facility=LOG_LOCAL4 >> >> >> >> _______________________________________________ >> Mailing list: http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi >> -bin/mailman/listinfo/openstack >> >> >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
