I have instrumented ipconfig, and determined that the ultimate source of the
problem
is that, for the case of multiple interfaces, ipconfig has a dependency on the
kernel's probe order of the network interfaces.
For whatever reason, the -31 kernel probes the network devices in one
order (e.g., ens3 then ens4), and the -57 kernel in the other order
(ens4 first then ens3).
The probe order of network devices (and PCI devices in general) is
explicitly not defined, and so this is not a bug in the kernel itself;
ipconfig is failing due to its dependency on a specific enumeration
order.
The issue in ipconfig is that it is using a single packet socket to
attempt to multiplex packet traffic on multiple interfaces. Presuming
that ens3 will answer DHCP and ens4 will not, for the case that works,
the order ends up being something like:
send DHCP request on ens3
send DHCP request on ens4
[ system gets DHCP response via ens3 ]
try to receive DHCP reply sent by peer for ens3; this matches, and all is happy
For the case that it fails, the sequence is roughly:
send DHCP request on ens4
send DHCP request on ens3
[ system gets DHCP response via ens3 ]
try to receive DHCP reply sent by peer for ens4; the reply is actually for
ens3, so ipconfig
throws it away (as the XID, et al, don't match what is expected for the ens4
DHCP request).
This repeats until ipconfig gives up.
As I said above, the issue is that ipconfig is trying to multiplex
traffic for two interfaces on one packet socket. This is fine for
sending, but for receiving on an unbound packet socket, there is no way
to receive a packet sent to a specific interface. Packets are delivered
to recvfrom/recvmsg in the order received.
I note that ipconfig sets sll.sll_ifindex on the msghdr provided to
recvfrom and recvmsg system calls; perhaps the author believed that this
limits received packets to only packets received on that ifindex. This
is not the case, and the sll_ifindex passed to recvfrom/recvmsg is
ignored.
I'm looking into whether or not there is an simple fix for this that
will let ipconfig function without major rework to utilize one packet
socket per interface.
** Tags removed: kernel-key
** Package changed: linux (Ubuntu) => klibc (Ubuntu)
** Changed in: klibc (Ubuntu)
Status: Triaged => Confirmed
** Changed in: klibc (Ubuntu)
Assignee: (unassigned) => Jay Vosburgh (jvosburgh)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1652348
Title:
initrd dhcp fails / ignores valid response
Status in klibc package in Ubuntu:
Confirmed
Status in klibc source package in Xenial:
Triaged
Bug description:
Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been
(re?)introduced that is breaking dhcp booting in the initrd
environment. This is stopping instances that use iscsi storage from
being able to connect.
Over serial console it outputs:
IP-Config: no response after 2 secs - giving up
IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
IP-Config: no response after 3 secs - giving up
with increasing delays until it fails. At which point a simple
ipconfig -t dhcp -d "ens2f0" works. The console output is slightly
garbled but should give you an idea:
(initramfs) ipconfig -t dhcp -[ 728.379793] ixgbe 0000:13:00.0 ens2f0:
changing MTU from 1500 to 9000
d "ens2f0"
IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
IP-Config: ens2f0 guessed broadcast address 10.0.1.255
IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
addres[ 728.980448] ixgbe 0000:13:00.0 ens2f0: detected SFP+: 3
s: 10.0.1.56 broadcast: 10.0.1.255 netmask: 255.255.255.0
gateway: 10.0.1.1 [ 729.148410] ixgbe 0000:13:00.0 ens2f0: NIC Link is Up
10 Gbps, Flow Control: RX/TX
dns0 : 169.254.169.254 dns1 : 0.0.0.0
rootserver: 169.254.169.254 rootpath:
filename : /ipxe.efi
tcpdumps show that dhcp requests are being received from the host, and
responses sent, but not accepted by the host. When the ipconfig
command is issued manually, an identical dhcp request and response
happens, only this time it is accepted. It doesn't appear to be that
the messages are being sent and received incorrectly, just silently
ignored by ipconfig.
I was seeing this behaviour earlier this year, which I was able to fix
by specifying "ip=dhcp" as a kernel parameter. About a month ago that
was identified as causing us other problems (long story) and we
dropped it, at which point we discovered the original bug was no
longer an issue.
Putting "ip=dhcp" back on with this kernel no longer fixes the
problem.
I've compared the two initrds and effectively the only thing that has
changed between the two is the kernel components.
Ubuntu kernel bisect offending commit:
# first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per
mount namespace limit on the number of mounts
Ubuntu kernel bisect offending commit submission:
https://lkml.org/lkml/2016/10/5/308
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
