Hi David,
Thanks for the reply.
I just did some more troubleshooting and I can definitely see using
dhcpdump on the VR that it only receives a DHCPREQUEST when the Guest VM is
on the same KVM Host. If I migrate the guest to any other KVM Host that
request never even makes it to the VR. I monitored the physical KVM host
running the VR, but it never registered the DHCPREQUEST regardless of where
the guest VM was running.
I even turned off the firewall on both KVM Hosts and still no DHCPREQUEST.
There actually is another physical DHCP server on the same network. *10.97.38.1
Router & DHCP*
However we (corporate IT Admin) have blocked off a range of IPs
{10.97.38.[110 -170]} from the 10.97.38.1 Router & DHCP and filtered out
any DHCPREQUESTS made by a '06' MAC address per these instructions (written
by you I believe):
http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html
So that tells me that we probably did something wrong on the router and
it's got to be the 10.97.38.1 Router/Switch/DHCP server that's stopping
those requests from ever reaching the built in VR, but I have no visibility
into that unfortunately so I cannot continue to test. I need to be able to
give very precise instructions to my IT Admin on what to check.
Thoughts?
*dhcpdump from VR when Guest VM is on same KVM Host:*
root@r-21-VM:~# dhcpdump -i eth0
TIME: 2013-11-18 08:24:46.997
IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
OP: 1 (BOOTPREQUEST)
HTYPE: 1 (Ethernet)
HLEN: 6
HOPS: 0
XID: 060d4c79
SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST)
OPTION: 50 ( 4) Request IP address 10.97.38.116
OPTION: 12 ( 16) Host name centos6-template
OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask)
28 (Broadcast address)
2 (Time offset)
121 (Classless Static Route)
15 (Domainname)
6 (DNS server)
12 (Host name)
40 (NIS domain)
41 (NIS servers)
42 (NTP servers)
26 (Interface MTU)
119 (Domain Search)
3 (Routers)
121 (Classless Static Route)
249 (MSFT - Classless route)
42 (NTP servers)
---------------------------------------------------------------------------
TIME: 2013-11-18 08:24:53.998
IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
OP: 1 (BOOTPREQUEST)
HTYPE: 1 (Ethernet)
HLEN: 6
HOPS: 0
XID: 060d4c79
SECS: 7
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST)
OPTION: 50 ( 4) Request IP address 10.97.38.116
OPTION: 12 ( 16) Host name centos6-template
OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask)
28 (Broadcast address)
2 (Time offset)
121 (Classless Static Route)
15 (Domainname)
6 (DNS server)
12 (Host name)
40 (NIS domain)
41 (NIS servers)
42 (NTP servers)
26 (Interface MTU)
119 (Domain Search)
3 (Routers)
121 (Classless Static Route)
249 (MSFT - Classless route)
42 (NTP servers)
---------------------------------------------------------------------------
TIME: 2013-11-18 08:25:02.003
IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
OP: 1 (BOOTPREQUEST)
HTYPE: 1 (Ethernet)
HLEN: 6
HOPS: 0
XID: 91cdcd74
SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION: 53 ( 1) DHCP message type 1 (DHCPDISCOVER)
OPTION: 50 ( 4) Request IP address 10.97.38.116
OPTION: 12 ( 16) Host name centos6-template
OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask)
28 (Broadcast address)
2 (Time offset)
121 (Classless Static Route)
15 (Domainname)
6 (DNS server)
12 (Host name)
40 (NIS domain)
41 (NIS servers)
42 (NTP servers)
26 (Interface MTU)
119 (Domain Search)
3 (Routers)
121 (Classless Static Route)
249 (MSFT - Classless route)
42 (NTP servers)
---------------------------------------------------------------------------
TIME: 2013-11-18 08:25:02.014
IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
OP: 2 (BOOTPREPLY)
HTYPE: 1 (Ethernet)
HLEN: 6
HOPS: 0
XID: 91cdcd74
SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.97.38.116
SIADDR: 10.97.38.113
GIADDR: 0.0.0.0
CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION: 53 ( 1) DHCP message type 2 (DHCPOFFER)
OPTION: 54 ( 4) Server identifier 10.97.38.113
OPTION: 51 ( 4) IP address leasetime -1 ()
OPTION: 1 ( 4) Subnet mask 255.255.255.0
OPTION: 28 ( 4) Broadcast address 10.97.38.255
OPTION: 12 ( 16) Host name centos6-template
OPTION: 6 ( 8) DNS server 10.97.38.113,10.97.32.20
OPTION: 3 ( 4) Routers 10.97.38.1
OPTION: 15 ( 17) Domainname cs1cloud.internal
---------------------------------------------------------------------------
TIME: 2013-11-18 08:25:02.015
IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
OP: 1 (BOOTPREQUEST)
HTYPE: 1 (Ethernet)
HLEN: 6
HOPS: 0
XID: 91cdcd74
SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST)
OPTION: 54 ( 4) Server identifier 10.97.38.113
OPTION: 50 ( 4) Request IP address 10.97.38.116
OPTION: 12 ( 16) Host name centos6-template
OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask)
28 (Broadcast address)
2 (Time offset)
121 (Classless Static Route)
15 (Domainname)
6 (DNS server)
12 (Host name)
40 (NIS domain)
41 (NIS servers)
42 (NTP servers)
26 (Interface MTU)
119 (Domain Search)
3 (Routers)
121 (Classless Static Route)
249 (MSFT - Classless route)
42 (NTP servers)
---------------------------------------------------------------------------
TIME: 2013-11-18 08:25:02.016
IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c)
OP: 2 (BOOTPREPLY)
HTYPE: 1 (Ethernet)
HLEN: 6
HOPS: 0
XID: 91cdcd74
SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.97.38.116
SIADDR: 10.97.38.113
GIADDR: 0.0.0.0
CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION: 53 ( 1) DHCP message type 5 (DHCPACK)
OPTION: 54 ( 4) Server identifier 10.97.38.113
OPTION: 51 ( 4) IP address leasetime -1 ()
OPTION: 1 ( 4) Subnet mask 255.255.255.0
OPTION: 28 ( 4) Broadcast address 10.97.38.255
OPTION: 12 ( 16) Host name centos6-template
OPTION: 6 ( 8) DNS server 10.97.38.113,10.97.32.20
OPTION: 3 ( 4) Routers 10.97.38.1
OPTION: 15 ( 17) Domainname cs1cloud.internal
---------------------------------------------------------------------------
root@r-21-VM:~#
Best Regards,
Adam Scarcella
On Sun, Nov 17, 2013 at 2:30 PM, David Nalley <[email protected]> wrote:
> So having seen this specific problem scores of times, it's almost
> always network config related. Several potential problems to look out
> for:
>
> 1. Another DHCP server on the same network. (There are ways of
> mitigating this, but for the time being, lets just say that
> CloudStack's VR should be the only DHCP server on the network.
>
> 2. Shell into the physical machine that the VR is running on. Using
> tcpdump (with filters of course) Do you see the broadcast asking for a
> DHCP address coming from the VM on the other physical machine? Do you
> see the VR answering the request? If so, move to step 3. If not (for
> either of those questions) - you aren't communicating between the two
> physical hosts - your switch config is suspect.
>
> 3. Shell into the physical machine that the VM is running on - You've
> seen the response with an IP address assigned go out - so see if it is
> seen by the physical host NIC? If not, you've again. If you see the
> response (and are sure it's from the VR and not another DHCP server)
> come back and tell us - something funky is going on.
>
> --David
>
> On Sun, Nov 17, 2013 at 10:11 AM, Adam <[email protected]> wrote:
> > Yes, I've disabled the firewall on both KVM hosts in question and still
> no
> > dice. I can't even ping the VR from my guest VM, but when I set the eth0
> > device on the guest to static everything works fine, which makes no sense
> > at all to me. Simply setting the NIC to static allows me see the VR.
> > Switching back to DHCP kills it again. I do not understand what is
> required
> > for the guests to acquire a DHCP lease from the VR. I know that the VR is
> > running dnsmasq, and I've tailed the /var/log/dnsmasq.log for more info,
> > but only see a DHCP request when the guest is on the same KVM host as the
> > VR.
> >
> > Does anyone know exactly how to troubleshoot this scenario? I'm not even
> > sure what to look for.
> >
> > -Adam
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Sat, Nov 16, 2013 at 5:43 PM, Carlos Reátegui <[email protected]
> >wrote:
> >
> >> Did you also check the host firewall? Try disabling.
> >>
> >> > On Nov 16, 2013, at 1:51 PM, Adam <[email protected]> wrote:
> >> >
> >> > The hosts have static IPs (10.97.38.[10-14]) and can all see and talk
> to
> >> > each other via IP and hostname. I'm only using a basic zone so no VLAN
> >> > tagging or anything funky like that.
> >> >
> >> > Best Regards,
> >> >
> >> >
> >> >
> >> > Adam Scarcella
> >> >
> >> >
> >> > On Sat, Nov 16, 2013 at 4:17 PM, Andrei Mikhailovsky <
> [email protected]
> >> >wrote:
> >> >
> >> >>
> >> >>
> >> >> Adam, it sounds like a networking issue, check that all hosts can
> talk
> >> to
> >> >> each other on the same vlan that is used for guest network. I had the
> >> same
> >> >> issue with advanced networking when my vlans were not properly setup
> on
> >> the
> >> >> switches.
> >> >>
> >> >> Andrei
> >> >>
> >> >> ----- Original Message -----
> >> >>
> >> >> From: "Adam" <[email protected]>
> >> >> To: [email protected]
> >> >> Sent: Saturday, 16 November, 2013 7:08:50 PM
> >> >> Subject: Guest VMs not able to acquire DHCP IP from Virtual Router
> >> unless
> >> >> Guest and VR are on the same KVM host
> >> >>
> >> >> Hi All,
> >> >>
> >> >> I have a new and very strange issue that for the life of me I cannot
> >> seem
> >> >> to track down and fix.
> >> >>
> >> >> I have CS 4.2 running on 5 hosts in a simple basic zone. All is
> working
> >> >> fine, except that my Guest VMs cannot seem to get a DHCP lease from
> the
> >> >> Virtual Router unless I migrate the Guest VM to the same physical KVM
> >> >> Host running the VR. That would seem to indicate a firewall issue,
> but
> >> I've
> >> >> tested by turning off both the firewalls (VR KVM Host iptables &
> Guest
> >> VM
> >> >> KVM Host iptables). It didn't help. The only way to fix it is to
> migrate
> >> >> the Guest VM to the same KVM Host that's running the Virtual Router.
> >> >>
> >> >> NOTE: The Console Proxy has worked flawlessly this whole time.
> >> >>
> >> >> So, if a Guest VM starts on a different physical KVM Host, it will
> not
> >> get
> >> >> an internal IP of 169.254.x.x. All along the Console Proxy works
> fine.
> >> Then
> >> >> if I migrate the Guest VM to the same KVM host that's running the VR,
> >> DHCP
> >> >> automatically starts working and the Guest VM receives a proper IP
> >> address
> >> >> of 10.97.38.x. Then I can migrate the Guest VM back to any other
> >> physical
> >> >> KVM Host and believe it or not, it continues to work flawlessly,
> until I
> >> >> either reboot the VM or restart the network services. Then it cannot
> see
> >> >> the VR again and instead receives an internal IP of 169.254.x.x. If I
> >> set a
> >> >> static IP address & DNS everything works fine, no matter where the
> >> Guest VM
> >> >> is running.
> >> >>
> >> >> Setting static IPs is not an option
> >> >> Running all the Guest VMs on the same physical KVM host is not an
> option
> >> >>
> >> >> I desperately need to track down the root cause of this issue so I
> can
> >> >> release this cloud to my entire department by Monday morning. Someone
> >> >> please help!
> >> >>
> >> >> Best Regards,
> >> >>
> >> >>
> >> >>
> >> >> Adam Scarcella
> >> >>
> >> >>
> >>
>
