Hello
Cloudstack VM Issue, no outgoing access
There is an issue in cloudstack where a VM cannot gain outgoing access to
the internet. This is a
blocking issue since Cloudstack users cannot gain Internet access to do
patch updates, download
software, etc.
Environment:
Cloudstack version 4.3
System VM from January 2014
CentOS6.5 for all servers
4 servers (management server, 2 compute nodes, and a server that serves both
primary and secondary
storage)
Advanced networking zone with 1 zone, 1 pod, 1 cluster, 2 hosts
2 NICs on the compute node hosts
- 1 NIC hosts the management and storage
- 1 NIC hosts the public and guest networks
Primary and Secondard storage use NFS
Management network is 172.16.10.x/24
Guest networks are 10.0.0.x/24
Public network uses a /24 public range
Here is how to reproduce the prove the issue.
1. Create a new VM using a new isolated network
2. Create the egress rules to allow 10.0.0.0/24 out for everything
3. Create a port map rule for SSH to the new VM and also allow external
acess to SSH
4. Login to the console on the new VM
5. Ping google.com, see 100% packet loss.
6. Login to the console on the virtual router for this network and type this
command:
Ping results
[root@NewTestVM ~]# ping google.com
PING google.com (74.125.225.128) 56(84) bytes of data.
^C
--- google.com ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time
6127ms
[root@NewTestVM ~]#
All the iptables rules look as they should, especially look at the SNAT rule
in the nat table for
POSTROUTING. But, still can't ping. How do we know there isn't some other
problem in the
network. Well, first we were able to SSH to the new VM externally from the
Internet to do the ping.
Second, we found that if we add a second identical SNAT rule, now everything
works.
Okay so now add the second SNAT rule like this;
iptables -t nat -A POSTROUTING -j SNAT –t-source 68.70.175.6
Yes, pinging google.com still does not work, losing 100% of packets..
Now, Pinging from the new VM works like a charm. What is going on??
[root@NewTestVM ~]# ping google.com
PING google.com (74.125.225.132) 56(84) bytes of data.
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=1 ttl=52
time=2.12 ms
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=2 ttl=52
time=2.09 ms
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=3 ttl=52
time=2.22 ms
64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=4 ttl=52
time=2.24 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3556ms
rtt min/avg/max/mdev = 2.095/2.172/2.249/0.064 ms
[root@NewTestVM ~]#
The first SNAT rule should have worked but it doesn't. WE've proven, I
think, that everything else is
setup correctly by being able to access the VM externally via SSH and by
simply adding the SNAT rule
to the virtual router. Why doesn't the first rule work.
It seems cloudstack is doing exactly what it is supposed to do, adding the
SNAT rule. So, it seems the
actual problem is with either iptables or with the debian kernel for the
system VM. I think we are using
the most recent system VM.
Note, neither restarting the new network nor cleaning it makes any
difference. It just seems having the
one SNAT rule will not work. I've done all the tcpdump debugging, that is
how I found this issue to
begin with. Tcpdump shows that the ping/icmp packets go through the virtual
router unaltered using
the private IP address of the VM (10.0.0.x). After the second SNAT rule is
added tcpdump shows then
that the ping/icmp packets do get translated.
I am totally floored by these results. How can iptables or the debian kernel
not recognize the first
SNAT rule and then recognize the second one??
Okay, so before someone tells me that this is not a cloudstack issue and to
find a resolution else,
cloudstack depends on this technique for isolated networks. If guest users
can't make their Vms work
(note, they don't have access to the virtual routers like I do), isolated
networks are broken and
cloudstack is not usable.
I have screen shots i can send if it will help.
