Hi Jeff, Thanks for looking into this but the masquerade still not working. I have more information and hope you will be able to help.
I have a single bare metal with everything installed ( Nova-compute, network node, controller, etc... ) There four NIC on that box NIC em1 connect to external network with IP 10.38.5.251 NIC em3 connect to internal network with no IP configured em2 and em4 are disabled After everything is configured ( adding router, net, sub-net ,etc.. ) and running, I run ifconfig and found out em1's has no more ip but a bridge has created brq7f248f20-a6 Link encap:Ethernet HWaddr 00:21:9B:95:99:7A inet addr:10.38.15.251 Bcast:10.38.255.255 Mask:255.255.0.0 em1 Link encap:Ethernet HWaddr 00:21:9B:95:99:7A UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 I think this is how the quantum/linuxbridge work. I also create a floatingIP range ( 10.38.17.1-254 ). Then I saw a virtual NIC is created with IP 10.38.17.1 which I believe is the router IP for the NAT qg-0503ddc6-1d Link encap:Ethernet HWaddr 8E:57:D6:DA:2B:AA inet addr:10.38.17.1 Bcast:10.38.17.255 Mask:255.255.255.0 Now I run tcpdump on the openstack box ( ie 10.38.5.251 ) and the target machine ( 10.38.1.2 ). Then ping 10.38.1.2 from my VM ( 192.168.151.4 ). I saw the packet did arrive to 10.38.1.2 but with ip address 192.168.151.4. I supposed to see 10.38.17.1 right? 20:52:43.492160 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq 5, length 64 20:52:43.492170 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 5, length 64 20:52:44.492597 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq 6, length 64 20:52:44.492608 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 6, length 64 20:52:45.492894 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq 7, length 64 20:52:45.492906 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 7, length 64 20:52:46.493183 IP 192.168.151.4 > 10.38.1.2: ICMP echo request, id 17665, seq 8, length 64 20:52:46.493193 IP 10.38.1.2 > 192.168.151.4: ICMP echo reply, id 17665, seq 8, length 64 I also think it is the IP masquerade rule, but it didn't work. I tried all three interface ( em1, brq7f248f20-a6 and qg-0503ddc6-1d ) but none of them work. For some reason SNAT didn't seem to happen.. Here is the iptables status service iptables status Table: nat Chain PREROUTING (policy ACCEPT) num target prot opt source destination 1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 2 quantum-l3-agent-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 Chain POSTROUTING (policy ACCEPT) num target prot opt source destination 1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 2 quantum-l3-agent-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 3 quantum-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0 4 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 5 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0 6 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 7 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 8 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) num target prot opt source destination 1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 2 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-api-OUTPUT (1 references) num target prot opt source destination Chain nova-api-POSTROUTING (1 references) num target prot opt source destination Chain nova-api-PREROUTING (1 references) num target prot opt source destination Chain nova-api-float-snat (1 references) num target prot opt source destination Chain nova-api-snat (1 references) num target prot opt source destination 1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-compute-OUTPUT (1 references) num target prot opt source destination Chain nova-compute-POSTROUTING (1 references) num target prot opt source destination Chain nova-compute-PREROUTING (1 references) num target prot opt source destination Chain nova-compute-float-snat (1 references) num target prot opt source destination Chain nova-compute-snat (1 references) num target prot opt source destination 1 nova-compute-float-snat all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-postrouting-bottom (1 references) num target prot opt source destination 1 nova-compute-snat all -- 0.0.0.0/0 0.0.0.0/0 2 nova-api-snat all -- 0.0.0.0/0 0.0.0.0/0 Chain quantum-l3-agent-OUTPUT (1 references) num target prot opt source destination Chain quantum-l3-agent-POSTROUTING (1 references) num target prot opt source destination 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! ctstate DNAT Chain quantum-l3-agent-PREROUTING (1 references) num target prot opt source destination Chain quantum-l3-agent-float-snat (1 references) num target prot opt source destination Chain quantum-l3-agent-snat (1 references) num target prot opt source destination 1 quantum-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0 2 SNAT all -- 192.168.151.0/24 0.0.0.0/0 to:10.38.17.1 Chain quantum-postrouting-bottom (1 references) num target prot opt source destination 1 quantum-l3-agent-snat all -- 0.0.0.0/0 0.0.0.0/0 Table: filter Chain INPUT (policy ACCEPT) num target prot opt source destination 1 nova-compute-INPUT all -- 0.0.0.0/0 0.0.0.0/0 2 quantum-l3-agent-INPUT all -- 0.0.0.0/0 0.0.0.0/0 3 nova-api-INPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT) num target prot opt source destination 1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0 2 nova-compute-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0 4 quantum-l3-agent-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 5 nova-api-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) num target prot opt source destination 1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0 2 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0 4 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 5 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-api-FORWARD (1 references) num target prot opt source destination Chain nova-api-INPUT (1 references) num target prot opt source destination 1 ACCEPT tcp -- 0.0.0.0/0 10.38.15.251 tcp dpt:8775 Chain nova-api-OUTPUT (1 references) num target prot opt source destination Chain nova-api-local (1 references) num target prot opt source destination Chain nova-compute-FORWARD (1 references) num target prot opt source destination Chain nova-compute-INPUT (1 references) num target prot opt source destination Chain nova-compute-OUTPUT (1 references) num target prot opt source destination Chain nova-compute-inst-24 (1 references) num target prot opt source destination 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp spt:67 dpt:68 5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0 6 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-compute-inst-25 (1 references) num target prot opt source destination 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp spt:67 dpt:68 5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0 6 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 7 ACCEPT icmp -- 192.168.151.4 0.0.0.0/0 8 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 9 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-compute-local (1 references) num target prot opt source destination 1 nova-compute-inst-24 all -- 0.0.0.0/0 192.168.151.3 2 nova-compute-inst-25 all -- 0.0.0.0/0 192.168.151.4 Chain nova-compute-provider (2 references) num target prot opt source destination Chain nova-compute-sg-fallback (2 references) num target prot opt source destination 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 Chain nova-filter-top (2 references) num target prot opt source destination 1 nova-compute-local all -- 0.0.0.0/0 0.0.0.0/0 2 nova-api-local all -- 0.0.0.0/0 0.0.0.0/0 Chain quantum-filter-top (2 references) num target prot opt source destination 1 quantum-l3-agent-local all -- 0.0.0.0/0 0.0.0.0/0 Chain quantum-l3-agent-FORWARD (1 references) num target prot opt source destination Chain quantum-l3-agent-INPUT (1 references) num target prot opt source destination Chain quantum-l3-agent-OUTPUT (1 references) num target prot opt source destination Chain quantum-l3-agent-local (1 references) num target prot opt source destination ----- Original Message ---- From: Jeff Peeler <jpee...@redhat.com> To: Barrow Kwan <barrowk...@yahoo.com> Sent: Wed, March 6, 2013 10:48:00 AM Subject: Re: [Openstack] VM guest can't access outside world. I'm not replying to the list because I don't know for sure what to tell you. If this does work for you, feel free to CC the list. What I mean is by whichever interface uses 10.38.1.2, add a rule to iptables like: iptables --table nat -A POSTROUTING -o <above interface> -j MASQUERADE You're right that this may be a bug in quantum. I haven't really dug too deeply to confirm that or not yet. If the above works for you, you may also have to explicitly set the dns nameserver for the subnet in quantum to something that makes sense. Jeff On Mon, Mar 04, 2013 at 11:37:41AM -0800, Barrow Kwan wrote: > Hi, > Thanks Jeff. this is what I got from tcpdump. The target (10.38.1.2 ) > didn't > seem to reply. might be the target ( 10.38.1.2 ) didn't know how to route > the > > packet to 192.168.151.3? could that be SNAT issue? or like you said it needs >IP > > masquerading rule. might be a bug in Quantum? > > > Barrow > > > > > tcpdump: WARNING: em1: no IPv4 address assigned > tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 65535 >bytes > 11:31:02.825150 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP >(1), > > > length 84) > 192.168.151.3 > 10.38.1.2: ICMP echo request, id 11910, seq 133, length 64 > 11:31:03.825338 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP >(1), > > > length 84) > 192.168.151.3 > 10.38.1.2: ICMP echo request, id 11910, seq 134, length 64 > 2 packets captured > 3 packets received by filter > 0 packets dropped by kernel > > > > > ----- Original Message ---- > From: Jeff Peeler <jpee...@redhat.com> > To: openstack@lists.launchpad.net > Sent: Mon, March 4, 2013 7:39:03 AM > Subject: Re: [Openstack] VM guest can't access outside world. > > On Wed, Feb 27, 2013 at 12:38:45PM -0800, Barrow Kwan wrote: > > [root@optst01 quantum]# service iptables status > > Table: nat > > Chain PREROUTING (policy ACCEPT) > > num target prot opt source destination > > 1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > 2 quantum-l3-agent-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > > 3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain POSTROUTING (policy ACCEPT) > > num target prot opt source destination > > 1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > 2 quantum-l3-agent-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > > > > 3 quantum-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > > > > 4 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 > > 5 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > > Chain OUTPUT (policy ACCEPT) > > num target prot opt source destination > > 1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 > > 2 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > 3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain nova-api-OUTPUT (1 references) > > num target prot opt source destination > > > > Chain nova-api-POSTROUTING (1 references) > > num target prot opt source destination > > > > Chain nova-api-PREROUTING (1 references) > > num target prot opt source destination > > > > Chain nova-api-float-snat (1 references) > > num target prot opt source destination > > > > Chain nova-api-snat (1 references) > > num target prot opt source destination > > 1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain nova-compute-OUTPUT (1 references) > > num target prot opt source destination > > > > Chain nova-compute-POSTROUTING (1 references) > > num target prot opt source destination > > > > Chain nova-compute-PREROUTING (1 references) > > num target prot opt source destination > > > > Chain nova-compute-float-snat (1 references) > > num target prot opt source destination > > > > Chain nova-compute-snat (1 references) > > num target prot opt source destination > > 1 nova-compute-float-snat all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > > Chain nova-postrouting-bottom (1 references) > > num target prot opt source destination > > 1 nova-compute-snat all -- 0.0.0.0/0 0.0.0.0/0 > > 2 nova-api-snat all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain quantum-l3-agent-OUTPUT (1 references) > > num target prot opt source destination > > > > Chain quantum-l3-agent-POSTROUTING (1 references) > > num target prot opt source destination > > 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! ctstate > > DNAT > > > > Chain quantum-l3-agent-PREROUTING (1 references) > > num target prot opt source destination > > > > Chain quantum-l3-agent-float-snat (1 references) > > num target prot opt source destination > > > > Chain quantum-l3-agent-snat (1 references) > > num target prot opt source destination > > 1 quantum-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > > 2 SNAT all -- 192.168.151.0/24 0.0.0.0/0 >to:10.38.17.1 > > > > > > > > > Chain quantum-postrouting-bottom (1 references) > > num target prot opt source destination > > 1 quantum-l3-agent-snat all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > Table: filter > > Chain INPUT (policy ACCEPT) > > num target prot opt source destination > > 1 nova-compute-INPUT all -- 0.0.0.0/0 0.0.0.0/0 > > 2 quantum-l3-agent-INPUT all -- 0.0.0.0/0 0.0.0.0/0 > > > > > 3 nova-api-INPUT all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain FORWARD (policy ACCEPT) > > num target prot opt source destination > > 1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0 > > 2 nova-compute-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 > > 3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0 > > 4 quantum-l3-agent-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > 5 nova-api-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain OUTPUT (policy ACCEPT) > > num target prot opt source destination > > 1 nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0 > > 2 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 > > 3 quantum-filter-top all -- 0.0.0.0/0 0.0.0.0/0 > > 4 quantum-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > 5 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain nova-api-FORWARD (1 references) > > num target prot opt source destination > > > > Chain nova-api-INPUT (1 references) > > num target prot opt source destination > > 1 ACCEPT tcp -- 0.0.0.0/0 10.38.15.251 tcp >dpt:8775 > > > > > > > Chain nova-api-OUTPUT (1 references) > > num target prot opt source destination > > > > Chain nova-api-local (1 references) > > num target prot opt source destination > > > > Chain nova-compute-FORWARD (1 references) > > num target prot opt source destination > > > > Chain nova-compute-INPUT (1 references) > > num target prot opt source destination > > > > Chain nova-compute-OUTPUT (1 references) > > num target prot opt source destination > > > > Chain nova-compute-inst-20 (1 references) > > num target prot opt source destination > > 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state >INVALID > > > > > > > 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state > > RELATED,ESTABLISHED > > 3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0 > > > > 4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp > > spt:67 > > dpt:68 > > 5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0 > > 6 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 > > 7 ACCEPT icmp -- 192.168.151.3 0.0.0.0/0 > > 8 ACCEPT icmp -- 192.168.151.4 0.0.0.0/0 > > 9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > > dpt:22 > > 10 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > Chain nova-compute-inst-21 (1 references) > > num target prot opt source destination > > 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 state >INVALID > > > > > > > 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state > > RELATED,ESTABLISHED > > 3 nova-compute-provider all -- 0.0.0.0/0 0.0.0.0/0 > > > > 4 ACCEPT udp -- 192.168.151.2 0.0.0.0/0 udp > > spt:67 > > dpt:68 > > 5 ACCEPT all -- 192.168.151.0/24 0.0.0.0/0 > > 6 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 > > 7 ACCEPT icmp -- 192.168.151.3 0.0.0.0/0 > > 8 ACCEPT icmp -- 192.168.151.4 0.0.0.0/0 > > 9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > > dpt:22 > > 10 nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > > > Chain nova-compute-local (1 references) > > num target prot opt source destination > > 1 nova-compute-inst-20 all -- 0.0.0.0/0 192.168.151.3 > > 2 nova-compute-inst-21 all -- 0.0.0.0/0 192.168.151.4 > > > > Chain nova-compute-provider (2 references) > > num target prot opt source destination > > > > Chain nova-compute-sg-fallback (2 references) > > num target prot opt source destination > > 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain nova-filter-top (2 references) > > num target prot opt source destination > > 1 nova-compute-local all -- 0.0.0.0/0 0.0.0.0/0 > > 2 nova-api-local all -- 0.0.0.0/0 0.0.0.0/0 > > > > Chain quantum-filter-top (2 references) > > num target prot opt source destination > > 1 quantum-l3-agent-local all -- 0.0.0.0/0 0.0.0.0/0 > > > > > > > Chain quantum-l3-agent-FORWARD (1 references) > > num target prot opt source destination > > > > Chain quantum-l3-agent-INPUT (1 references) > > num target prot opt source destination > > > > Chain quantum-l3-agent-OUTPUT (1 references) > > num target prot opt source destination > > > > Chain quantum-l3-agent-local (1 references) > > num target prot opt source destination > > Have you tried running tcpdump on the public interface to see how far > the packets are getting? Maybe something like: tcpdump -n -c2 icmp -i em1, > then try pinging from the VM. It could be that you're attempting to send > unroutable packets, in which case an IP masquerading rule needs adding. > > Jeff > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp