Jayapal, I did a ping test and traced as you suggested. tcpdump monitoring was done on the public facing interface of the VR. >From within the VR, ping to public IP functions correctly, source address is >the public IP assigned to the VR. >From within the guest, ping to same public IP, does not function, source >address is (as you suspected) the IP of guest on the guest network of VR. Therefore, it must be: the SNAT rule in iptables in the VR is being bypassed... that is, the packets are being forwarded without SNAT being performed on them correctly. Noel
> From: [email protected] > To: [email protected] > Subject: Re: Advanced Network - SNAT not working > Date: Mon, 16 Sep 2013 05:14:53 +0000 > > Hi, > > I think when the packets are going out the packets are NATed with private ip, > that can't reach back to router. > From the VR when you ping public network observe with what source ip address > the packet is going out and > From the guest VM when you access public n/w observe on VR with what source > ip the packet is going out. > In later case I think the source ip address is different. > > Thanks, > Jayapal > > > On 16-Sep-2013, at 2:30 AM, Noel Kendall <[email protected]> wrote: > > > No other NAT. There is nothing but copper between the KVM host machine and > > the ISP router.There is an L2/L3 switch that the packets travel through. > > However, there is no forwarding in the switch,just straight through. I've > > had a well-functioning V4.0.1 environment running on this same > > configurationin the past. What is new is the conversion to 4.1 (which was a > > clean install). > > It's very mysterious, I have never seen anything like this before. There > > are two other VRs, both having same issue. > > I will try your suggestion. > > Noel > >> Date: Sun, 15 Sep 2013 21:20:41 +0100 > >> Subject: Re: Advanced Network - SNAT not working > >> From: [email protected] > >> To: [email protected] > >> > >> This is mostly confusing that the packets are not seen on the VR public > >> interface, seeing as other services are working. > >> If it was a local NAT issue then the packet would atleast get into that > >> interface. Do you have any upstream devices providing NAT? Or any other VR > >> with the issue? > >> > >> It may be worth recreating the VR, by stopping and destroying it and > >> creating another guest to start a fresh. > >> > >> Marty > >> > >> > >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall > >> <[email protected]>wrote: > >> > >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the guest, > >>> while running a tcpdumpon the public i/f of the VR: > >>> - I can see the outbound packets going out- I do not see a response packet > >>> coming back in > >>> FYI there are no firewalls outbound from the KVM host. The host bridges vi > >>> CS networkingdirectly out on to the internet via a switch. > >>> Note that traffic from outside (ssh, web) can happily traverse the VR to > >>> the guest. I get the usualits working html page from the guest. This tells > >>> me that there is nothing outbound from the VR thatis filtering packets. > >>> Am truly stumped. This is mysterious indeed. > >>> From within the VR, can happily telnet to <www.xyz.com> 80 and receive > >>> response.Only if packet came from guest and was forwarded does the > >>> response > >>> not show up. > >>> In short: > >>> wget from VR to www.xyz.com works, response received and saved > >>> wget from guest to www.xyz.com does not work, network not available > >>> displayed on guest, response packets not seen on the public i/f of VR at > >>> all > >>> Noel > >>> > >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100 > >>>> Subject: Re: Advanced Network - SNAT not working > >>>> From: [email protected] > >>>> To: [email protected] > >>>> > >>>> Hi Noel, > >>>> > >>>> Can you answer: Does the traffic come back on the public interface? and > >>>> then onto the Guest interface? > >>>> > >>>> Thanks, > >>>> Marty > >>>> > >>>> > >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall <[email protected] > >>>> wrote: > >>>> > >>>>> Indeed, yes, a wget executed on the VR to a public website works just > >>> fine. > >>>>> Noel > >>>>> > >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100 > >>>>>> Subject: Re: Advanced Network - SNAT not working > >>>>>> From: [email protected] > >>>>>> To: [email protected] > >>>>>> > >>>>>> Hi Noel, > >>>>>> > >>>>>> Does the traffic come back on the public interface? and then onto the > >>>>> Guest > >>>>>> interface? > >>>>>> > >>>>>> Does a wget on the VR work? > >>>>>> > >>>>>> Marty > >>>>>> > >>>>>> > >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall < > >>> [email protected] > >>>>>> wrote: > >>>>>> > >>>>>>> I have that Marty. I see the http outbound request coming in on the > >>>>> guest > >>>>>>> interface of the VR,and see the http request being sent out on the > >>>>> public > >>>>>>> interface of the VR. > >>>>>>> The traffic is flowing fine from guest to the outbound i/f of the > >>> VR. > >>>>>>> This is tcpdump on the public i/f while guest is doing wget to > >>>>>>> 6x.xxx.xxx.xxx > >>>>>>> > >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00, ethertype > >>> IPv4 > >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80: Flags > >>> [S], > >>>>> seq > >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348 ecr > >>>>>>> 0,nop,wscale 4], length 0 0x0000: 4500 003c ad1d 4000 3f06 2d13 > >>> 0a0b > >>>>> 4fb2 > >>>>>>> 0x0010: 416e c660 98a2 0050 6ed2 de56 0000 0000 > >>> 0x0020: > >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a 0x0030: 01a3 7444 > >>> 0000 > >>>>>>> 0000 0103 0304 > >>>>>>> > >>>>>>> > >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100 > >>>>>>>> Subject: Re: Advanced Network - SNAT not working > >>>>>>>> From: [email protected] > >>>>>>>> To: [email protected] > >>>>>>>> > >>>>>>>> Hi Noel, > >>>>>>>> > >>>>>>>> Can you run a tcpdump on both VR interfaces, this should make it > >>>>> apparent > >>>>>>>> what is happening? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Marty > >>>>>>>> > >>>>>>>> > >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall < > >>>>> [email protected] > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> http://pastebin.com/3FZmFnvZ > >>>>>>>>> Many thanks Marty. > >>>>>>>>> Noel > >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100 > >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working > >>>>>>>>>> From: [email protected] > >>>>>>>>>> To: [email protected] > >>>>>>>>>> > >>>>>>>>>> Hi Noel, > >>>>>>>>>> > >>>>>>>>>> Could you put the IP tables on pastebin? GMail has collapsed > >>> the > >>>>>>> lines > >>>>>>>>>> horrifically. > >>>>>>>>>> Have you also tried a tcpdump on both interfaces on the VR? > >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be called > >>>>>>>>>> > >>>>>>>>>> I would expect worse connectivity if it was a pure NAT issue, > >>>>> but I > >>>>>>> will > >>>>>>>>>> review the tables later. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Marty > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall < > >>>>>>> [email protected] > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Not seeing return packets on VR. Suspect, therefore, that > >>> SNAT > >>>>> is > >>>>>>>>> fouled > >>>>>>>>>>> up in some way.I have been doing wget to from guest, can > >>> see > >>>>> the > >>>>>>>>> outgoing > >>>>>>>>>>> request fine, both in the guest andthe VR. > >>>>>>>>>>> Could it be that the SNAT table entries from the > >>>>> 10.11.0.0/16subnet > >>>>>>>>> to > >>>>>>>>>>> dpt www are interfering withthe SNAT to public ip?? (wild > >>>>> guess) - > >>>>>>> not > >>>>>>>>> an > >>>>>>>>>>> iptables expert by any stretch of the imagination > >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178 is the > >>> guest > >>>>> IP on > >>>>>>>>> guest > >>>>>>>>>>> network > >>>>>>>>>>> iptables _L -t nat on the VR shows... > >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target prot opt source > >>>>>>>>>>> destination DNAT tcp -- anywhere > >>>>>>> anywhere > >>>>>>>>>>> tcp dpt:domain to:10.11.0.1 DNAT tcp -- > >>> anywhere > >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:www to:10.11.79.178:80 DNAT > >>>>>>> tcp -- > >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:www > >>>>>>>>> to:10.11.79.178:80DNAT tcp -- anywhere > >>>>>>> 67.xxx.xxx.56 > >>>>>>>>> tcp dpt:https > >>>>>>>>>>> to:10.11.79.178:443 DNAT tcp -- anywhere > >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT > >>>>>>> tcp > >>>>>>>>> -- > >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:ssh > >>>>>>>>> to:10.11.79.178:22DNAT tcp -- anywhere > >>>>>>> 67.xxx.xxx.56 > >>>>>>>>> tcp dpt:ssh > >>>>>>>>>>> to:10.11.79.178:22 DNAT tcp -- anywhere > >>>>>>>>> 67.xxx.xxx.56 > >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- > >>>>> anywhere > >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:ftp to:10.11.79.178:21DNAT > >>>>>>>>> tcp > >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp > >>> dpt:5901 to: > >>>>>>>>>>> 10.11.79.178:5901 DNAT tcp -- anywhere > >>>>>>>>> 67.xxx.xxx.56 > >>>>>>>>>>> tcp dpt:5901 to:10.11.79.178:5901 > >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target prot opt source > >>>>>>>>>>> destination SNAT all -- anywhere > >>>>>>> anywhere > >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere > >>>>>>>>> anywhere > >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere > >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- > >>>>> anywhere > >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- > >>>>> anywhere > >>>>>>>>>>> anywhere to:67.xxx.xxx.56SNAT all -- > >>>>>>> anywhere > >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all > >>> -- > >>>>>>> anywhere > >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT > >>> all -- > >>>>>>>>> anywhere > >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT > >>> tcp > >>>>> -- > >>>>>>>>>>> 10.11.0.0/16 myguest tcp dpt:www > >>>>> to:10.11.0.1 > >>>>>>> SNAT > >>>>>>>>>>> tcp -- 10.11.0.0/16 myguest tcp > >>>>>>> dpt:https > >>>>>>>>>>> to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 > >>> myguest > >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 > >>>>>>>>> myguest > >>>>>>>>>>> tcp dpt:ftp to:10.11.0.1 SNAT tcp -- > >>>>>>> 10.11.0.0/16 > >>>>>>>>>>> myguest tcp dpt:5901 to:10.11.0.1 SNAT > >>>>> all > >>>>>>> -- > >>>>>>>>>>> anywhere anywhere to:67.xxx.xxx.56 > >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target prot opt source > >>>>>>>>>>> destination DNAT tcp -- anywhere > >>>>>>>>> 67.xxx.xxx.56 > >>>>>>>>>>> tcp dpt:www to:10.11.79.178:80 DNAT tcp -- > >>>>> anywhere > >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT > >>>>>>>>> tcp > >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp dpt:ssh > >>> to: > >>>>>>>>>>> 10.11.79.178:22 DNAT tcp -- anywhere > >>>>>>> 67.xxx.xxx.56 > >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- > >>>>> anywhere > >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:5901 to:10.11.79.178:5901 > >>>>>>>>>>> > >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100 > >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT not working > >>>>>>>>>>>> From: [email protected] > >>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>> > >>>>>>>>>>>> Hi Noel, > >>>>>>>>>>>> > >>>>>>>>>>>> Can you try using telnet to connect to an external > >>> webserver? > >>>>>>> telnet > >>>>>>>>>>>> www.google.com 80 > >>>>>>>>>>>> Can you also clarify: do you see the response packets > >>> reach > >>>>> the > >>>>>>> VR > >>>>>>>>> and/or > >>>>>>>>>>>> on what interfaces? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Marty > >>>>>>>>>>>> > >>>>>>>>>>>> On Saturday, September 14, 2013, Noel Kendall wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Guest OS cannot receive responses to http GETs from > >>>>> resources > >>>>>>> on > >>>>>>>>> the > >>>>>>>>>>>>> Internet. > >>>>>>>>>>>>> Network is advanced, VLAN isolated. > >>>>>>>>>>>>> What is working: > >>>>>>>>>>>>> - can browse guest website from internet- can ssh to > >>> guest > >>>>> from > >>>>>>>>>>> internet- > >>>>>>>>>>>>> can VPN to guest network from internet > >>>>>>>>>>>>> - network VR can access internet sites no problem > >>>>>>>>>>>>> What is not working: > >>>>>>>>>>>>> - guest http traffic to external website gets to VR on > >>>>> internal > >>>>>>>>> NIC, > >>>>>>>>>>>>> packets forwarded to external site via external NIC > >>>>>>>>>>>>> > >>>>>>>>>>>>> Response traffic is not seen. Appears to be dropped. > >>>>>>>>>>>>> Have been looking hard at IPTABLES rules, doing > >>> tcpdumps, > >>>>> etc. > >>>>>>>>>>>>> Am at this point stumped. > >>>>>>>>>>>>> Any ideas on what could be wrong, or how to determine > >>> what > >>>>>>> could be > >>>>>>>>>>> wrong? > >>>>>>>>>>>>> Thanks in advance everyone who tries to help! > >>>>>>>>>>>>> N. > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >>> > > >
