Hi all, I feel obligated to share update, to close the issue:
Nothing to do with kernel/qemu etc.. Seem that hidden Docker NAT/Masquerade rules don't play nice with VNET... Description of the problem as given originally still is valid, but root cause is as above... Apologies for wasting everyone's time and thanks for all the inputs. Andrija On 10 October 2017 at 12:18, Wei ZHOU <ustcweiz...@gmail.com> wrote: > Andrija, > > We had similar issue before. However, we use advanced zone with security > groups, and the issue is because some security groups rules (iptables > rules) are not applied by security_group.py successfully. > is there any iptables rules on the hypervisors ? > > -Wei > > 2017-10-10 11:23 GMT+02:00 Andrija Panic <andrija.pa...@gmail.com>: > > > Hi, > > > > @Wei, no we are using VXLAN, advanced networking... problem is that > packet > > not passed from bridge to the VNET - that is "all"... > > > > @Ivan, we did upgrade few hosts to kernel, 4.4 (made available from > Ubuntu > > 16.04 to Ubuntu 14.04), but again we there had some issues with FortiOS > > (some special OS, not Linux based as I was told), that RDP apps behind > this > > FW are "slow" (probably laggy), when this FortiGate VM is on new > kernel... > > > > But I'm sure we will move to 4.4, this bug is really driving me crazy... > :( > > > > THx > > > > On 10 October 2017 at 09:52, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > wrote: > > > > > Andrija, I saw it in the past. Problem might be coolnnected with kernel > > > version and vnet itself. Try to look for it. I don't remember how we > > > overcame it in the past... > > > > > > 10 окт. 2017 г. 8:07 ДП пользователь "Wei ZHOU" <ustcweiz...@gmail.com > > > > > написал: > > > > > > > Hi Andrija, > > > > > > > > Are using advanced zone with isolated network or security groups ? > > > > > > > > -Wei > > > > > > > > > > > > 2017-10-09 22:52 GMT+02:00 Andrija Panic <andrija.pa...@gmail.com>: > > > > > > > > > Hi guys, > > > > > > > > > > we have occasional but serious problem, that starts happening as it > > > seems > > > > > randomly (i.e. NOT under high load) - not ACS related afaik, > purely > > > KVM, > > > > > but feedback is really welcomed. > > > > > > > > > > - VM is reachable in general from everywhere, but not reachable > from > > > > > specific IP address ?! > > > > > - VM is NOT under high load, network traffic next to zero, same for > > > > > CPU/disk... > > > > > - We mitigate this problem by migrating VM away to another host, > not > > > much > > > > > of a solution... > > > > > > > > > > Description of problem: > > > > > > > > > > We let ping from "problematic" source IP address to the problematic > > VM, > > > > and > > > > > we capture traffic on KVM host where the problematic VM lives: > > > > > > > > > > - Tcpdump on VXLAN interface (physical incoming interface on the > > host) > > > - > > > > we > > > > > see packet fine > > > > > - tcpdump on BRIDGE = we see packet fine > > > > > - tcpdump on VNET = we DON'T see packet. > > > > > > > > > > In the scenario above, I need to say that : > > > > > - we can tcpdump packets from other source IPs on the VNET > interface > > > just > > > > > fine (as expected), so should also see this problematic source IP's > > > > packets > > > > > - we can actually ping in oposite direction - from the problematic > VM > > > to > > > > > the problematic "source" IP > > > > > > > > > > We checked everything possible, from bridge port forwarding, to > > > > mac-to-vtep > > > > > mapping, to many other things, removed traffic shaping from VNET > > > > interface, > > > > > no iptables/ebtables, no STP on bridge, remove and rejoin > interfaces > > to > > > > > bridge, destroy bridge and create manually on the fly, > > > > > > > > > > Problem is really crazy, and I can not explain it - no iptables, no > > > > > ebtables for troubleshooting pruposes (on this host) and > > > > > > > > > > We mitigate this problem by migrating VM away to another host, not > > much > > > > of > > > > > a solution... > > > > > > > > > > This is Ubuntu 14.04, Qemu 2.5 (libvirt 1.3.1), > > > > > Stock kernel 3.16-xx, regular bridge (not OVS) > > > > > > > > > > Anyone else ever heard of such problem - this is not intermittent > > > packet > > > > > dropping, but complete blackout/packet drop in some way... > > > > > > > > > > Thanks, > > > > > > > > > > -- > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > -- > > > > Andrija Panić > > > -- Andrija Panić
