Re: Can't create Basic Zone with master
Mike, absolutely right and I would usually just say "okay revert" but in
this case I want to look to fix forward. This commit is an attempt to make
network types more pluggable. So let me see if I can find a solution. I had
trusted on the smoke tests to pick this up.
On Thu, Oct 12, 2017 at 10:50 PM, Tutkowski, Mike wrote:
> It looks like this commit broke support for Basic Zones:
>
> a06530d (Daan: Can you look into this?)
>
> 4.10
>
> @Override
> protected boolean canHandle(NetworkOffering offering, DataCenter dc) {
> // this guru handles system Direct pod based network
> if (dc.getNetworkType() == NetworkType.Basic &&
> isMyTrafficType(offering.getTrafficType())) {
> return true;
> } else {
> s_logger.trace("We only take care of Guest Direct Pod based
> networks");
> return false;
> }
> }
>
> master
>
> @Override
> protected boolean canHandle(NetworkOffering offering, DataCenter dc,
> PhysicalNetwork physnet) {
> // this guru handles system Direct pod based network
> if (dc.getNetworkType() == NetworkType.Basic &&
> isMyTrafficType(offering.getTrafficType()) &&
> isMyIsolationMethod(physnet)) {
> return true;
> } else {
> s_logger.trace("We only take care of Guest Direct Pod based
> networks");
> return false;
> }
> }
>
> isMyIsolationMethod(physnet) fails (new to master):
>
> protected boolean isMyIsolationMethod(PhysicalNetwork
> physicalNetwork) {
> for (IsolationMethod m : _isolationMethods) {
> if (physicalNetwork.getIsolationMethods().contains(m.toString()))
> {
> return true;
> }
> }
> return false;
> }
>
> _isolationMethods = new IsolationMethod[] { new IsolationMethod("VLAN") };
>
> _isolationMethods is only of size = 1 and m.toString() returns “VLAN”
> whereas physicalNetwork.getIsolationMethods() is of size = 0. This method
> returns false and none of the network gurus can support the creation of a
> Basic Zone.
>
> On 10/12/17, 7:03 AM, "Tutkowski, Mike" wrote:
>
> Yes, I can debug it more. I’m in the middle of some other task at the
> moment, so I might not be able to get to it until later this evening. In
> the meanwhile, I figured I’d send out the previous e-mail in case it might
> sound like something one of our networking devs recognize. :) Thanks!
>
> On Oct 12, 2017, at 12:18 AM, Rohit Yadav mailto:rohit.ya...@shapeblue.com>> wrote:
>
> Hi Mike,
>
>
> I think you can repeat the process with 4.10 and compare with master
> as to which NetworkGuru is supposed to design and return you a network,
> then you can do some git log tracing to find what was changed and why the
> respective network guru is not returning a network. It could be
> environment/config issue or change/regression in code.
>
>
> For local/nfs storage, and all three hypervisor no major regression
> was seen in last trillian smoke test runs.
>
>
> - Rohit
>
>
> From: Tutkowski, Mike mike.tutkow...@netapp.com>>
> Sent: Thursday, October 12, 2017 1:17:50 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Can't create Basic Zone with master
>
> Hi Rohit,
>
> I walked through the debugger with this and now have more info:
>
> The following network gurus are leveraged to create networks
> successfully:
>
> PublicNetworkGuru
> PodBasedNetworkGuru
> ControlNetworkGuru
> StorageNetworkGuru
>
> Once the zone-creation wizard reaches the “Enabling Security Group
> provider” part, NetworkOrchestrator.setupNetwork is again invoked and
> this time throws an exception.
>
> It walks through all of its network gurus and none of them returns a
> pointer to a Network object.
>
>for (final NetworkGuru guru : networkGurus) {
>final Network network = guru.design(offering, plan,
> predefined, owner);
>if (network == null) {
>continue;
>}
>
> Since no Network object was returned, the list of Network objects is
> empty and an exception is thrown.
>
> Thanks,
> Mike
>
> On 10/11/17, 1:44 AM, "Rohit Yadav" rohit.ya...@shapeblue.com>> wrote:
>
>Mike,
>
>
>Can you re-deploy a fresh environment and see what's failing,
> stepping through the workflow. The specific line it's throwing exception is
> caused when there are no network (network size is 0) setup:
>
>
>https://github.com/apache/cloudstack/blob/master/engine/
> orchestration/src/org/apache/cloudstack/engine/orchestration/
> NetworkOrchestrator.java#L715
>
>
>- Rohit
>
>
>From: Tutkowski, Mike mike.tutkow...@netapp.com>>
>Sent: Wednesday, October 11, 2017 6:08:10 AM
>To: dev@cloudstack.apache.org
>Subject: Can't create Basic Zone with master
>
>
Re: Can't create Basic Zone with master
Mike, i created https://github.com/apache/cloudstack/pull/2296 after
investigation. haven't tested it yet.
On Fri, Oct 13, 2017 at 9:29 AM, Daan Hoogland
wrote:
> Mike, absolutely right and I would usually just say "okay revert" but in
> this case I want to look to fix forward. This commit is an attempt to make
> network types more pluggable. So let me see if I can find a solution. I had
> trusted on the smoke tests to pick this up.
>
> On Thu, Oct 12, 2017 at 10:50 PM, Tutkowski, Mike <
> mike.tutkow...@netapp.com> wrote:
>
>> It looks like this commit broke support for Basic Zones:
>>
>> a06530d (Daan: Can you look into this?)
>>
>> 4.10
>>
>> @Override
>> protected boolean canHandle(NetworkOffering offering, DataCenter dc) {
>> // this guru handles system Direct pod based network
>> if (dc.getNetworkType() == NetworkType.Basic &&
>> isMyTrafficType(offering.getTrafficType())) {
>> return true;
>> } else {
>> s_logger.trace("We only take care of Guest Direct Pod based
>> networks");
>> return false;
>> }
>> }
>>
>> master
>>
>> @Override
>> protected boolean canHandle(NetworkOffering offering, DataCenter dc,
>> PhysicalNetwork physnet) {
>> // this guru handles system Direct pod based network
>> if (dc.getNetworkType() == NetworkType.Basic &&
>> isMyTrafficType(offering.getTrafficType()) &&
>> isMyIsolationMethod(physnet)) {
>> return true;
>> } else {
>> s_logger.trace("We only take care of Guest Direct Pod based
>> networks");
>> return false;
>> }
>> }
>>
>> isMyIsolationMethod(physnet) fails (new to master):
>>
>> protected boolean isMyIsolationMethod(PhysicalNetwork
>> physicalNetwork) {
>> for (IsolationMethod m : _isolationMethods) {
>> if (physicalNetwork.getIsolationM
>> ethods().contains(m.toString())) {
>> return true;
>> }
>> }
>> return false;
>> }
>>
>> _isolationMethods = new IsolationMethod[] { new IsolationMethod("VLAN") };
>>
>> _isolationMethods is only of size = 1 and m.toString() returns “VLAN”
>> whereas physicalNetwork.getIsolationMethods() is of size = 0. This
>> method returns false and none of the network gurus can support the creation
>> of a Basic Zone.
>>
>> On 10/12/17, 7:03 AM, "Tutkowski, Mike"
>> wrote:
>>
>> Yes, I can debug it more. I’m in the middle of some other task at the
>> moment, so I might not be able to get to it until later this evening. In
>> the meanwhile, I figured I’d send out the previous e-mail in case it might
>> sound like something one of our networking devs recognize. :) Thanks!
>>
>> On Oct 12, 2017, at 12:18 AM, Rohit Yadav > > wrote:
>>
>> Hi Mike,
>>
>>
>> I think you can repeat the process with 4.10 and compare with master
>> as to which NetworkGuru is supposed to design and return you a network,
>> then you can do some git log tracing to find what was changed and why the
>> respective network guru is not returning a network. It could be
>> environment/config issue or change/regression in code.
>>
>>
>> For local/nfs storage, and all three hypervisor no major regression
>> was seen in last trillian smoke test runs.
>>
>>
>> - Rohit
>>
>>
>> From: Tutkowski, Mike > mike.tutkow...@netapp.com>>
>> Sent: Thursday, October 12, 2017 1:17:50 AM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Can't create Basic Zone with master
>>
>> Hi Rohit,
>>
>> I walked through the debugger with this and now have more info:
>>
>> The following network gurus are leveraged to create networks
>> successfully:
>>
>> PublicNetworkGuru
>> PodBasedNetworkGuru
>> ControlNetworkGuru
>> StorageNetworkGuru
>>
>> Once the zone-creation wizard reaches the “Enabling Security Group
>> provider” part, NetworkOrchestrator.setupNetwork is again invoked and
>> this time throws an exception.
>>
>> It walks through all of its network gurus and none of them returns a
>> pointer to a Network object.
>>
>>for (final NetworkGuru guru : networkGurus) {
>>final Network network = guru.design(offering, plan,
>> predefined, owner);
>>if (network == null) {
>>continue;
>>}
>>
>> Since no Network object was returned, the list of Network objects is
>> empty and an exception is thrown.
>>
>> Thanks,
>> Mike
>>
>> On 10/11/17, 1:44 AM, "Rohit Yadav" > lto:rohit.ya...@shapeblue.com>> wrote:
>>
>>Mike,
>>
>>
>>Can you re-deploy a fresh environment and see what's failing,
>> stepping through the workflow. The specific line it's throwing exception is
>> caused when there are no network (network size is 0) setup:
>>
>>
>>https://github.com/apache/cloudstack/blob/master/engine/orc
>> hestration/src/org/apache/cloudstack/engine/orchestra
Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the database
sorry Cristian, I wasn't clear. Can you add a jira ticket at issues.apache.org/jira? @devs I have heard this before but am not sure if it was exactly the same. Anybody else? Was this fixed in 4.10 or master yet? tl;dr setting password doesn't work for Cristian due to the firewall on the VR. On Fri, Oct 13, 2017 at 1:44 PM, Ciobanu Cristian wrote: > Daan, > > Can I track this issue somewhere? > > > Thank you! > Cristian > > -Original Message- > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com] > Sent: Thursday, October 12, 2017 9:46 AM > To: us...@cloudstack.apache.org > Subject: Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the database > > Cristian, > > I am cross-posting this to dev@coudstack.a.o, sorry for the noise. I have > heard this before but am not sure if it was exactly the same. It sounds > like a bug to me. Can you enter one in jira? > Anybody else? Was this fixed in 4.10 or master yet? > > On Wed, Oct 11, 2017 at 11:23 PM, Ciobanu Cristian > > wrote: > > > Hello, > > > > After I continued the investigation I found that router is not > > allowing the VM to fetch the password (port 8080) not sure what is > > wrong with this version but in order to fix I have added this rule "-A > > INPUT -p tcp -m tcp --dport 8080 -j ACCEPT" on virtual router.. > > > > Any idea how to fix this, permanent? In case reinstall the router > > for sure I will have to do this again. > > > > > > Thank you. > > Cristian > > > > -Original Message- > > From: Ciobanu Cristian [mailto:cristian.c@istream.today] > > Sent: Wednesday, October 11, 2017 9:45 PM > > To: 'us...@cloudstack.apache.org' > > Subject: RE: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the > > database > > > > I was able to update the virtual router only after I have restarted > > the iptables-persistend form virtual router side, not sure what is > > wrong with this router version but I don't get the right status if I > > don't restart the iptables from virtual router, I have did multiple > tests : > > > > Example : > > - Reboot router > > - getting this error : job-3247/job-3248, cmd: CheckSshCommand) > > (logid:6b3de9fa) Trying to > > > connect to 149.56.xxx.xx ( I see this error in management log ) > > - restart iptables from router side > > - works > > > > In case I destroy/restart the router I have to repeat the same > > thing, restart iptables... > > > > I also tested a new deploy with a template used before on this > > setup, looks like the password set for new deploy is not working... > > > > > > Any idea why I have those issue? > > > > Upgraded from 4.5.2 to 4.9.3 ( shapeblue packages ) > > > > > > Regards, > > Cristian > > > > > > -Original Message- > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com] > > Sent: Wednesday, October 11, 2017 6:10 PM > > To: us...@cloudstack.apache.org > > Subject: Re: Upgrade Cloudstack 4.5 to 4.9 - Unable to upgrade the > > database > > > > that is a tricky one. You can destroy the vm from VmWare and set the > > state to stopped in the DB. It should then start a new one. > > check if r-4-VM/ROOT-4-01.vmdk has gone from the datastore. > > > > the cause is probably some synchronisation issue. We've seen and > > solved it before but are yet to find the root cause. > > > > > > On Wed, Oct 11, 2017 at 4:23 PM, Ciobanu Cristian > > > > > > wrote: > > > > > Hello, > > > > > >Was my mistake, forgot to add the system template from cloudstack > > > interface before starting the upgrade, now I have other issue, looks > > > like after I have enabled one of the zone and click on upgrade > > > virtual router I still see the state "starting" if I check the > > > router from vSphere side is up. > > > > > > > > > This is what is see : > > > > > > tting > > > catalina.out:INFO [c.c.h.v.r.VmwareResource] > > > (DirectAgent-381:ctx-5f16ff50 ns53.ip-149-56-xx.net, > > > job-3247/job-3248, cmd: StartCommand) (logid:6b3de9fa) Chose disk > > > controller for vol ROOT -> osdefault, based on root disk controller > > > settings at global configuration setting. > > > catalina.out:INFO [c.c.h.v.m.DatastoreMO] > > > (DirectAgent-381:ctx-5f16ff50 ns53.ip-149-56-xx.net, > > > job-3247/job-3248, cmd: StartCommand) > > > (logid:6b3de9fa) Folder r-4-VM exists on datastore catalina.out:INFO > > > [c.c.h.v.m.DatastoreMO] (DirectAgent-381:ctx-5f16ff50 > > > ns53.ip-149-56-xx.net, job-3247/job-3248, cmd: StartCommand) > > > (logid:6b3de9fa) Search file ROOT-4-01.vmdk on [datastore1] > > > catalina.out:INFO [c.c.h.v.m.DatastoreMO] > > > (DirectAgent-381:ctx-5f16ff50 ns53.ip-149-56-xx.net, > > > job-3247/job-3248, cmd: StartCommand) > > > (logid:6b3de9fa) File [datastore1] ROOT-4-01.vmdk exists on > > > datastore catalina.out:INFO [c.c.h.v.m.DatastoreMO] > > > (DirectAgent-381:ctx-5f16ff50 ns53.ip-149-56-xx.net, > > > job-3247/job-3248, cmd: StartCommand) > > > (logid:6b3de9fa) Search file ROOT-4-01-flat.vmdk on [datastore1] > > > catalina.out:I
Re: Help/Advice needed - some traffic don't reach VNET / VM
Hi all, I feel obligated to share update, to close the issue: Nothing to do with kernel/qemu etc.. Seem that hidden Docker NAT/Masquerade rules don't play nice with VNET... Description of the problem as given originally still is valid, but root cause is as above... Apologies for wasting everyone's time and thanks for all the inputs. Andrija On 10 October 2017 at 12:18, Wei ZHOU wrote: > Andrija, > > We had similar issue before. However, we use advanced zone with security > groups, and the issue is because some security groups rules (iptables > rules) are not applied by security_group.py successfully. > is there any iptables rules on the hypervisors ? > > -Wei > > 2017-10-10 11:23 GMT+02:00 Andrija Panic : > > > Hi, > > > > @Wei, no we are using VXLAN, advanced networking... problem is that > packet > > not passed from bridge to the VNET - that is "all"... > > > > @Ivan, we did upgrade few hosts to kernel, 4.4 (made available from > Ubuntu > > 16.04 to Ubuntu 14.04), but again we there had some issues with FortiOS > > (some special OS, not Linux based as I was told), that RDP apps behind > this > > FW are "slow" (probably laggy), when this FortiGate VM is on new > kernel... > > > > But I'm sure we will move to 4.4, this bug is really driving me crazy... > :( > > > > THx > > > > On 10 October 2017 at 09:52, Ivan Kudryavtsev > > wrote: > > > > > Andrija, I saw it in the past. Problem might be coolnnected with kernel > > > version and vnet itself. Try to look for it. I don't remember how we > > > overcame it in the past... > > > > > > 10 окт. 2017 г. 8:07 ДП пользователь "Wei ZHOU" > > > > написал: > > > > > > > Hi Andrija, > > > > > > > > Are using advanced zone with isolated network or security groups ? > > > > > > > > -Wei > > > > > > > > > > > > 2017-10-09 22:52 GMT+02:00 Andrija Panic : > > > > > > > > > Hi guys, > > > > > > > > > > we have occasional but serious problem, that starts happening as it > > > seems > > > > > randomly (i.e. NOT under high load) - not ACS related afaik, > purely > > > KVM, > > > > > but feedback is really welcomed. > > > > > > > > > > - VM is reachable in general from everywhere, but not reachable > from > > > > > specific IP address ?! > > > > > - VM is NOT under high load, network traffic next to zero, same for > > > > > CPU/disk... > > > > > - We mitigate this problem by migrating VM away to another host, > not > > > much > > > > > of a solution... > > > > > > > > > > Description of problem: > > > > > > > > > > We let ping from "problematic" source IP address to the problematic > > VM, > > > > and > > > > > we capture traffic on KVM host where the problematic VM lives: > > > > > > > > > > - Tcpdump on VXLAN interface (physical incoming interface on the > > host) > > > - > > > > we > > > > > see packet fine > > > > > - tcpdump on BRIDGE = we see packet fine > > > > > - tcpdump on VNET = we DON'T see packet. > > > > > > > > > > In the scenario above, I need to say that : > > > > > - we can tcpdump packets from other source IPs on the VNET > interface > > > just > > > > > fine (as expected), so should also see this problematic source IP's > > > > packets > > > > > - we can actually ping in oposite direction - from the problematic > VM > > > to > > > > > the problematic "source" IP > > > > > > > > > > We checked everything possible, from bridge port forwarding, to > > > > mac-to-vtep > > > > > mapping, to many other things, removed traffic shaping from VNET > > > > interface, > > > > > no iptables/ebtables, no STP on bridge, remove and rejoin > interfaces > > to > > > > > bridge, destroy bridge and create manually on the fly, > > > > > > > > > > Problem is really crazy, and I can not explain it - no iptables, no > > > > > ebtables for troubleshooting pruposes (on this host) and > > > > > > > > > > We mitigate this problem by migrating VM away to another host, not > > much > > > > of > > > > > a solution... > > > > > > > > > > This is Ubuntu 14.04, Qemu 2.5 (libvirt 1.3.1), > > > > > Stock kernel 3.16-xx, regular bridge (not OVS) > > > > > > > > > > Anyone else ever heard of such problem - this is not intermittent > > > packet > > > > > dropping, but complete blackout/packet drop in some way... > > > > > > > > > > Thanks, > > > > > > > > > > -- > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > -- > > > > Andrija Panić > > > -- Andrija Panić
Re: Help/Advice needed - some traffic don't reach VNET / VM
Hi Andrija, Good to see your update and know you found the root cause. -Wei 2017-10-13 22:16 GMT+02:00 Andrija Panic : > Hi all, > > I feel obligated to share update, to close the issue: > > Nothing to do with kernel/qemu etc.. Seem that hidden Docker NAT/Masquerade > rules don't play nice with VNET... > > Description of the problem as given originally still is valid, but root > cause is as above... > > Apologies for wasting everyone's time and thanks for all the inputs. > > Andrija > > On 10 October 2017 at 12:18, Wei ZHOU wrote: > > > Andrija, > > > > We had similar issue before. However, we use advanced zone with security > > groups, and the issue is because some security groups rules (iptables > > rules) are not applied by security_group.py successfully. > > is there any iptables rules on the hypervisors ? > > > > -Wei > > > > 2017-10-10 11:23 GMT+02:00 Andrija Panic : > > > > > Hi, > > > > > > @Wei, no we are using VXLAN, advanced networking... problem is that > > packet > > > not passed from bridge to the VNET - that is "all"... > > > > > > @Ivan, we did upgrade few hosts to kernel, 4.4 (made available from > > Ubuntu > > > 16.04 to Ubuntu 14.04), but again we there had some issues with FortiOS > > > (some special OS, not Linux based as I was told), that RDP apps behind > > this > > > FW are "slow" (probably laggy), when this FortiGate VM is on new > > kernel... > > > > > > But I'm sure we will move to 4.4, this bug is really driving me > crazy... > > :( > > > > > > THx > > > > > > On 10 October 2017 at 09:52, Ivan Kudryavtsev < > kudryavtsev...@bw-sw.com> > > > wrote: > > > > > > > Andrija, I saw it in the past. Problem might be coolnnected with > kernel > > > > version and vnet itself. Try to look for it. I don't remember how we > > > > overcame it in the past... > > > > > > > > 10 окт. 2017 г. 8:07 ДП пользователь "Wei ZHOU" < > ustcweiz...@gmail.com > > > > > > > написал: > > > > > > > > > Hi Andrija, > > > > > > > > > > Are using advanced zone with isolated network or security groups ? > > > > > > > > > > -Wei > > > > > > > > > > > > > > > 2017-10-09 22:52 GMT+02:00 Andrija Panic >: > > > > > > > > > > > Hi guys, > > > > > > > > > > > > we have occasional but serious problem, that starts happening as > it > > > > seems > > > > > > randomly (i.e. NOT under high load) - not ACS related afaik, > > purely > > > > KVM, > > > > > > but feedback is really welcomed. > > > > > > > > > > > > - VM is reachable in general from everywhere, but not reachable > > from > > > > > > specific IP address ?! > > > > > > - VM is NOT under high load, network traffic next to zero, same > for > > > > > > CPU/disk... > > > > > > - We mitigate this problem by migrating VM away to another host, > > not > > > > much > > > > > > of a solution... > > > > > > > > > > > > Description of problem: > > > > > > > > > > > > We let ping from "problematic" source IP address to the > problematic > > > VM, > > > > > and > > > > > > we capture traffic on KVM host where the problematic VM lives: > > > > > > > > > > > > - Tcpdump on VXLAN interface (physical incoming interface on the > > > host) > > > > - > > > > > we > > > > > > see packet fine > > > > > > - tcpdump on BRIDGE = we see packet fine > > > > > > - tcpdump on VNET = we DON'T see packet. > > > > > > > > > > > > In the scenario above, I need to say that : > > > > > > - we can tcpdump packets from other source IPs on the VNET > > interface > > > > just > > > > > > fine (as expected), so should also see this problematic source > IP's > > > > > packets > > > > > > - we can actually ping in oposite direction - from the > problematic > > VM > > > > to > > > > > > the problematic "source" IP > > > > > > > > > > > > We checked everything possible, from bridge port forwarding, to > > > > > mac-to-vtep > > > > > > mapping, to many other things, removed traffic shaping from VNET > > > > > interface, > > > > > > no iptables/ebtables, no STP on bridge, remove and rejoin > > interfaces > > > to > > > > > > bridge, destroy bridge and create manually on the fly, > > > > > > > > > > > > Problem is really crazy, and I can not explain it - no iptables, > no > > > > > > ebtables for troubleshooting pruposes (on this host) and > > > > > > > > > > > > We mitigate this problem by migrating VM away to another host, > not > > > much > > > > > of > > > > > > a solution... > > > > > > > > > > > > This is Ubuntu 14.04, Qemu 2.5 (libvirt 1.3.1), > > > > > > Stock kernel 3.16-xx, regular bridge (not OVS) > > > > > > > > > > > > Anyone else ever heard of such problem - this is not intermittent > > > > packet > > > > > > dropping, but complete blackout/packet drop in some way... > > > > > > > > > > > > Thanks, > > > > > > > > > > > > -- > > > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > > -- > > Andrija Panić >
