Hi Ivan, I would suggest you to create jira tickets for each problem you found in your testing, and create a github pull request for a jira ticket. It is convenient for reviewers.
Kind regards, Wei 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>: > Hello, Ozhan > > https://github.com/apache/cloudstack/pull/2320 > > fixes everything I found right now. It enables functioning of everything > correctly even if no IPv6 CIDR specified for network (at least for Ubuntu > 14.04). > For IPv6 configuration instruction please take a look at: > https://github.com/apache/cloudstack/commit/f10c8bfe0c99a762c2606459413a47 > 219614e775 > (oh my god,I spend several hours trying to find how to configure IPv6 for > 4.10). > > Please, don't forget to recreate SSVM because there is a fix for templates > too: > https://github.com/apache/cloudstack/pull/2322 > > > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <oruzgarkara...@gmail.com > >: > > > Hi Ivan; > > Does this hotfixes also solve qoutes and shell script interprets problem? > > We have no ipv6 setup and today we made similar test with fresh install > > 4.10. We noticed that we receive similar error on security groups stage > > while br_netfilter module is already active on our environment. We made > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts > > > > Logs are below: > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.LibvirtComputingResource] > > (agentRequest-Handler-1:null) Executing: > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6 > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules > > I:tcp:1:65535: > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT > ; > > 2017-11-13 11:47:41,773 WARN [kvm.resource.LibvirtComputingResource] > > (agentRequest-Handler-1:null) Exception: > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6 > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules > > I:tcp:1:65535: > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT > ; > > java.lang.NullPointerException > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) > > at com.cloud.utils.script.Script.execute(Script.java:214) > > at com.cloud.utils.script.Script.execute(Script.java:182) > > at > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > addNetworkRules(LibvirtComputingResource.java:3429) > > at > > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesComma > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57) > > at > > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesComma > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36) > > at > > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute( > > LibvirtRequestWrapper.java:75) > > at > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > executeRequest( > > LibvirtComputingResource.java:1369) > > at com.cloud.agent.Agent.processRequest(Agent.java:525) > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833) > > at com.cloud.utils.nio.Task.call(Task.java:83) > > at com.cloud.utils.nio.Task.call(Task.java:29) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2017-11-13 11:47:41,774 WARN > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper] > > (agentRequest-Handler-1:null) Failed to program network rules for vm > > i-2-5-VM > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent] > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093: { Ans: , > MgmtId: > > 345048635880, via: 1, Ver: v1, Flags: 110, > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{" > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_ > > FAILED","result":false,"details":"programming > > network rules failed","wait":0}}] } > > > > > > When we execute command with double quotas for rules section from command > > line it executes without a problem like below: > > root@kvmt3:/var/log/cloudstack/agent# > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6 > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules > > "I:tcp:1:65535: > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT > ;" > > root@kvmt3:/var/log/cloudstack/agent# echo $? > > 0 > > root@kvmt3:/var/log/cloudstack/agent# > > > > Thanks > > Özhan > > > > > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev < > > kudryavtsev...@bw-sw.com> > > wrote: > > > > > Hello, I implemented some hotfixes for 4.10 to work > > > > > > https://github.com/apache/cloudstack/pull/2319 - to master (load > > > br_netfilter module) > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes > SG > > > failures related to ipv6. > > > > > > > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <kudryavtsev...@bw-sw.com > >: > > > > > > > Following up with previous question. I managed to make it work by > > > removing > > > > all and heading to ubuntu 14.04 hypervisor host. > > > > > > > > Also, what I found more: > > > > > > > > 1. when setup databases (management server) if custom port is > > specified, > > > > databases themself is not created. If create manually, import scripts > > > work > > > > fine. > > > > 2. UI: unable to download ISO to __all__ zones. Have to specify > certain > > > > zone, else UI gives an error. > > > > 3. Ubuntu doesn't load module *br_netfilter* but > > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > > > > > > > uses it and nothing good as a result: > > > > > > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call- > > > arptables=1 > > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call- > > iptables=1 > > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call- > > > ip6tables=1 > > > > > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's > necessary > > to > > > > add > > > > > > > > in script something like: modprobe br_netfilter (or smarter thing). > > > > > > > > But It doesn't work completely, actually, security groups are > > > unfunctional: > > > > > > > > ==> /var/log/cloudstack/agent/agent.log <== > > > > 2017-11-11 15:40:41,326 WARN [kvm.resource. > LibvirtComputingResource] > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception: > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4 > > --vmip6 > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: > --rules > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I: > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E: > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT; > > > > java.lang.NullPointerException > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) > > > > at com.cloud.utils.script.Script.execute(Script.java:214) > > > > at com.cloud.utils.script.Script.execute(Script.java:182) > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > > > addNetworkRules(LibvirtComputingResource.java:3429) > > > > at com.cloud.hypervisor.kvm.resource.wrapper. > > > > LibvirtSecurityGroupRulesCommandWrapper.execute( > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57) > > > > at com.cloud.hypervisor.kvm.resource.wrapper. > > > > LibvirtSecurityGroupRulesCommandWrapper.execute( > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36) > > > > at com.cloud.hypervisor.kvm.resource.wrapper. > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75) > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > > > executeRequest(LibvirtComputingResource.java:1369) > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525) > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833) > > > > at com.cloud.utils.nio.Task.call(Task.java:83) > > > > at com.cloud.utils.nio.Task.call(Task.java:29) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > > ThreadPoolExecutor.java:1149) > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > > ThreadPoolExecutor.java:624) > > > > at java.lang.Thread.run(Thread.java:748) > > > > 2017-11-11 15:40:41,327 WARN [resource.wrapper. > > > > LibvirtSecurityGroupRulesCommandWrapper] > (agentRequest-Handler-2:null) > > > > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM > > > > > > > > So, no rules are actually created. Script doesn't call... I suppose > may > > > be > > > > quotes are required because shell interprets ';' as command > separator. > > I > > > > suppose that optimization introduced in 4.10, because in 4.9 SGs work > > > like > > > > a charm... > > > > > > > > > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <paul.an...@shapeblue.com>: > > > > > > > >> Ivan, > > > >> > > > >> Can you paste a larger section of unfiltered logs. There would > always > > > be > > > >> a message explaining why the mgmt. server thought that a VR should > be > > > shut > > > >> down > > > >> > > > >> > > > >> > > > >> Kind regards, > > > >> > > > >> Paul Angus > > > >> > > > >> paul.an...@shapeblue.com > > > >> www.shapeblue.com > > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > >> @shapeblue > > > >> > > > >> > > > >> > > > >> > > > >> -----Original Message----- > > > >> From: Simon Weller [mailto:swel...@ena.com.INVALID] > > > >> Sent: 10 November 2017 18:39 > > > >> To: dev@cloudstack.apache.org > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > >> > > > >> What VR template image are you using? > > > >> > > > >> > > > >> ________________________________ > > > >> From: Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > > >> Sent: Friday, November 10, 2017 11:59 AM > > > >> To: dev@cloudstack.apache.org > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > >> > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other > system > > > vms > > > >> are working. I even added to communication script on compute node > > "sleep > > > >> 3600" before ssh, so response to management is delayed, I logged so > to > > > VR, > > > >> all interfaces are up, iptables rules are OK. > > > >> > > > >> So agent rolls vr good, but stops it by management order with no > > obvious > > > >> reason. > > > >> > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller" > > > <swel...@ena.com.invalid > > > >> > > > > >> написал: > > > >> > > > >> > Is the storage ceph? > > > >> > > > > >> > > > > >> > ________________________________ > > > >> > From: Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > > >> > Sent: Friday, November 10, 2017 11:52 AM > > > >> > To: dev@cloudstack.apache.org > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > >> > > > > >> > Hi, I did, and it does the things right, I even added "tee" to ssh > > > >> > 3922 communication script to out vr response to additional log and > > it > > > >> > only receives VR version line and sends all info (the same from > > > >> > pastebin) to ACS and receives "stop" order. > > > >> > > > > >> > I'll try to provide additional info, but ad you can see, > management > > > >> > receives proper response and sends stop next op. It looks very > > freaky > > > >> > without any notification... > > > >> > > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller" > > > >> > <swel...@ena.com.invalid > > > >> > > > > > >> > написал: > > > >> > > > > >> > > Ivan, > > > >> > > > > > >> > > > > > >> > > Can you put the host agents into debug mode? Hopefully that will > > > >> > > provide more information. > > > >> > > > > > >> > > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/ > > > KVM+agent+deb > > > >> > > ug > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...< > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/ > > KVM+agent+debug> > > > >> cwiki.apache.org > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_ > > > socket,address=8787 > > > >> ... > > > >> > > > >> > > > >> > > > >> > > > > > >> > > > > > >> > > - Si > > > >> > > > > > >> > > ________________________________ > > > >> > > From: Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > > >> > > Sent: Friday, November 10, 2017 11:34 AM > > > >> > > To: dev@cloudstack.apache.org > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > >> > > > > > >> > > Hello, Devs. > > > >> > > > > > >> > > I experience VR Start Problem in the fresh ACS 4.10 deployment > > > >> > > > > > >> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N > > > >> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N > > > > > >> > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin. > > > >> com/iBXRBA5N> > > > >> pastebin.com > > > >> > > > >> > > > >> > > > >> > > > > > >> > > Basically, the situation looks like: > > > >> > > > > > >> > > 1. Management Server tries to launch VR 2. It gets from Agent > > proper > > > >> > > VR response with VR details 3. It sends StopCommand without > > > >> > > explanation. > > > >> > > > > > >> > > I'm trying to figure out what happens inside, but the codebase > is > > > >> > > huge > > > >> > and > > > >> > > still no positive results. Please, let me know if you have any > > ideas > > > >> > which > > > >> > > could help me finding the reason. Thanks a lot. > > > >> > > > > > >> > > -- > > > >> > > With best regards, Ivan Kudryavtsev > > > >> > > Bitworks Software, Ltd. > > > >> > > Cell: +7-923-414-1515 > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > > > > > -- > > > > With best regards, Ivan Kudryavtsev > > > > Bitworks Software, Ltd. > > > > Cell: +7-923-414-1515 > > > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > > > > > > > > > > -- > > > With best regards, Ivan Kudryavtsev > > > Bitworks Software, Ltd. > > > Cell: +7-923-414-1515 > > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > > -- > With best regards, Ivan Kudryavtsev > Bitworks Software, Ltd. > Cell: +7-923-414-1515 > WWW: http://bitworks.software/ <http://bw-sw.com/> >
