Hi Ivan; I like to try virtio-scsi, i like to use its benefits like enlarging data disks without rebooting vm and getting unused space in vm back to my central storage, we also waited for long time, lets wait some more for 4.10 to become more stable :) Thanks for all replies
Özhan On Mon, Nov 13, 2017 at 12:29 PM, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com > wrote: > Ozhan, I suggest 4.10 only if you need KVM VM snapshots, virtio-scsi and > ipv6. Don't see other reasons to go from 4.9 (at least for kvm). But I > waited for 4.10 for a long time... > > 13 нояб. 2017 г. 16:21 пользователь "Özhan Rüzgar Karaman" < > oruzgarkara...@gmail.com> написал: > > > ok we will try to compile 4.10 from source and try it on our test > > environment. > > > > We currently use 4.9.x on production, i make this test for deciding if > 4.10 > > is suitable & stable for us, we also use very simple and generic setup > for > > this test no ipv6 just simple environment . So for production it looks > like > > we need to wait some time for upcoming releases, what do you think about > > that? > > > > There was a thread about this question one month ago and i remember that > > most of people still stick to 4.9 release for their production > > environments... > > > > Thanks > > Özhan > > > > On Mon, Nov 13, 2017 at 12:18 PM, Wei ZHOU <ustcweiz...@gmail.com> > wrote: > > > > > Hi Ivan, > > > > > > I would suggest you to create jira tickets for each problem you found > in > > > your testing, and create a github pull request for a jira ticket. > > > It is convenient for reviewers. > > > > > > Kind regards, > > > Wei > > > > > > 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <kudryavtsev...@bw-sw.com > >: > > > > > > > Hello, Ozhan > > > > > > > > https://github.com/apache/cloudstack/pull/2320 > > > > > > > > fixes everything I found right now. It enables functioning of > > everything > > > > correctly even if no IPv6 CIDR specified for network (at least for > > Ubuntu > > > > 14.04). > > > > For IPv6 configuration instruction please take a look at: > > > > https://github.com/apache/cloudstack/commit/ > > > f10c8bfe0c99a762c2606459413a47 > > > > 219614e775 > > > > (oh my god,I spend several hours trying to find how to configure IPv6 > > for > > > > 4.10). > > > > > > > > Please, don't forget to recreate SSVM because there is a fix for > > > templates > > > > too: > > > > https://github.com/apache/cloudstack/pull/2322 > > > > > > > > > > > > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman < > > > oruzgarkara...@gmail.com > > > > >: > > > > > > > > > Hi Ivan; > > > > > Does this hotfixes also solve qoutes and shell script interprets > > > problem? > > > > > We have no ipv6 setup and today we made similar test with fresh > > install > > > > > 4.10. We noticed that we receive similar error on security groups > > stage > > > > > while br_netfilter module is already active on our environment. We > > made > > > > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts > > > > > > > > > > Logs are below: > > > > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource. > > LibvirtComputingResource] > > > > > (agentRequest-Handler-1:null) Executing: > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 > > > --vmip6 > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: > > --rules > > > > > I:tcp:1:65535: > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535: > > > 0.0.0.0/0,NEXT > > > > ; > > > > > 2017-11-13 11:47:41,773 WARN [kvm.resource. > > LibvirtComputingResource] > > > > > (agentRequest-Handler-1:null) Exception: > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 > > > --vmip6 > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: > > --rules > > > > > I:tcp:1:65535: > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535: > > > 0.0.0.0/0,NEXT > > > > ; > > > > > java.lang.NullPointerException > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) > > > > > at com.cloud.utils.script.Script.execute(Script.java:214) > > > > > at com.cloud.utils.script.Script.execute(Script.java:182) > > > > > at > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > > > > addNetworkRules(LibvirtComputingResource.java:3429) > > > > > at > > > > > com.cloud.hypervisor.kvm.resource.wrapper. > > > LibvirtSecurityGroupRulesComma > > > > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57) > > > > > at > > > > > com.cloud.hypervisor.kvm.resource.wrapper. > > > LibvirtSecurityGroupRulesComma > > > > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36) > > > > > at > > > > > com.cloud.hypervisor.kvm.resource.wrapper. > > > LibvirtRequestWrapper.execute( > > > > > LibvirtRequestWrapper.java:75) > > > > > at > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > > > executeRequest( > > > > > LibvirtComputingResource.java:1369) > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525) > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask( > Agent.java:833) > > > > > at com.cloud.utils.nio.Task.call(Task.java:83) > > > > > at com.cloud.utils.nio.Task.call(Task.java:29) > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > > > > > ThreadPoolExecutor.java:1149) > > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > > > ThreadPoolExecutor.java:624) > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > 2017-11-13 11:47:41,774 WARN > > > > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper] > > > > > (agentRequest-Handler-1:null) Failed to program network rules for > vm > > > > > i-2-5-VM > > > > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent] > > > > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093: { Ans: , > > > > MgmtId: > > > > > 345048635880, via: 1, Ver: v1, Flags: 110, > > > > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{" > > > > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_ > > > > > FAILED","result":false,"details":"programming > > > > > network rules failed","wait":0}}] } > > > > > > > > > > > > > > > When we execute command with double quotas for rules section from > > > command > > > > > line it executes without a problem like below: > > > > > root@kvmt3:/var/log/cloudstack/agent# > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 > > > --vmip6 > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: > > --rules > > > > > "I:tcp:1:65535: > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535: > > > 0.0.0.0/0,NEXT > > > > ;" > > > > > root@kvmt3:/var/log/cloudstack/agent# echo $? > > > > > 0 > > > > > root@kvmt3:/var/log/cloudstack/agent# > > > > > > > > > > Thanks > > > > > Özhan > > > > > > > > > > > > > > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev < > > > > > kudryavtsev...@bw-sw.com> > > > > > wrote: > > > > > > > > > > > Hello, I implemented some hotfixes for 4.10 to work > > > > > > > > > > > > https://github.com/apache/cloudstack/pull/2319 - to master (load > > > > > > br_netfilter module) > > > > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which > > fixes > > > > SG > > > > > > failures related to ipv6. > > > > > > > > > > > > > > > > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev < > > > kudryavtsev...@bw-sw.com > > > > >: > > > > > > > > > > > > > Following up with previous question. I managed to make it work > by > > > > > > removing > > > > > > > all and heading to ubuntu 14.04 hypervisor host. > > > > > > > > > > > > > > Also, what I found more: > > > > > > > > > > > > > > 1. when setup databases (management server) if custom port is > > > > > specified, > > > > > > > databases themself is not created. If create manually, import > > > scripts > > > > > > work > > > > > > > fine. > > > > > > > 2. UI: unable to download ISO to __all__ zones. Have to specify > > > > certain > > > > > > > zone, else UI gives an error. > > > > > > > 3. Ubuntu doesn't load module *br_netfilter* but > > > > > > > > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_ > > group.py > > > > > > > > > > > > > > uses it and nothing good as a result: > > > > > > > > > > > > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call- > > > > > > arptables=1 > > > > > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call- > > > > > iptables=1 > > > > > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call- > > > > > > ip6tables=1 > > > > > > > > > > > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's > > > > necessary > > > > > to > > > > > > > add > > > > > > > > > > > > > > in script something like: modprobe br_netfilter (or smarter > > thing). > > > > > > > > > > > > > > But It doesn't work completely, actually, security groups are > > > > > > unfunctional: > > > > > > > > > > > > > > ==> /var/log/cloudstack/agent/agent.log <== > > > > > > > 2017-11-11 15:40:41,326 WARN [kvm.resource. > > > > LibvirtComputingResource] > > > > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception: > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_ > > group.py > > > > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip > 176.120.28.4 > > > > > --vmip6 > > > > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac > > > > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: > > > > --rules > > > > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I: > > > > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E: > > > > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT; > > > > > > > java.lang.NullPointerException > > > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) > > > > > > > at com.cloud.utils.script.Script.execute(Script.java:214) > > > > > > > at com.cloud.utils.script.Script.execute(Script.java:182) > > > > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > > > > > > addNetworkRules(LibvirtComputingResource.java:3429) > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper. > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute( > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57) > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper. > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute( > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36) > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper. > > > > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75) > > > > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource. > > > > > > > executeRequest(LibvirtComputingResource.java:1369) > > > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525) > > > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask( > > > Agent.java:833) > > > > > > > at com.cloud.utils.nio.Task.call(Task.java:83) > > > > > > > at com.cloud.utils.nio.Task.call(Task.java:29) > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > > > > > ThreadPoolExecutor.java:1149) > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > > > > > ThreadPoolExecutor.java:624) > > > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > 2017-11-11 15:40:41,327 WARN [resource.wrapper. > > > > > > > LibvirtSecurityGroupRulesCommandWrapper] > > > > (agentRequest-Handler-2:null) > > > > > > > (logid:eab9a328) Failed to program network rules for vm > i-2-7-VM > > > > > > > > > > > > > > So, no rules are actually created. Script doesn't call... I > > suppose > > > > may > > > > > > be > > > > > > > quotes are required because shell interprets ';' as command > > > > separator. > > > > > I > > > > > > > suppose that optimization introduced in 4.10, because in 4.9 > SGs > > > work > > > > > > like > > > > > > > a charm... > > > > > > > > > > > > > > > > > > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <paul.an...@shapeblue.com > >: > > > > > > > > > > > > > >> Ivan, > > > > > > >> > > > > > > >> Can you paste a larger section of unfiltered logs. There > would > > > > always > > > > > > be > > > > > > >> a message explaining why the mgmt. server thought that a VR > > should > > > > be > > > > > > shut > > > > > > >> down > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> Kind regards, > > > > > > >> > > > > > > >> Paul Angus > > > > > > >> > > > > > > >> paul.an...@shapeblue.com > > > > > > >> www.shapeblue.com > > > > > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > > >> @shapeblue > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> -----Original Message----- > > > > > > >> From: Simon Weller [mailto:swel...@ena.com.INVALID] > > > > > > >> Sent: 10 November 2017 18:39 > > > > > > >> To: dev@cloudstack.apache.org > > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > > > > >> > > > > > > >> What VR template image are you using? > > > > > > >> > > > > > > >> > > > > > > >> ________________________________ > > > > > > >> From: Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > > > > > >> Sent: Friday, November 10, 2017 11:59 AM > > > > > > >> To: dev@cloudstack.apache.org > > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > > > > >> > > > > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other > > > > system > > > > > > vms > > > > > > >> are working. I even added to communication script on compute > > node > > > > > "sleep > > > > > > >> 3600" before ssh, so response to management is delayed, I > logged > > > so > > > > to > > > > > > VR, > > > > > > >> all interfaces are up, iptables rules are OK. > > > > > > >> > > > > > > >> So agent rolls vr good, but stops it by management order with > no > > > > > obvious > > > > > > >> reason. > > > > > > >> > > > > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller" > > > > > > <swel...@ena.com.invalid > > > > > > >> > > > > > > > >> написал: > > > > > > >> > > > > > > >> > Is the storage ceph? > > > > > > >> > > > > > > > >> > > > > > > > >> > ________________________________ > > > > > > >> > From: Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > > > > > >> > Sent: Friday, November 10, 2017 11:52 AM > > > > > > >> > To: dev@cloudstack.apache.org > > > > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > > > > >> > > > > > > > >> > Hi, I did, and it does the things right, I even added "tee" > to > > > ssh > > > > > > >> > 3922 communication script to out vr response to additional > log > > > and > > > > > it > > > > > > >> > only receives VR version line and sends all info (the same > > from > > > > > > >> > pastebin) to ACS and receives "stop" order. > > > > > > >> > > > > > > > >> > I'll try to provide additional info, but ad you can see, > > > > management > > > > > > >> > receives proper response and sends stop next op. It looks > very > > > > > freaky > > > > > > >> > without any notification... > > > > > > >> > > > > > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller" > > > > > > >> > <swel...@ena.com.invalid > > > > > > >> > > > > > > > > >> > написал: > > > > > > >> > > > > > > > >> > > Ivan, > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > Can you put the host agents into debug mode? Hopefully > that > > > will > > > > > > >> > > provide more information. > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/ > > > > > > KVM+agent+deb > > > > > > >> > > ug > > > > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...< > > > > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/ > > > > > KVM+agent+debug> > > > > > > >> cwiki.apache.org > > > > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit > > > > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_ > > > > > > socket,address=8787 > > > > > > >> ... > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > - Si > > > > > > >> > > > > > > > > >> > > ________________________________ > > > > > > >> > > From: Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > > > > > > >> > > Sent: Friday, November 10, 2017 11:34 AM > > > > > > >> > > To: dev@cloudstack.apache.org > > > > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem > > > > > > >> > > > > > > > > >> > > Hello, Devs. > > > > > > >> > > > > > > > > >> > > I experience VR Start Problem in the fresh ACS 4.10 > > deployment > > > > > > >> > > > > > > > > >> > > Intersting place of logs is here: > > > https://pastebin.com/iBXRBA5N > > > > > > >> [https://pastebin.com/i/facebook.png]<https:// > > > pastebin.com/iBXRBA5N > > > > > > > > > > > >> > > > > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] > > > > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com< > > https://pastebin. > > > > > > >> com/iBXRBA5N> > > > > > > >> pastebin.com > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > >> > > Basically, the situation looks like: > > > > > > >> > > > > > > > > >> > > 1. Management Server tries to launch VR 2. It gets from > > Agent > > > > > proper > > > > > > >> > > VR response with VR details 3. It sends StopCommand > without > > > > > > >> > > explanation. > > > > > > >> > > > > > > > > >> > > I'm trying to figure out what happens inside, but the > > codebase > > > > is > > > > > > >> > > huge > > > > > > >> > and > > > > > > >> > > still no positive results. Please, let me know if you have > > any > > > > > ideas > > > > > > >> > which > > > > > > >> > > could help me finding the reason. Thanks a lot. > > > > > > >> > > > > > > > > >> > > -- > > > > > > >> > > With best regards, Ivan Kudryavtsev > > > > > > >> > > Bitworks Software, Ltd. > > > > > > >> > > Cell: +7-923-414-1515 > > > > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > With best regards, Ivan Kudryavtsev > > > > > > > Bitworks Software, Ltd. > > > > > > > Cell: +7-923-414-1515 > > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > With best regards, Ivan Kudryavtsev > > > > > > Bitworks Software, Ltd. > > > > > > Cell: +7-923-414-1515 > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > With best regards, Ivan Kudryavtsev > > > > Bitworks Software, Ltd. > > > > Cell: +7-923-414-1515 > > > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > > >
