Secondary Storage VM timeout issue every hour

Hello People I have a strange issue where mgt server times out to send a command to secondary storage VM every hour and because of this UI won't be accessible for a short duration of time. Sometimes I have to restart mgt server to get it back to working state and sometimes I don't need to restart

[GitHub] [cloudstack-documentation] andrijapanicsb opened a new pull request #57: Minor L2 networks doc update

andrijapanicsb opened a new pull request #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57 This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #57: Minor L2 networks doc update

andrijapanicsb commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514961934 requesting docbuild This is an automated message from the Apache Git S

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #57: Minor L2 networks doc update

ACSGitBot commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514961976 Your request had been received, i'll go and build the documentation and check the output log for errors. This shouldn't take long.

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #57: Minor L2 networks doc update

ACSGitBot commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514962477 Build finished. You can review it at: https://acs-www.shapeblue.com/docs/pr57 Build Log Output: No log errors found to

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #57: Minor L2 networks doc update

andrijapanicsb commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514964539 @nvazquez @anuragaw @shwstppr - kindly LGTM - just image added. This i

[GitHub] [cloudstack-documentation] shwstppr commented on a change in pull request #57: Minor L2 networks doc update

shwstppr commented on a change in pull request #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#discussion_r307199327 ## File path: source/adminguide/networking.rst ## @@ -105,6 +105,10 @@ IP addresses. - Userdata and meta

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #57: Minor L2 networks doc update

andrijapanicsb commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514971006 requesting docbuild This is an automated message from the Apache Git S

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #57: Minor L2 networks doc update

ACSGitBot commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514971034 Your request had been received, i'll go and build the documentation and check the output log for errors. This shouldn't take long.

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #57: Minor L2 networks doc update

ACSGitBot commented on issue #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57#issuecomment-514971558 Build finished. You can review it at: https://acs-www.shapeblue.com/docs/pr57 Build Log Output: No log errors found to

[GitHub] [cloudstack-documentation] andrijapanicsb merged pull request #57: Minor L2 networks doc update

andrijapanicsb merged pull request #57: Minor L2 networks doc update URL: https://github.com/apache/cloudstack-documentation/pull/57 This is an automated message from the Apache Git Service. To respond to the message, please

Re: Latest Qemu KVM EV appears to be broken with ACS #3278

Hey Bobby, thanks Bobby. Which version of QEMU do you use? You mean 4.12 and later? Which CentOS and libvirt? We used the actually master, SystemVM and qemu 3.10 and 3.12. it seems it related to the cloudstack version 4.12 and above… Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actua

Re: Latest Qemu KVM EV appears to be broken with ACS #3278

On my KVM host I have: CentOS Linux release 7.6.1810 (Core) Running the following qemu related packages qemu-img-ev-2.12.0-18.el7_6.5.1.x86_64 libvirt-daemon-driver-qemu-4.5.0-10.el7_6.12.x86_64 ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch centos-release-qemu-ev-1.0-4.el7.centos.noarch qem

Re: Latest Qemu KVM EV appears to be broken with ACS #3278

Hi Sven, ACS 4.12 does not have the fix to make it work with qemu-ev, we only recently got it in 4.11 and it is only available curently with 4.11.3.0 or latest master. Regards, Rohit Yadav Software Architect, ShapeBlue https://www.shapeblue.com From: Boris

Re: Secondary Storage VM timeout issue every hour

Do you have a load balancer in front of cloudstack? Did you set the global settings "host" to the ip of the mgmt server? Le jeu. 25 juill. 2019 03 h 24, Rakesh Venkatesh a écrit : > Hello People > > > I have a strange issue where mgt server times out to send a command to > secondary storage VM

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #58: Hiding columns and keyboard labeling doc update

andrijapanicsb commented on issue #58: Hiding columns and keyboard labeling doc update URL: https://github.com/apache/cloudstack-documentation/pull/58#issuecomment-515041451 requesting docbuild This is an automated message f

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #58: Hiding columns and keyboard labeling doc update

ACSGitBot commented on issue #58: Hiding columns and keyboard labeling doc update URL: https://github.com/apache/cloudstack-documentation/pull/58#issuecomment-515041479 Your request had been received, i'll go and build the documentation and check the output log for errors. This sho

[GitHub] [cloudstack-documentation] andrijapanicsb opened a new pull request #58: Hiding columns and keyboard labeling doc update

andrijapanicsb opened a new pull request #58: Hiding columns and keyboard labeling doc update URL: https://github.com/apache/cloudstack-documentation/pull/58 UI modification documentation This is an automated message from the

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #58: Hiding columns and keyboard labeling doc update

ACSGitBot commented on issue #58: Hiding columns and keyboard labeling doc update URL: https://github.com/apache/cloudstack-documentation/pull/58#issuecomment-515041996 Build finished. You can review it at: https://acs-www.shapeblue.com/docs/pr58 Build Log Output: No

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #58: Hiding columns and keyboard labeling doc update

andrijapanicsb commented on issue #58: Hiding columns and keyboard labeling doc update URL: https://github.com/apache/cloudstack-documentation/pull/58#issuecomment-515047044 The new section here: https://acs-www.shapeblue.com/docs/pr58/adminguide/ui.html#modifying-the-ui @nvazquez

Re: Secondary Storage VM timeout issue every hour

Yes I have set the ip's of the three MGT servers in the "host" field Sent from my iPhone > On 25-Jul-2019, at 2:14 PM, Pierre-Luc Dion wrote: > > Do you have a load balancer in front of cloudstack? Did you set the global > settings "host" to the ip of the mgmt server? > > > Le jeu. 25 juill.

Re: Secondary Storage VM timeout issue every hour

Can you observe the status of SSVM (is it UP/Connecting/Disconnected/Down) while you have issues? I would advise checking your Secondary Storage itself - and also running the SSVM diagnose script /usr/local/cloud/systemvm/ssvm-check.sh - observe if any errors with NFS or others. Lastly - and don

Re: Secondary Storage VM timeout issue every hour

Yes I was monitoring it continuously. Below are the steps which I was doing when issue happened 1. Ping from MGT server to ssvm 2. Ping from ssvm to secondary storage ip 3. Ping from ssvm to public IP like 8.8.8.8 4. Ping from MGT server to node in which ssvm was running Out of all these, the

Re: Secondary Storage VM timeout issue every hour

since basic network connectivity (ping failures) was down between mgmts and nodes (and SSVM on it) - I would point my finger to your networking equipment - i.e. I expect zero problems with ACS (since pings fail). Let us know how it goes. Andrija On Thu, 25 Jul 2019 at 16:04, Rakesh v wrote: >

Re: Secondary Storage VM timeout issue every hour

The ping between mgt server and ssvm fails because mgt sends disconnect message to all nodes. If you look at the logs I pasted in first email, the mgt server thinks ssvm is lagging behind on ping and sends a disconnect message without investigation for all nodes. Also it happens at the beginning

Re: Secondary Storage VM timeout issue every hour

In your previous mail, I understond that you used the OS tool "ping" and NOT refering to internal ACS pings? "Out of all these, the ping drops were observed from MGT server to ssvm and mgt server to nodes. Basically all nodes lost connection. Then it recovered itself after 1 minute." On Thu, 25 J

答复: Ansible deploy cloudstack and add zone

Thank you all, I found a script for deploying zones on the github provided by Andrija, which is very helpful to me. https://github.com/shapeblue/Trillian/blob/master/Ansible/roles/cloudstack-config/templates/deployzone.sh.j2 发件人: Andrija Panic 发送时间: 2019年7月25日 6:51

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #59: Volumes from snaps doc update

ACSGitBot commented on issue #59: Volumes from snaps doc update URL: https://github.com/apache/cloudstack-documentation/pull/59#issuecomment-515090608 Your request had been received, i'll go and build the documentation and check the output log for errors. This shouldn't take long.

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #59: Volumes from snaps doc update

andrijapanicsb commented on issue #59: Volumes from snaps doc update URL: https://github.com/apache/cloudstack-documentation/pull/59#issuecomment-515090582 requesting docbuild This is an automated message from the Apache Git

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #59: Volumes from snaps doc update

ACSGitBot commented on issue #59: Volumes from snaps doc update URL: https://github.com/apache/cloudstack-documentation/pull/59#issuecomment-515091202 Build finished. You can review it at: https://acs-www.shapeblue.com/docs/pr59 Build Log Output: No log errors found to

[GitHub] [cloudstack-documentation] andrijapanicsb opened a new pull request #59: Volumes from snaps doc update

andrijapanicsb opened a new pull request #59: Volumes from snaps doc update URL: https://github.com/apache/cloudstack-documentation/pull/59 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #59: Volumes from snaps doc update

andrijapanicsb commented on issue #59: Volumes from snaps doc update URL: https://github.com/apache/cloudstack-documentation/pull/59#issuecomment-515096781 The new section here: https://acs-www.shapeblue.com/docs/pr59/adminguide/storage.html#creating-a-new-volume @nvazquez @anuragaw

[GitHub] [cloudstack-documentation] andrijapanicsb opened a new pull request #60: Volumes metrics doc update

andrijapanicsb opened a new pull request #60: Volumes metrics doc update URL: https://github.com/apache/cloudstack-documentation/pull/60 Volume metrics doc/explanation This is an automated message from the Apache Git Service.

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #60: Volumes metrics doc update

andrijapanicsb commented on issue #60: Volumes metrics doc update URL: https://github.com/apache/cloudstack-documentation/pull/60#issuecomment-515104806 requesting docbuild This is an automated message from the Apache Git Ser

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #60: Volumes metrics doc update

ACSGitBot commented on issue #60: Volumes metrics doc update URL: https://github.com/apache/cloudstack-documentation/pull/60#issuecomment-515104834 Your request had been received, i'll go and build the documentation and check the output log for errors. This shouldn't take long.

[GitHub] [cloudstack-documentation] ACSGitBot commented on issue #60: Volumes metrics doc update

ACSGitBot commented on issue #60: Volumes metrics doc update URL: https://github.com/apache/cloudstack-documentation/pull/60#issuecomment-515105317 Build finished. You can review it at: https://acs-www.shapeblue.com/docs/pr60 Build Log Output: No log errors found to re

[GitHub] [cloudstack-documentation] andrijapanicsb commented on issue #60: Volumes metrics doc update

andrijapanicsb commented on issue #60: Volumes metrics doc update URL: https://github.com/apache/cloudstack-documentation/pull/60#issuecomment-515105943 The new section here: https://acs-www.shapeblue.com/docs/pr60/adminguide/storage.html#volume-metrics @nvazquez @anuragaw @shwstppr

Re: Secondary Storage VM timeout issue every hour

True, but I thought since mgt sends disconnect to ssvm, I thought it will reset the interface on which is connecting Sent from my iPhone > On 25-Jul-2019, at 4:57 PM, Andrija Panic wrote: > > In your previous mail, I understond that you used the OS tool "ping" and > NOT refering to internal AC

Re: Secondary Storage VM timeout issue every hour

Irrelevant if mgmt sends (TCP) disconnect to the SSVM (which shoul not affect regular OS pings from mgmt to SSVM) - if you have issues pinging from mgmt to hypervisor node - than again I would bet on misbehaving switch equipment. On Thu, 25 Jul 2019 at 18:52, Rakesh v wrote: > True, but I thoug