Hi Joris,
here is the management log: http://pastebin.com/zxnKxFhk
Interesting parts (to me): in bold
2014-05-30 13:56:21,899 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync inspecting src type
TEMPLATE copyAsync inspecting dest type VOLUME
2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Sending {
Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","origUrl":"
http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2
","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"QCOW2","accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","
*hvm":true*
,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD://
cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD://
cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":1064,"vmName":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
}
2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-12:null) Seq 4-1248669612: Processing: { Ans: ,
MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"9c440d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}]
}
2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Received: {
Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2014-05-30 13:56:23,773 DEBUG
[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic NicProfile[
*1092-801-null*-46.232.xxx.246-vlan://untagged of type Public from the nics
passed on vm start. The nic will be plugged later
2014-05-30 13:56:23,773 DEBUG
[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vlan://44
of type Guest from the nics passed on vm start. The nic will be plugged
later
2014-05-30 13:56:23,773 DEBUG
[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vlan://43
of type Guest from the nics passed on vm start. The nic will be plugged
later
2014-05-30 13:56:23,773 DEBUG
[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3004
of type Guest from the nics passed on vm start. The nic will be plugged
later
2014-05-30 13:56:23,773 DEBUG
[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3004
of type Guest from the nics passed on vm start
Thanks,
On 30 May 2014 13:54, Joris van Lieshout <jvanliesh...@schubergphilis.com>
wrote:
> Hi Andrija,
>
> Just the start of the VR should be sufficient.
>
> Kind regards,
> Joris van Lieshout
>
> Schuberg Philis
> Boeingavenue 271
> 1119 PD Schiphol-Rijk
> schubergphilis.com
>
> +31 20-7506672
> +31 6-51428188
>
>
>
>
> On 30/05/14 13:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
>
> >Hi Joris,
> >
> >just to be sure - you want me to capture the log from the moment I reboot
> >router - or you want me to stop it, then start capturing log, and start it
> >(and continue capture untill ethnull errors inside VR) ?
> >
> >Thanks,
> >
> >
> >On 30 May 2014 13:39, Joris van Lieshout <jvanliesh...@schubergphilis.com
> >
> >wrote:
> >
> >> Hi Andrija,
> >>
> >> Thanks for the answers. In deed your situation is different so PV/HVM is
> >> not the issue.
> >>
> >> When reading back the log output you have provided I noted that the VR
> >> messages log indicates that it's waiting for ethnull to be up. This
> >>raises
> >> the question where null was introduced instead of 1. The ACS management
> >> log output you send was, what I think, later down the road where ACS
> >>gives
> >> up trying to wait for the VR to come up. If you would capture the
> >> job-executor in the management log from startCommand till the exception,
> >> do you see anywhere a mention of ethnull? You might need to reed into
> >>the
> >> DirectAgent executing the startCommand to find a clue. The thing is
> >>that I
> >> only have experience with XS based environment so I cannot point you to
> >> the exact output to look for. On XS, at least, it is
> >> "[c.c.h.x.r.CitrixResourceBase] (DirectAgent-351:ctx-4a51bb9e) Created a
> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1"
> >>
> >> Kind regards,
> >> Joris van Lieshout
> >>
> >> Schuberg Philis
> >>
> >>
> >>
> >>
> >>
> >> On 30/05/14 10:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
> >>
> >> >Hi Deen,
> >> >no, in DB there is field "vlan_id" with value "untagged" - that
> >> >"vlan://untagged" is shown from ACS gui, and is used in API call (or
> >> >better
> >> >said commands that are seen in management server logs).
> >> >
> >> >Best,
> >> >Andrija
> >> >
> >> >
> >> >On 30 May 2014 10:37, Daan Hoogland <daan.hoogl...@gmail.com> wrote:
> >> >
> >> >> Andrija,
> >> >>
> >> >> Do not just assign a second net vlan://500 You have one like that and
> >> >> you don't want conflicting nets using the same vlan. I am wondering
> >> >> why 'untagged' comes out as 'vlan://untagged'. I think that is the
> >> >> bug. Did you find the string 'vlan://untagged' in your db?
> >> >>
> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic
> >> >><andrija.pa...@gmail.com>
> >> >> wrote:
> >> >> > Hi Joris,
> >> >> >
> >> >> > thank you for taking time to address this issue :)
> >> >> >
> >> >> > So...:
> >> >> >
> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank for CEPH
> >>support),
> >> >>OS
> >> >> is
> >> >> > Centos 6.5, libvirt 1.2.3 compiled.
> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine
> >> >> > - not XS, so I guess no answers for this part :)
> >> >> > - guest_os_id is 184 = Debian 7 x64
> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 x64
> >> >> >
> >> >> > This worked previously on 4.2.1 = template was ofcourse
> >> >>systemvm-kvm-4.2
> >> >> -
> >> >> > but that was also Debian 7 x64 type... so this should not be the
> >> >>issues
> >> >> > (guest not supported by host...)
> >> >> >
> >> >> > The only thing that might be out of "standard" = all SVMs are on
> >>CEPH
> >> >>-
> >> >> > there are official docs on altering database to make some new
> >>System
> >> >> > Offering as default for SSVM and CPVM - what I did, I also have
> >>done
> >> >>same
> >> >> > config in DB, to make VR use another System Offering as default -
> >> >>which
> >> >> is
> >> >> > NOT explained in the docs - you could use "Change Offering..."
> >>button
> >> >>on
> >> >> > exiting, shutdown VR to change it per docs...
> >> >> > But still this worked all fine on 4.2.1...
> >> >> >
> >> >> > - regarding /var/cache/cloud/cmdline the content is folowing at the
> >> >> moment
> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline
> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal dns1=8.8.8.8 dns2=
> >> >> template=domP
> >> >> > name=r-801-VM eth0ip=169.254.0.75 eth0mask=255.255.0.0
> >>type=vpcrouter
> >> >> > disable_rp_filter=true
> >> >> >
> >> >> > Also please note that only eth1 does not have IP info, eth0
> >>(control
> >> >> > 169.xxx) and all other eh2 and up that are used for Tiers get IP
> >>info
> >> >> fine.
> >> >> > I could also manually add IP for eth1 (public NIC) and start ifup
> >> >>eth1 -
> >> >> > and it works fine, but adding new IP Port Forwarding etc does not
> >> >>work...
> >> >> >
> >> >> > Daan or somebody said it could be realted to my "Public" network
> >>(in
> >> >>the
> >> >> > Zones, Physical Network, eth1 listing) is NOT tagged
> >> >>(vlan://untagged)...
> >> >> > Interestingly the only VR that does work fine is the VR used in
> >>Shared
> >> >> > network, but that VR is using IP from Guest IP range (also
> >>efectively
> >> >> > public IPs but on vlan 500)
> >> >> >
> >> >> > I was instructed to try to change Public IP range from untagged to
> >> >>vlan
> >> >> > 500, but I'm not sure how to do this, if there is any way at all
> >> >>(editing
> >> >> > "vlan" table and changing to vlan 500 does not work, after
> >>rebooting
> >> >>VR
> >> >> > from ACS gui).
> >> >> >
> >> >> > :)
> >> >> >
> >> >> > So, not sure what is roughly expected date for 4.4, but right now,
> >>I'm
> >> >> > pretty stuck with a big problem of all VPC not operational at
> >>all...
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 30 May 2014 08:27, Joris van Lieshout <
> >> >> jvanliesh...@schubergphilis.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Hi Andrija,
> >> >> >>
> >> >> >> Daan asked me to have a look at this as well. Looking at you
> >>issue I
> >> >> >> recall having seen something similar. Back then when upgrading
> >>4.2.1
> >> >>to
> >> >> >> 4.3 I though it had to do with out own custom build svm template.
> >> >> >> Let me fire off some questions before explaining what the cause
> >>was
> >> >>in
> >> >> our
> >> >> >> case. :)
> >> >> >>
> >> >> >> - what hypervisor (and version) are you using?
> >> >> >> - if XS, is the new VR a para-virtualised instance (PV) or
> >>hardware
> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR uuid and check
> >>that
> >> >> >> param PV-args is set and HVM-boot-policy is unset.
> >> >> >> - what is the OS type of the VR in ACS (guest_os_id in vm_instance
> >> >>table
> >> >> >> and match with table guest_os)
> >> >> >> - what is the OS type of the SVM template?
> >> >> >>
> >> >> >> Now for the explaining. :)
> >> >> >> In our case the OS type of the new template was not supported on
> >>the
> >> >> >> XenServer version we are running. Therefore the VR was started by
> >>XS
> >> >>as
> >> >> a
> >> >> >> HVM guest. System vms on XS rely on the arguments passed to them
> >>in
> >> >>the
> >> >> >> PV-args param (ends up on the guest in /var/cache/cloud/cmdline
> >> >>which in
> >> >> >> turn is used by cloud-early-config) in order to work. cmdline
> >> >>contains
> >> >> the
> >> >> >> NIC configuration information.
> >> >> >> So, long story short, if a VR gets started as a HVM it will not
> >>get
> >> >>the
> >> >> >> information needed to configure it's NICs.
> >> >> >>
> >> >> >> Workaround
> >> >> >> We corrected the os_type_id in the DB (yes I know editing the DB
> >>is
> >> >> >> something you usually don't want but there is no other way in this
> >> >>case)
> >> >> >> of the existing VR's and of the systemvmtemplate to something
> >> >>supported
> >> >> by
> >> >> >> XenServer.
> >> >> >>
> >> >> >> Kind regards,
> >> >> >> Joris van Lieshout
> >> >> >>
> >> >> >> Schuberg Philis
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 29/05/14 12:18, "Andrija Panic" <andrija.pa...@gmail.com>
> >>wrote:
> >> >> >>
> >> >> >> >They are 2 traffic types on 1 physical net (that is both tagged
> >>vlan
> >> >> 500,
> >> >> >> >and untagged packets travel over same KVM bridge, and over eth1
> >>to
> >> >> outside
> >> >> >> >world)...
> >> >> >> >
> >> >> >> >
> >> >> >> >On 29 May 2014 12:04, Daan Hoogland <daan.hoogl...@gmail.com>
> >> wrote:
> >> >> >> >
> >> >> >> >> Are these two traffic types in one physical net? or two
> >>physical
> >> >>nets
> >> >> >> >> on the same interface (seems wrong).
> >> >> >> >>
> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal Reddy Uradi
> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote:
> >> >> >> >> > I don't think editing DB table will work.
> >> >> >> >> >
> >> >> >> >> > -Jayapal
> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija Panic
> >> >><andrija.pa...@gmail.com
> >> >> >
> >> >> >> >> wrote:
> >> >> >> >> >
> >> >> >> >> >> It's like this:
> >> >> >> >> >>
> >> >> >> >> >> I have public subnet /24.
> >> >> >> >> >>
> >> >> >> >> >> half is dedicated for Guest traffic (vlan 500) and the
> >>second
> >> >> half is
> >> >> >> >> >> dedicated to Public traffic/network (no vlan tags, that is
> >> >> untagged
> >> >> >> >> packets)
> >> >> >> >> >>
> >> >> >> >> >> Both vlan500 and untagged packets travel over physical eth1
> >> >> >> >>interface on
> >> >> >> >> >> hypervisors and can reach Internet.
> >> >> >> >> >>
> >> >> >> >> >> Thanks,
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland
> >><daan.hoogl...@gmail.com>
> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM, Andrija Panic <
> >> >> >> >> andrija.pa...@gmail.com>
> >> >> >> >> >>> wrote:
> >> >> >> >> >>>> 500
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> is 500 the vlan of your guestnetwork or your physical
> >>network?
> >> >> You
> >> >> >> >> >>> wouldn't want to have two nets with vlan 500!
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> Daan
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >>
> >> >> >> >> >> Andrija Panić
> >> >> >> >> >> --------------------------------------
> >> >> >> >> >> http://admintweets.com
> >> >> >> >> >> --------------------------------------
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Daan
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >--
> >> >> >> >
> >> >> >> >Andrija Panić
> >> >> >> >--------------------------------------
> >> >> >> > http://admintweets.com
> >> >> >> >--------------------------------------
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrija Panić
> >> >> > --------------------------------------
> >> >> > http://admintweets.com
> >> >> > --------------------------------------
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Daan
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >
> >> >Andrija Panić
> >> >--------------------------------------
> >> > http://admintweets.com
> >> >--------------------------------------
> >>
> >>
> >
> >
> >--
> >
> >Andrija Panić
> >--------------------------------------
> > http://admintweets.com
> >--------------------------------------
>
>
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
