Hi Joris, I have turned on DEBUG loging in agent.log on cs1.xxx/net host:
So, management logs again: http://pastebin.com/F6BRf7Y9 Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC Not playing smart, but there is some error: [kvm.resource.KVMGuestOsMapper] (agentRequest-Handler-3:) Can't find the mapping of guest os: Debian GNU/Linux 7(64-bit) Best, Andrija On 30 May 2014 14:26, Joris van Lieshout <jvanliesh...@schubergphilis.com> wrote: > Hi Andrija, > > Bold formatting does not come trough on the dev list. :) > But u might need a bit more info. > > At a certain point I see this line > > 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request] > (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 1-609104082: Sending { > Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net), Ver: v1, Flags: 100111, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM"... > > > This is where the information is passed on to the agent handles. For XS > this would initiate an agent handler on the management server but for KVM, > if I remember correctly, it passed the command on to the cloudstack agent > service on the hypervisor. > > Can you check the cloud service log on the KVM hypervisor executing the > request? it's this server cs1.xxxxx.net and then search top down for > 609104082 in the log. See if you can provide the log from the agent > handler thread started by that sequence. > > Kind regards, > Joris van Lieshout > > Schuberg Philis > Boeingavenue 271 > 1119 PD Schiphol-Rijk > schubergphilis.com > > +31 20-7506672 > +31 6-51428188 > > > > > On 30/05/14 14:08, "Andrija Panic" <andrija.pa...@gmail.com> wrote: > > >Hi Joris, > > > >here is the management log: http://pastebin.com/zxnKxFhk > > > >Interesting parts (to me): in bold > > > >2014-05-30 13:56:21,899 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync inspecting src type > >TEMPLATE copyAsync inspecting dest type VOLUME > >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Sending { > >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, Flags: > 100011, > >[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach > >e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0f-b0b4 > >-5e351e7cae55","origUrl":" > > > http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mast > >er-kvm.qcow2.bz2 > >","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"QCOW2", > >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff"," > >*hvm":true* > >,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.cloudstack > >.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1 > >dc6","id":209,"poolType":"RBD","host":" > >cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD:// > > > cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88 > >a8-2203f79b1dc6 > >"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorType":" > >KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid" > >:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataStore":{" > >org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a6 > >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":" > >cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD:// > > > cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88 > >a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":1064,"vm > >Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":0,"hyp > >ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}] > >} > >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] > >(AgentManager-Handler-12:null) Seq 4-1248669612: Processing: { Ans: , > >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, > >[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.a > >pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"9c4 > >40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id":0}}," > >result":true,"wait":0}}] > >} > >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Received: { > >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { CopyCmdAnswer > >} } > >2014-05-30 13:56:23,773 DEBUG > >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic NicProfile[ > >*1092-801-null*-46.232.xxx.246-vlan://untagged of type Public from the > >nics > >passed on vm start. The nic will be plugged later > >2014-05-30 13:56:23,773 DEBUG > >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > >NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vlan://4 > >4 > >of type Guest from the nics passed on vm start. The nic will be plugged > >later > >2014-05-30 13:56:23,773 DEBUG > >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > >NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vlan://4 > >3 > >of type Guest from the nics passed on vm start. The nic will be plugged > >later > >2014-05-30 13:56:23,773 DEBUG > >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > >NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3 > >004 > >of type Guest from the nics passed on vm start. The nic will be plugged > >later > >2014-05-30 13:56:23,773 DEBUG > >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic > >NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3 > >004 > >of type Guest from the nics passed on vm start > > > > > >Thanks, > > > > > >On 30 May 2014 13:54, Joris van Lieshout <jvanliesh...@schubergphilis.com > > > >wrote: > > > >> Hi Andrija, > >> > >> Just the start of the VR should be sufficient. > >> > >> Kind regards, > >> Joris van Lieshout > >> > >> Schuberg Philis > >> Boeingavenue 271 > >> 1119 PD Schiphol-Rijk > >> schubergphilis.com > >> > >> +31 20-7506672 > >> +31 6-51428188 > >> > >> > >> > >> > >> On 30/05/14 13:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote: > >> > >> >Hi Joris, > >> > > >> >just to be sure - you want me to capture the log from the moment I > >>reboot > >> >router - or you want me to stop it, then start capturing log, and > >>start it > >> >(and continue capture untill ethnull errors inside VR) ? > >> > > >> >Thanks, > >> > > >> > > >> >On 30 May 2014 13:39, Joris van Lieshout > >><jvanliesh...@schubergphilis.com > >> > > >> >wrote: > >> > > >> >> Hi Andrija, > >> >> > >> >> Thanks for the answers. In deed your situation is different so > >>PV/HVM is > >> >> not the issue. > >> >> > >> >> When reading back the log output you have provided I noted that the > >>VR > >> >> messages log indicates that it's waiting for ethnull to be up. This > >> >>raises > >> >> the question where null was introduced instead of 1. The ACS > >>management > >> >> log output you send was, what I think, later down the road where ACS > >> >>gives > >> >> up trying to wait for the VR to come up. If you would capture the > >> >> job-executor in the management log from startCommand till the > >>exception, > >> >> do you see anywhere a mention of ethnull? You might need to reed into > >> >>the > >> >> DirectAgent executing the startCommand to find a clue. The thing is > >> >>that I > >> >> only have experience with XS based environment so I cannot point you > >>to > >> >> the exact output to look for. On XS, at least, it is > >> >> "[c.c.h.x.r.CitrixResourceBase] (DirectAgent-351:ctx-4a51bb9e) > >>Created a > >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1" > >> >> > >> >> Kind regards, > >> >> Joris van Lieshout > >> >> > >> >> Schuberg Philis > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On 30/05/14 10:48, "Andrija Panic" <andrija.pa...@gmail.com> wrote: > >> >> > >> >> >Hi Deen, > >> >> >no, in DB there is field "vlan_id" with value "untagged" - that > >> >> >"vlan://untagged" is shown from ACS gui, and is used in API call (or > >> >> >better > >> >> >said commands that are seen in management server logs). > >> >> > > >> >> >Best, > >> >> >Andrija > >> >> > > >> >> > > >> >> >On 30 May 2014 10:37, Daan Hoogland <daan.hoogl...@gmail.com> > wrote: > >> >> > > >> >> >> Andrija, > >> >> >> > >> >> >> Do not just assign a second net vlan://500 You have one like that > >>and > >> >> >> you don't want conflicting nets using the same vlan. I am > >>wondering > >> >> >> why 'untagged' comes out as 'vlan://untagged'. I think that is the > >> >> >> bug. Did you find the string 'vlan://untagged' in your db? > >> >> >> > >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic > >> >> >><andrija.pa...@gmail.com> > >> >> >> wrote: > >> >> >> > Hi Joris, > >> >> >> > > >> >> >> > thank you for taking time to address this issue :) > >> >> >> > > >> >> >> > So...: > >> >> >> > > >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank for CEPH > >> >>support), > >> >> >>OS > >> >> >> is > >> >> >> > Centos 6.5, libvirt 1.2.3 compiled. > >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine > >> >> >> > - not XS, so I guess no answers for this part :) > >> >> >> > - guest_os_id is 184 = Debian 7 x64 > >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 x64 > >> >> >> > > >> >> >> > This worked previously on 4.2.1 = template was ofcourse > >> >> >>systemvm-kvm-4.2 > >> >> >> - > >> >> >> > but that was also Debian 7 x64 type... so this should not be the > >> >> >>issues > >> >> >> > (guest not supported by host...) > >> >> >> > > >> >> >> > The only thing that might be out of "standard" = all SVMs are on > >> >>CEPH > >> >> >>- > >> >> >> > there are official docs on altering database to make some new > >> >>System > >> >> >> > Offering as default for SSVM and CPVM - what I did, I also have > >> >>done > >> >> >>same > >> >> >> > config in DB, to make VR use another System Offering as default > >>- > >> >> >>which > >> >> >> is > >> >> >> > NOT explained in the docs - you could use "Change Offering..." > >> >>button > >> >> >>on > >> >> >> > exiting, shutdown VR to change it per docs... > >> >> >> > But still this worked all fine on 4.2.1... > >> >> >> > > >> >> >> > - regarding /var/cache/cloud/cmdline the content is folowing at > >>the > >> >> >> moment > >> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline > >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal dns1=8.8.8.8 dns2= > >> >> >> template=domP > >> >> >> > name=r-801-VM eth0ip=169.254.0.75 eth0mask=255.255.0.0 > >> >>type=vpcrouter > >> >> >> > disable_rp_filter=true > >> >> >> > > >> >> >> > Also please note that only eth1 does not have IP info, eth0 > >> >>(control > >> >> >> > 169.xxx) and all other eh2 and up that are used for Tiers get IP > >> >>info > >> >> >> fine. > >> >> >> > I could also manually add IP for eth1 (public NIC) and start > >>ifup > >> >> >>eth1 - > >> >> >> > and it works fine, but adding new IP Port Forwarding etc does > >>not > >> >> >>work... > >> >> >> > > >> >> >> > Daan or somebody said it could be realted to my "Public" network > >> >>(in > >> >> >>the > >> >> >> > Zones, Physical Network, eth1 listing) is NOT tagged > >> >> >>(vlan://untagged)... > >> >> >> > Interestingly the only VR that does work fine is the VR used in > >> >>Shared > >> >> >> > network, but that VR is using IP from Guest IP range (also > >> >>efectively > >> >> >> > public IPs but on vlan 500) > >> >> >> > > >> >> >> > I was instructed to try to change Public IP range from untagged > >>to > >> >> >>vlan > >> >> >> > 500, but I'm not sure how to do this, if there is any way at all > >> >> >>(editing > >> >> >> > "vlan" table and changing to vlan 500 does not work, after > >> >>rebooting > >> >> >>VR > >> >> >> > from ACS gui). > >> >> >> > > >> >> >> > :) > >> >> >> > > >> >> >> > So, not sure what is roughly expected date for 4.4, but right > >>now, > >> >>I'm > >> >> >> > pretty stuck with a big problem of all VPC not operational at > >> >>all... > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > On 30 May 2014 08:27, Joris van Lieshout < > >> >> >> jvanliesh...@schubergphilis.com> > >> >> >> > wrote: > >> >> >> > > >> >> >> >> Hi Andrija, > >> >> >> >> > >> >> >> >> Daan asked me to have a look at this as well. Looking at you > >> >>issue I > >> >> >> >> recall having seen something similar. Back then when upgrading > >> >>4.2.1 > >> >> >>to > >> >> >> >> 4.3 I though it had to do with out own custom build svm > >>template. > >> >> >> >> Let me fire off some questions before explaining what the cause > >> >>was > >> >> >>in > >> >> >> our > >> >> >> >> case. :) > >> >> >> >> > >> >> >> >> - what hypervisor (and version) are you using? > >> >> >> >> - if XS, is the new VR a para-virtualised instance (PV) or > >> >>hardware > >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR uuid and > >>check > >> >>that > >> >> >> >> param PV-args is set and HVM-boot-policy is unset. > >> >> >> >> - what is the OS type of the VR in ACS (guest_os_id in > >>vm_instance > >> >> >>table > >> >> >> >> and match with table guest_os) > >> >> >> >> - what is the OS type of the SVM template? > >> >> >> >> > >> >> >> >> Now for the explaining. :) > >> >> >> >> In our case the OS type of the new template was not supported > >>on > >> >>the > >> >> >> >> XenServer version we are running. Therefore the VR was started > >>by > >> >>XS > >> >> >>as > >> >> >> a > >> >> >> >> HVM guest. System vms on XS rely on the arguments passed to > >>them > >> >>in > >> >> >>the > >> >> >> >> PV-args param (ends up on the guest in /var/cache/cloud/cmdline > >> >> >>which in > >> >> >> >> turn is used by cloud-early-config) in order to work. cmdline > >> >> >>contains > >> >> >> the > >> >> >> >> NIC configuration information. > >> >> >> >> So, long story short, if a VR gets started as a HVM it will not > >> >>get > >> >> >>the > >> >> >> >> information needed to configure it's NICs. > >> >> >> >> > >> >> >> >> Workaround > >> >> >> >> We corrected the os_type_id in the DB (yes I know editing the > >>DB > >> >>is > >> >> >> >> something you usually don't want but there is no other way in > >>this > >> >> >>case) > >> >> >> >> of the existing VR's and of the systemvmtemplate to something > >> >> >>supported > >> >> >> by > >> >> >> >> XenServer. > >> >> >> >> > >> >> >> >> Kind regards, > >> >> >> >> Joris van Lieshout > >> >> >> >> > >> >> >> >> Schuberg Philis > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> On 29/05/14 12:18, "Andrija Panic" <andrija.pa...@gmail.com> > >> >>wrote: > >> >> >> >> > >> >> >> >> >They are 2 traffic types on 1 physical net (that is both > >>tagged > >> >>vlan > >> >> >> 500, > >> >> >> >> >and untagged packets travel over same KVM bridge, and over > >>eth1 > >> >>to > >> >> >> outside > >> >> >> >> >world)... > >> >> >> >> > > >> >> >> >> > > >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland <daan.hoogl...@gmail.com> > >> >> wrote: > >> >> >> >> > > >> >> >> >> >> Are these two traffic types in one physical net? or two > >> >>physical > >> >> >>nets > >> >> >> >> >> on the same interface (seems wrong). > >> >> >> >> >> > >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal Reddy Uradi > >> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote: > >> >> >> >> >> > I don't think editing DB table will work. > >> >> >> >> >> > > >> >> >> >> >> > -Jayapal > >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija Panic > >> >> >><andrija.pa...@gmail.com > >> >> >> > > >> >> >> >> >> wrote: > >> >> >> >> >> > > >> >> >> >> >> >> It's like this: > >> >> >> >> >> >> > >> >> >> >> >> >> I have public subnet /24. > >> >> >> >> >> >> > >> >> >> >> >> >> half is dedicated for Guest traffic (vlan 500) and the > >> >>second > >> >> >> half is > >> >> >> >> >> >> dedicated to Public traffic/network (no vlan tags, that > >>is > >> >> >> untagged > >> >> >> >> >> packets) > >> >> >> >> >> >> > >> >> >> >> >> >> Both vlan500 and untagged packets travel over physical > >>eth1 > >> >> >> >> >>interface on > >> >> >> >> >> >> hypervisors and can reach Internet. > >> >> >> >> >> >> > >> >> >> >> >> >> Thanks, > >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland > >> >><daan.hoogl...@gmail.com> > >> >> >> wrote: > >> >> >> >> >> >> > >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM, Andrija Panic < > >> >> >> >> >> andrija.pa...@gmail.com> > >> >> >> >> >> >>> wrote: > >> >> >> >> >> >>>> 500 > >> >> >> >> >> >>> > >> >> >> >> >> >>> > >> >> >> >> >> >>> is 500 the vlan of your guestnetwork or your physical > >> >>network? > >> >> >> You > >> >> >> >> >> >>> wouldn't want to have two nets with vlan 500! > >> >> >> >> >> >>> > >> >> >> >> >> >>> -- > >> >> >> >> >> >>> Daan > >> >> >> >> >> >>> > >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> -- > >> >> >> >> >> >> > >> >> >> >> >> >> Andrija Panić > >> >> >> >> >> >> -------------------------------------- > >> >> >> >> >> >> http://admintweets.com > >> >> >> >> >> >> -------------------------------------- > >> >> >> >> >> > > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> -- > >> >> >> >> >> Daan > >> >> >> >> >> > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> >-- > >> >> >> >> > > >> >> >> >> >Andrija Panić > >> >> >> >> >-------------------------------------- > >> >> >> >> > http://admintweets.com > >> >> >> >> >-------------------------------------- > >> >> >> >> > >> >> >> >> > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > > >> >> >> > Andrija Panić > >> >> >> > -------------------------------------- > >> >> >> > http://admintweets.com > >> >> >> > -------------------------------------- > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Daan > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> >-- > >> >> > > >> >> >Andrija Panić > >> >> >-------------------------------------- > >> >> > http://admintweets.com > >> >> >-------------------------------------- > >> >> > >> >> > >> > > >> > > >> >-- > >> > > >> >Andrija Panić > >> >-------------------------------------- > >> > http://admintweets.com > >> >-------------------------------------- > >> > >> > > > > > >-- > > > >Andrija Panić > >-------------------------------------- > > http://admintweets.com > >-------------------------------------- > > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
