I agree, I actually brought this issue up several months ago when we had
all of the 'untagged' discussions. I believe I pointed out that vlan:// was
now in the db. My impression was that the change was intentional, if you're
telling me you didn't intend to do that then we can change it back.
At this point we have to either write a fix for all 4.3.1+ to change the
database value back if it has vlan:// AND make sure all of the comparisons
still work (though perhaps upgrades like this are catching those points for
us), or stick with the vlan:// in the DB and move forward. Either way is
fine with me, as long as we can make all installations consistent.
On Fri, May 30, 2014 at 10:02 AM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:
> Marcus, I don't think it should be 'vlan://untagged'. If it works as
> you describe this seems a bug. 'untagged' should be
> broadcastdomaintype agnostic, shouldn't it?
> If it should work that way then it is a bug/omission in the db upgrade.
>
> On Fri, May 30, 2014 at 5:57 PM, Marcus <shadow...@gmail.com> wrote:
> > Actually, if you're in the position to play a bit... New deployments
> seem
> > to work, and I believe it's because that broadcastUri is stored in the db
> > in the new format:
> >
> > mysql> select id,vlan_id from vlan where network_id = (select id from
> > networks where traffic_type="Public");
> > +----+-----------------+
> > | id | vlan_id |
> > +----+-----------------+
> > | 1 | vlan://untagged |
> > +----+-----------------+
> > 1 row in set (0.00 sec)
> >
> >
> > I believe 4.2 and earlier would say just 'untagged' there. If you want to
> > attempt changing that value to include vlan:// (if it is in fact
> missing),
> > then restarting everything, that may fix the issue.
> >
> >
> >
> > On Fri, May 30, 2014 at 9:38 AM, Marcus <shadow...@gmail.com> wrote:
> >
> >> I thinnk the commit that caused the change was this or related to it. Is
> >> there any way you could test a fix? Do you need me to build 4.3 RPMs or
> can
> >> I just provide a patch? What works for you?
> >>
> >> commit 53d09c6f1843f04c5f1ab76be9419f5584302d1e
> >> Date: Mon Aug 5 11:52:40 2013 +0200
> >>
> >> uri code per broadcast/isolation type , default is to accept
> anything
> >> as uri , vlan and lswitch need some extra tlc
> >>
> >>
> >>
> >> On Fri, May 30, 2014 at 9:36 AM, Marcus <shadow...@gmail.com> wrote:
> >>
> >>> Note the differences in broadcastUri, here is your plug command:
> >>>
> >>> {
> >>> "com.cloud.agent.api.PlugNicCommand": {
> >>> "nic": {
> >>> "deviceId": 1,
> >>> "networkRateMbps": 99999,
> >>> "defaultNic": true,
> >>> "uuid": "6c782af3-2071-4543-acdc-cb30096e89ff",
> >>> "ip": "46.232.xxx.246",
> >>> "netmask": "255.255.255.0",
> >>> "gateway": "46.232.xxx.1",
> >>> "mac": "06:53:82:00:00:25",
> >>> "broadcastType": "Vlan",
> >>> "type": "Public",
> >>> "broadcastUri": "vlan://untagged",
> >>> "isolationUri": "vlan://untagged",
> >>> "isSecurityGroupEnabled": false,
> >>> "name": "breth1-500"
> >>> },
> >>> "instanceName": "r-801-VM",
> >>> "vmType": "DomainRouter",
> >>> "wait": 0
> >>> }
> >>> }
> >>>
> >>> and here is your ip associate command:
> >>>
> >>> {
> >>> "com.cloud.agent.api.routing.IpAssocVpcCommand": {
> >>> "ipAddresses": [
> >>> {
> >>> "accountId": 11,
> >>> "publicIp": "46.232.xxx.246",
> >>> "sourceNat": true,
> >>> "add": true,
> >>> "oneToOneNat": false,
> >>> "firstIP": false,
> >>> "broadcastUri": "untagged",
> >>> "vlanGateway": "46.232.xxx.1",
> >>> "vlanNetmask": "255.255.255.0",
> >>> "vifMacAddress": "06:53:82:00:00:25",
> >>> "networkRate": 99999,
> >>> "trafficType": "Public",
> >>> "networkName": "breth1-500"
> >>> }
> >>> ],
> >>> "accessDetails": {
> >>> "router.guest.ip": "46.232.xxx.246",
> >>> "zone.network.type": "Advanced",
> >>> "router.name": "r-801-VM",
> >>> "router.ip": "169.254.0.52"
> >>> },
> >>> "wait": 0
> >>> }
> >>> }
> >>>
> >>>
> >>> On Fri, May 30, 2014 at 9:27 AM, Andrija Panic <
> andrija.pa...@gmail.com>
> >>> wrote:
> >>>
> >>>> yes, correct, eth1 is present, and can be started by static IP
> >>>> configuration...
> >>>>
> >>>>
> >>>> On 30 May 2014 17:25, Marcus <shadow...@gmail.com> wrote:
> >>>>
> >>>> > Let me make sure I understand... the 'Plug' of the nic works fine,
> as
> >>>> it
> >>>> > seems you do have an eth1 and can manually assign the IP to get it
> to
> >>>> work?
> >>>> > If that's the case then there's probably not an issue in
> >>>> BridgeVifDriver or
> >>>> > the XML. It is definitely in fetching/matching the eth device here
> >>>> >
> >>>> > vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g
> >>>> > 46.232.xxx.1 -m 24 -n 46.232.xxx.0
> >>>> >
> >>>> >
> >>>> > On Fri, May 30, 2014 at 8:33 AM, Andrija Panic <
> >>>> andrija.pa...@gmail.com>
> >>>> > wrote:
> >>>> >
> >>>> > > Nope, started, did check, it is reported now as Debian 5 VM, but
> >>>> still
> >>>> > > doesn't work...
> >>>> > > Rebooted VPC (destroyed VR, new one created...)
> >>>> > >
> >>>> > > $ virsh dumpxml r-812-VM
> >>>> > > ...
> >>>> > > <description>Debian GNU/Linux 5(64-bit)</description>
> >>>> > > ...
> >>>> > > <os>
> >>>> > > <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
> >>>> > > <boot dev='cdrom'/>
> >>>> > > <boot dev='hd'/>
> >>>> > > </os>
> >>>> > >
> >>>> > > :(
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > On 30 May 2014 16:24, Andrija Panic <andrija.pa...@gmail.com>
> wrote:
> >>>> > >
> >>>> > > > I confirm, the highest is Debian 5 64bit
> >>>> > > >
> >>>> > > > Per docs
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-2-x-to-4-3
> >>>> > > > , you should use Debian 7.0 64bit as OS type for system-kvm-4.3
> >>>> > > template...
> >>>> > > > (same for xen and vmware templatest)
> >>>> > > >
> >>>> > > > Will change now DB to point to debian 5, and let you know.
> >>>> > > >
> >>>> > > > This is guest_os_id: 184 | 2 | NULL |
> >>>> > > > 986a0e98-39d6-11e3-8f93-0025904e4412 | Debian GNU/Linux
> 7(64-bit)
> >>>> > > >
> >>>> > > >
> >>>> > > > On 30 May 2014 15:45, Joris van Lieshout <
> >>>> > > jvanliesh...@schubergphilis.com>
> >>>> > > > wrote:
> >>>> > > >
> >>>> > > >> Andrija,
> >>>> > > >>
> >>>> > > >> The thing is I don't know who the os matching on KVM works.
> There
> >>>> must
> >>>> > > be
> >>>> > > >> a way to list supported os types.
> >>>> > > >>
> >>>> > > >> I also did some queuing on the guest_os_hypervisor table (ACS
> 4.3)
> >>>> > and I
> >>>> > > >> don't see Debian 7 for KVM listed.
> >>>> > > >>
> >>>> > > >> select * from guest_os join guest_os_hypervisor on
> >>>> > > >> guest_os.id=guest_os_hypervisor.guest_os_id where
> >>>> > > >> guest_os_hypervisor.hypervisor_type='KVM' and
> >>>> > > >> guest_os_hypervisor.guest_os_name like '%Debian%';
> >>>> > > >>
> >>>> > > >> What was the guest_os_id you where using? Could you try id 72
> >>>> (Debian
> >>>> > 5
> >>>> > > >> 64-bit)? Adjust both os_type_id in vm_instance and vm_template
> >>>> (where
> >>>> > > >> type='SYSTEM' and hypervisor_type='KVM').
> >>>> > > >>
> >>>> > > >> Kind regards,
> >>>> > > >> Joris van Lieshout
> >>>> > > >>
> >>>> > > >> Schuberg Philis
> >>>> > > >>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >> On 30/05/14 15:33, "Andrija Panic" <andrija.pa...@gmail.com>
> >>>> wrote:
> >>>> > > >>
> >>>> > > >> >Joris,
> >>>> > > >> >
> >>>> > > >> >do you have recommendation on how in particular to try ? I'm
> not
> >>>> sure
> >>>> > > how
> >>>> > > >> >to fix that, except playing with editing systemvm-4.3
> template to
> >>>> > > define
> >>>> > > >> >it
> >>>> > > >> >as another OS type... ?
> >>>> > > >> >
> >>>> > > >> >Thanks again,
> >>>> > > >> >Andrija
> >>>> > > >> >
> >>>> > > >> >
> >>>> > > >> >On 30 May 2014 15:30, Joris van Lieshout <
> >>>> > > >> jvanliesh...@schubergphilis.com>
> >>>> > > >> >wrote:
> >>>> > > >> >
> >>>> > > >> >> I've read back a bit in the code and if you look at
> >>>> > > >> BridgeVifDriver.java
> >>>> > > >> >> (this is where the log message with the nic profile is
> >>>> generated)
> >>>> > you
> >>>> > > >> >>can
> >>>> > > >> >> see that the nic information might be off already once ACS
> >>>> hits the
> >>>> > > >> >> LibvirtVMDef.InterfaceDef plug function. This leads be to
> >>>> believer
> >>>> > > that
> >>>> > > >> >> the HVM/PV OS mismatch issue might still be related. Try
> fixing
> >>>> > that
> >>>> > > >> >> first. At least it will allow us to exclude this from the
> list.
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >> >> Kind regards,
> >>>> > > >> >> Joris van Lieshout
> >>>> > > >> >>
> >>>> > > >> >> Schuberg Philis
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >> >> On 30/05/14 15:26, "Andrija Panic" <andrija.pa...@gmail.com
> >
> >>>> > wrote:
> >>>> > > >> >>
> >>>> > > >> >> >OK, thanks Joris.
> >>>> > > >> >> >
> >>>> > > >> >> >I will try playing with OS version option, on the
> >>>> systemvm-kvm-4.3
> >>>> > > >> >> >template...
> >>>> > > >> >> >
> >>>> > > >> >> >Let me know if I can help with anything more.
> >>>> > > >> >> >
> >>>> > > >> >> >Thanks.
> >>>> > > >> >> >Andrija
> >>>> > > >> >> >
> >>>> > > >> >> >
> >>>> > > >> >> >On 30 May 2014 15:19, Joris van Lieshout
> >>>> > > >> >><jvanliesh...@schubergphilis.com
> >>>> > > >> >> >
> >>>> > > >> >> >wrote:
> >>>> > > >> >> >
> >>>> > > >> >> >> Hi Andrija,
> >>>> > > >> >> >>
> >>>> > > >> >> >> That does sound familiar and in the start xml of KVM you
> >>>> can see
> >>>> > > >> >>"<type
> >>>> > > >> >> >> arch='x86_64' machine='pc'>hvm</type>". I don't know
> KVM+ACS
> >>>> > well
> >>>> > > >> >>enough
> >>>> > > >> >> >> to judge if this is the cause but I thing focusing on
> >>>> getting
> >>>> > the
> >>>> > > VR
> >>>> > > >> >> >> started as PV guest might be worth trying. On the other
> >>>> hand I
> >>>> > do
> >>>> > > >> see
> >>>> > > >> >> >> patchviasocket.pl being executed successfully...
> >>>> > > >> >> >>
> >>>> > > >> >> >> The other thing I see is, and now we're getting into java
> >>>> code,
> >>>> > is
> >>>> > > >> >>this:
> >>>> > > >> >> >>
> >>>> > > >> >> >> 2014-05-30 14:41:01,386{GMT} DEBUG
> >>>> > [kvm.resource.BridgeVifDriver]
> >>>> > > >> >> >> (agentRequest-Handler-3:)
> >>>> > > >> >> >>nic=[Nic:Public-46.232.xxx.246-vlan://untagged]
> >>>> > > >> >> >> 2014-05-30 14:41:01,502{GMT} DEBUG [cloud.agent.Agent]
> >>>> > > >> >> >> (agentRequest-Handler-3:) Processing command:
> >>>> > > >> >> >> com.cloud.agent.api.routing.IpAssocVpcCommand
> >>>> > > >> >> >> 2014-05-30 14:41:01,506{GMT} DEBUG
> >>>> > > >> >> >> [resource.virtualnetwork.VirtualRoutingResource]
> >>>> > > >> >> >>(agentRequest-Handler-3:)
> >>>> > > >> >> >> Executing:
> >>>> > > >> >> >>
> >>>> > /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
> >>>> > > >> >> >> vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c
> >>>> ethnull -g
> >>>> > > >> >> >> 46.232.xxx.1 -m 24 -n 46.232.xxx.0
> >>>> > > >> >> >>
> >>>> > > >> >> >> My suspicion is that somewhere in the translation from
> the
> >>>> nic
> >>>> > > >> >>profile
> >>>> > > >> >> >>to
> >>>> > > >> >> >> the actual route_proxy.sh command ACS failes to find the
> >>>> nic id
> >>>> > > and
> >>>> > > >> >> >> returns null.
> >>>> > > >> >> >>
> >>>> > > >> >> >> Let me dig a bit deeper and see what I can find but this
> is
> >>>> > where
> >>>> > > we
> >>>> > > >> >> >>might
> >>>> > > >> >> >> need some help from someone with knowledge of this pice
> of
> >>>> the
> >>>> > > code.
> >>>> > > >> >>:)
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >> Kind regards,
> >>>> > > >> >> >> Joris van Lieshout
> >>>> > > >> >> >>
> >>>> > > >> >> >> Schuberg Philis
> >>>> > > >> >> >> Boeingavenue 271
> >>>> > > >> >> >> 1119 PD Schiphol-Rijk
> >>>> > > >> >> >> schubergphilis.com
> >>>> > > >> >> >>
> >>>> > > >> >> >> +31 20-7506672
> >>>> > > >> >> >> +31 6-51428188
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >> On 30/05/14 14:49, "Andrija Panic" <
> andrija.pa...@gmail.com
> >>>> >
> >>>> > > wrote:
> >>>> > > >> >> >>
> >>>> > > >> >> >> >Hi Joris,
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >I have turned on DEBUG loging in agent.log on
> cs1.xxx/net
> >>>> host:
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >So, management logs again: http://pastebin.com/F6BRf7Y9
> >>>> > > >> >> >> >Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >Not playing smart, but there is some error:
> >>>> > > >> >> >> >[kvm.resource.KVMGuestOsMapper]
> >>>> > > >> >> >> >(agentRequest-Handler-3:) Can't find the mapping of
> guest
> >>>> os:
> >>>> > > >> Debian
> >>>> > > >> >> >> >GNU/Linux 7(64-bit)
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >Best,
> >>>> > > >> >> >> >Andrija
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >On 30 May 2014 14:26, Joris van Lieshout
> >>>> > > >> >> >><jvanliesh...@schubergphilis.com
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >wrote:
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >> Hi Andrija,
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> Bold formatting does not come trough on the dev list.
> :)
> >>>> > > >> >> >> >> But u might need a bit more info.
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> At a certain point I see this line
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request]
> >>>> > > >> >> >> >> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
> >>>> 1-609104082:
> >>>> > > >> >>Sending
> >>>> > > >> >> >> {
> >>>> > > >> >> >> >> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net),
> Ver:
> >>>> v1,
> >>>> > > >> Flags:
> >>>> > > >> >> >> >>100111,
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM
> >>>> > > >> >>>>>>".
> >>>> > > >> >> >>>>..
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> This is where the information is passed on to the
> agent
> >>>> > > handles.
> >>>> > > >> >>For
> >>>> > > >> >> >>XS
> >>>> > > >> >> >> >> this would initiate an agent handler on the management
> >>>> server
> >>>> > > but
> >>>> > > >> >>for
> >>>> > > >> >> >> >>KVM,
> >>>> > > >> >> >> >> if I remember correctly, it passed the command on to
> the
> >>>> > > >> >>cloudstack
> >>>> > > >> >> >> >>agent
> >>>> > > >> >> >> >> service on the hypervisor.
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> Can you check the cloud service log on the KVM
> hypervisor
> >>>> > > >> >>executing
> >>>> > > >> >> >>the
> >>>> > > >> >> >> >> request? it's this server cs1.xxxxx.net and then
> search
> >>>> top
> >>>> > > down
> >>>> > > >> >>for
> >>>> > > >> >> >> >> 609104082 in the log. See if you can provide the log
> >>>> from the
> >>>> > > >> >>agent
> >>>> > > >> >> >> >> handler thread started by that sequence.
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> Kind regards,
> >>>> > > >> >> >> >> Joris van Lieshout
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> Schuberg Philis
> >>>> > > >> >> >> >> Boeingavenue 271
> >>>> > > >> >> >> >> 1119 PD Schiphol-Rijk
> >>>> > > >> >> >> >> schubergphilis.com
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> +31 20-7506672
> >>>> > > >> >> >> >> +31 6-51428188
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> On 30/05/14 14:08, "Andrija Panic" <
> >>>> andrija.pa...@gmail.com>
> >>>> > > >> >>wrote:
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >> >Hi Joris,
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >here is the management log:
> >>>> http://pastebin.com/zxnKxFhk
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >Interesting parts (to me): in bold
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >2014-05-30 13:56:21,899 DEBUG
> >>>> > > >> >>[o.a.c.s.m.AncientDataMotionStrategy]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync
> >>>> > > inspecting
> >>>> > > >> >>src
> >>>> > > >> >> >> >>type
> >>>> > > >> >> >> >> >TEMPLATE copyAsync inspecting dest type VOLUME
> >>>> > > >> >> >> >> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
> >>>> > 4-1248669612:
> >>>> > > >> >> >>Sending
> >>>> > > >> >> >> >>{
> >>>> > > >> >> >> >> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net),
> >>>> Ver: v1,
> >>>> > > >> >>Flags:
> >>>> > > >> >> >> >> 100011,
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org
> >>>> > > >> >>>>>>>.a
> >>>> > > >> >> >>>>>pa
> >>>> > > >> >> >> >>>ch
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0
> >>>> > > >> >>>>>>>f-
> >>>> > > >> >> >>>>>b0
> >>>> > > >> >> >> >>>b4
> >>>> > > >> >> >> >> >-5e351e7cae55","origUrl":"
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mas
> >>>> > > >> >> >> >>t
> >>>> > > >> >> >> >> >er-kvm.qcow2.bz2
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"Q
> >>>> > > >> >>>>>>>CO
> >>>> > > >> >> >>>>>W2
> >>>> > > >> >> >> >>>",
> >>>> > > >> >> >> >>
> >>>> > >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","
> >>>> > > >> >> >> >> >*hvm":true*
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.clou
> >>>> > > >> >>>>>>>ds
> >>>> > > >> >> >>>>>ta
> >>>> > > >> >> >> >>>ck
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-220
> >>>> > > >> >>>>>>>3f
> >>>> > > >> >> >>>>>79
> >>>> > > >> >> >> >>>b1
> >>>> > > >> >> >> >> >dc6","id":209,"poolType":"RBD","host":"
> >>>> > > >> >> >> >> >cephmon.xxxxx.net
> >>>> > > >> ","path":"cloudstack","port":6789,"url":"RBD://
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8
> >>>> > > >> >> >> >>8
> >>>> > > >> >> >> >> >a8-2203f79b1dc6
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorT
> >>>> > > >> >>>>>>>yp
> >>>> > > >> >> >>>>>e"
> >>>> > > >> >> >> >>>:"
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
> >>>> > > >> >>>>>>>"u
> >>>> > > >> >> >>>>>ui
> >>>> > > >> >> >> >>>d"
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataSto
> >>>> > > >> >>>>>>>re
> >>>> > > >> >> >>>>>":
> >>>> > > >> >> >> >>>{"
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b9342
> >>>> > > >> >>>>>>>2e
> >>>> > > >> >> >>>>>-1
> >>>> > > >> >> >> >>>a6
> >>>> > > >> >> >> >>
> >>>> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
> >>>> > > >> >> >> >> >cephmon.xxxxx.net
> >>>> > > >> ","path":"cloudstack","port":6789,"url":"RBD://
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8
> >>>> > > >> >> >> >>8
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":10
> >>>> > > >> >>>>>>>64
> >>>> > > >> >> >>>>>,"
> >>>> > > >> >> >> >>>vm
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":
> >>>> > > >> >>>>>>>0,
> >>>> > > >> >> >>>>>"h
> >>>> > > >> >> >> >>>yp
> >>>> > > >> >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}
> >>>> > > >> >>>>>]
> >>>> > > >> >> >> >> >}
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
> >>>> > > >> >> >> >> >(AgentManager-Handler-12:null) Seq 4-1248669612:
> >>>> Processing:
> >>>> > > {
> >>>> > > >> >> >>Ans: ,
> >>>> > > >> >> >> >> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10,
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
> >>>> > > >> >>>>>>>"o
> >>>> > > >> >> >>>>>rg
> >>>> > > >> >> >> >>>.a
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path
> >>>> > > >> >>>>>>>":
> >>>> > > >> >> >>>>>"9
> >>>> > > >> >> >> >>>c4
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id"
> >>>> > > >> >>>>>>>:0
> >>>> > > >> >> >>>>>}}
> >>>> > > >> >> >> >>>,"
> >>>> > > >> >> >> >> >result":true,"wait":0}}]
> >>>> > > >> >> >> >> >}
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
> >>>> > 4-1248669612:
> >>>> > > >> >> >> >>Received: {
> >>>> > > >> >> >> >> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags:
> >>>> 10, {
> >>>> > > >> >> >> >>CopyCmdAnswer
> >>>> > > >> >> >> >> >} }
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
> >>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
> nic
> >>>> > > >> >>NicProfile[
> >>>> > > >> >> >> >> >*1092-801-null*-46.232.xxx.246-vlan://untagged of
> type
> >>>> > Public
> >>>> > > >> >>from
> >>>> > > >> >> >>the
> >>>> > > >> >> >> >> >nics
> >>>> > > >> >> >> >> >passed on vm start. The nic will be plugged later
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
> >>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
> nic
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vl
> >>>> > > >> >>>>>>>an
> >>>> > > >> >> >>>>>:/
> >>>> > > >> >> >> >>>/4
> >>>> > > >> >> >> >> >4
> >>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
> nic
> >>>> will
> >>>> > > be
> >>>> > > >> >> >>plugged
> >>>> > > >> >> >> >> >later
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
> >>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
> nic
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vl
> >>>> > > >> >>>>>>>an
> >>>> > > >> >> >>>>>:/
> >>>> > > >> >> >> >>>/4
> >>>> > > >> >> >> >> >3
> >>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
> nic
> >>>> will
> >>>> > > be
> >>>> > > >> >> >>plugged
> >>>> > > >> >> >> >> >later
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
> >>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
> nic
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl
> >>>> > > >> >>>>>>>an
> >>>> > > >> >> >>>>>:/
> >>>> > > >> >> >> >>>/3
> >>>> > > >> >> >> >> >004
> >>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
> nic
> >>>> will
> >>>> > > be
> >>>> > > >> >> >>plugged
> >>>> > > >> >> >> >> >later
> >>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
> >>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
> nic
> >>>> > > >> >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >>
> >>>> > > >>
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl
> >>>> > > >> >>>>>>>an
> >>>> > > >> >> >>>>>:/
> >>>> > > >> >> >> >>>/3
> >>>> > > >> >> >> >> >004
> >>>> > > >> >> >> >> >of type Guest from the nics passed on vm start
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >Thanks,
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >On 30 May 2014 13:54, Joris van Lieshout
> >>>> > > >> >> >> >><jvanliesh...@schubergphilis.com
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >wrote:
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >> Hi Andrija,
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >> Just the start of the VR should be sufficient.
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >> Kind regards,
> >>>> > > >> >> >> >> >> Joris van Lieshout
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >> Schuberg Philis
> >>>> > > >> >> >> >> >> Boeingavenue 271
> >>>> > > >> >> >> >> >> 1119 PD Schiphol-Rijk
> >>>> > > >> >> >> >> >> schubergphilis.com
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >> +31 20-7506672
> >>>> > > >> >> >> >> >> +31 6-51428188
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >> On 30/05/14 13:48, "Andrija Panic" <
> >>>> > andrija.pa...@gmail.com
> >>>> > > >
> >>>> > > >> >> >>wrote:
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >> >Hi Joris,
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >just to be sure - you want me to capture the log
> >>>> from the
> >>>> > > >> >>moment
> >>>> > > >> >> >>I
> >>>> > > >> >> >> >> >>reboot
> >>>> > > >> >> >> >> >> >router - or you want me to stop it, then start
> >>>> capturing
> >>>> > > log,
> >>>> > > >> >>and
> >>>> > > >> >> >> >> >>start it
> >>>> > > >> >> >> >> >> >(and continue capture untill ethnull errors inside
> >>>> VR) ?
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >Thanks,
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >On 30 May 2014 13:39, Joris van Lieshout
> >>>> > > >> >> >> >> >><jvanliesh...@schubergphilis.com
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >wrote:
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> Hi Andrija,
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> Thanks for the answers. In deed your situation
> is
> >>>> > > different
> >>>> > > >> >>so
> >>>> > > >> >> >> >> >>PV/HVM is
> >>>> > > >> >> >> >> >> >> not the issue.
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> When reading back the log output you have
> provided
> >>>> I
> >>>> > > noted
> >>>> > > >> >>that
> >>>> > > >> >> >> >>the
> >>>> > > >> >> >> >> >>VR
> >>>> > > >> >> >> >> >> >> messages log indicates that it's waiting for
> >>>> ethnull to
> >>>> > > be
> >>>> > > >> >>up.
> >>>> > > >> >> >> >>This
> >>>> > > >> >> >> >> >> >>raises
> >>>> > > >> >> >> >> >> >> the question where null was introduced instead
> of
> >>>> 1.
> >>>> > The
> >>>> > > >> ACS
> >>>> > > >> >> >> >> >>management
> >>>> > > >> >> >> >> >> >> log output you send was, what I think, later
> down
> >>>> the
> >>>> > > road
> >>>> > > >> >> >>where
> >>>> > > >> >> >> >>ACS
> >>>> > > >> >> >> >> >> >>gives
> >>>> > > >> >> >> >> >> >> up trying to wait for the VR to come up. If you
> >>>> would
> >>>> > > >> >>capture
> >>>> > > >> >> >>the
> >>>> > > >> >> >> >> >> >> job-executor in the management log from
> >>>> startCommand
> >>>> > till
> >>>> > > >> >>the
> >>>> > > >> >> >> >> >>exception,
> >>>> > > >> >> >> >> >> >> do you see anywhere a mention of ethnull? You
> might
> >>>> > need
> >>>> > > to
> >>>> > > >> >> >>reed
> >>>> > > >> >> >> >>into
> >>>> > > >> >> >> >> >> >>the
> >>>> > > >> >> >> >> >> >> DirectAgent executing the startCommand to find a
> >>>> clue.
> >>>> > > The
> >>>> > > >> >> >>thing
> >>>> > > >> >> >> >>is
> >>>> > > >> >> >> >> >> >>that I
> >>>> > > >> >> >> >> >> >> only have experience with XS based environment
> so I
> >>>> > > cannot
> >>>> > > >> >> >>point
> >>>> > > >> >> >> >>you
> >>>> > > >> >> >> >> >>to
> >>>> > > >> >> >> >> >> >> the exact output to look for. On XS, at least,
> it
> >>>> is
> >>>> > > >> >> >> >> >> >> "[c.c.h.x.r.CitrixResourceBase]
> >>>> > > >> >>(DirectAgent-351:ctx-4a51bb9e)
> >>>> > > >> >> >> >> >>Created a
> >>>> > > >> >> >> >> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1"
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> Kind regards,
> >>>> > > >> >> >> >> >> >> Joris van Lieshout
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> Schuberg Philis
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> On 30/05/14 10:48, "Andrija Panic" <
> >>>> > > >> andrija.pa...@gmail.com>
> >>>> > > >> >> >> >>wrote:
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >Hi Deen,
> >>>> > > >> >> >> >> >> >> >no, in DB there is field "vlan_id" with value
> >>>> > > "untagged" -
> >>>> > > >> >> >>that
> >>>> > > >> >> >> >> >> >> >"vlan://untagged" is shown from ACS gui, and is
> >>>> used
> >>>> > in
> >>>> > > >> API
> >>>> > > >> >> >>call
> >>>> > > >> >> >> >>(or
> >>>> > > >> >> >> >> >> >> >better
> >>>> > > >> >> >> >> >> >> >said commands that are seen in management
> server
> >>>> > logs).
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >Best,
> >>>> > > >> >> >> >> >> >> >Andrija
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >On 30 May 2014 10:37, Daan Hoogland
> >>>> > > >> >><daan.hoogl...@gmail.com>
> >>>> > > >> >> >> >> wrote:
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> Andrija,
> >>>> > > >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> Do not just assign a second net vlan://500
> You
> >>>> have
> >>>> > > one
> >>>> > > >> >>like
> >>>> > > >> >> >> >>that
> >>>> > > >> >> >> >> >>and
> >>>> > > >> >> >> >> >> >> >> you don't want conflicting nets using the
> same
> >>>> > vlan. I
> >>>> > > >> am
> >>>> > > >> >> >> >> >>wondering
> >>>> > > >> >> >> >> >> >> >> why 'untagged' comes out as
> 'vlan://untagged'. I
> >>>> > think
> >>>> > > >> >>that
> >>>> > > >> >> >>is
> >>>> > > >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >> bug. Did you find the string
> 'vlan://untagged'
> >>>> in
> >>>> > your
> >>>> > > >> >>db?
> >>>> > > >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija
> Panic
> >>>> > > >> >> >> >> >> >> >><andrija.pa...@gmail.com>
> >>>> > > >> >> >> >> >> >> >> wrote:
> >>>> > > >> >> >> >> >> >> >> > Hi Joris,
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > thank you for taking time to address this
> >>>> issue :)
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > So...:
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by
> >>>> Inktank
> >>>> > > for
> >>>> > > >> >>CEPH
> >>>> > > >> >> >> >> >> >>support),
> >>>> > > >> >> >> >> >> >> >>OS
> >>>> > > >> >> >> >> >> >> >> is
> >>>> > > >> >> >> >> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled.
> >>>> > > >> >> >> >> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was
> fine
> >>>> > > >> >> >> >> >> >> >> > - not XS, so I guess no answers for this
> part
> >>>> :)
> >>>> > > >> >> >> >> >> >> >> > - guest_os_id is 184 = Debian 7 x64
> >>>> > > >> >> >> >> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 =
> >>>> Debian 7
> >>>> > > x64
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > This worked previously on 4.2.1 = template
> was
> >>>> > > >> ofcourse
> >>>> > > >> >> >> >> >> >> >>systemvm-kvm-4.2
> >>>> > > >> >> >> >> >> >> >> -
> >>>> > > >> >> >> >> >> >> >> > but that was also Debian 7 x64 type... so
> this
> >>>> > > should
> >>>> > > >> >>not
> >>>> > > >> >> >>be
> >>>> > > >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >>issues
> >>>> > > >> >> >> >> >> >> >> > (guest not supported by host...)
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > The only thing that might be out of
> >>>> "standard" =
> >>>> > all
> >>>> > > >> >>SVMs
> >>>> > > >> >> >> >>are on
> >>>> > > >> >> >> >> >> >>CEPH
> >>>> > > >> >> >> >> >> >> >>-
> >>>> > > >> >> >> >> >> >> >> > there are official docs on altering
> database
> >>>> to
> >>>> > make
> >>>> > > >> >>some
> >>>> > > >> >> >>new
> >>>> > > >> >> >> >> >> >>System
> >>>> > > >> >> >> >> >> >> >> > Offering as default for SSVM and CPVM -
> what I
> >>>> > did,
> >>>> > > I
> >>>> > > >> >>also
> >>>> > > >> >> >> >>have
> >>>> > > >> >> >> >> >> >>done
> >>>> > > >> >> >> >> >> >> >>same
> >>>> > > >> >> >> >> >> >> >> > config in DB, to make VR use another System
> >>>> > Offering
> >>>> > > >> as
> >>>> > > >> >> >> >>default
> >>>> > > >> >> >> >> >>-
> >>>> > > >> >> >> >> >> >> >>which
> >>>> > > >> >> >> >> >> >> >> is
> >>>> > > >> >> >> >> >> >> >> > NOT explained in the docs - you could use
> >>>> "Change
> >>>> > > >> >> >> >>Offering..."
> >>>> > > >> >> >> >> >> >>button
> >>>> > > >> >> >> >> >> >> >>on
> >>>> > > >> >> >> >> >> >> >> > exiting, shutdown VR to change it per
> docs...
> >>>> > > >> >> >> >> >> >> >> > But still this worked all fine on 4.2.1...
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > - regarding /var/cache/cloud/cmdline the
> >>>> content
> >>>> > is
> >>>> > > >> >> >>folowing
> >>>> > > >> >> >> >>at
> >>>> > > >> >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >> moment
> >>>> > > >> >> >> >> >> >> >> > root@r-801-VM:~# cat
> /var/cache/cloud/cmdline
> >>>> > > >> >> >> >> >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal
> >>>> > > >> dns1=8.8.8.8
> >>>> > > >> >> >> dns2=
> >>>> > > >> >> >> >> >> >> >> template=domP
> >>>> > > >> >> >> >> >> >> >> > name=r-801-VM eth0ip=169.254.0.75
> >>>> > > eth0mask=255.255.0.0
> >>>> > > >> >> >> >> >> >>type=vpcrouter
> >>>> > > >> >> >> >> >> >> >> > disable_rp_filter=true
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > Also please note that only eth1 does not
> have
> >>>> IP
> >>>> > > info,
> >>>> > > >> >> >>eth0
> >>>> > > >> >> >> >> >> >>(control
> >>>> > > >> >> >> >> >> >> >> > 169.xxx) and all other eh2 and up that are
> >>>> used
> >>>> > for
> >>>> > > >> >>Tiers
> >>>> > > >> >> >> >>get IP
> >>>> > > >> >> >> >> >> >>info
> >>>> > > >> >> >> >> >> >> >> fine.
> >>>> > > >> >> >> >> >> >> >> > I could also manually add IP for eth1
> (public
> >>>> NIC)
> >>>> > > and
> >>>> > > >> >> >>start
> >>>> > > >> >> >> >> >>ifup
> >>>> > > >> >> >> >> >> >> >>eth1 -
> >>>> > > >> >> >> >> >> >> >> > and it works fine, but adding new IP Port
> >>>> > Forwarding
> >>>> > > >> >>etc
> >>>> > > >> >> >>does
> >>>> > > >> >> >> >> >>not
> >>>> > > >> >> >> >> >> >> >>work...
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > Daan or somebody said it could be realted
> to
> >>>> my
> >>>> > > >> >>"Public"
> >>>> > > >> >> >> >>network
> >>>> > > >> >> >> >> >> >>(in
> >>>> > > >> >> >> >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >> > Zones, Physical Network, eth1 listing) is
> NOT
> >>>> > tagged
> >>>> > > >> >> >> >> >> >> >>(vlan://untagged)...
> >>>> > > >> >> >> >> >> >> >> > Interestingly the only VR that does work
> fine
> >>>> is
> >>>> > the
> >>>> > > >> VR
> >>>> > > >> >> >>used
> >>>> > > >> >> >> >>in
> >>>> > > >> >> >> >> >> >>Shared
> >>>> > > >> >> >> >> >> >> >> > network, but that VR is using IP from
> Guest IP
> >>>> > range
> >>>> > > >> >>(also
> >>>> > > >> >> >> >> >> >>efectively
> >>>> > > >> >> >> >> >> >> >> > public IPs but on vlan 500)
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > I was instructed to try to change Public IP
> >>>> range
> >>>> > > from
> >>>> > > >> >> >> >>untagged
> >>>> > > >> >> >> >> >>to
> >>>> > > >> >> >> >> >> >> >>vlan
> >>>> > > >> >> >> >> >> >> >> > 500, but I'm not sure how to do this, if
> >>>> there is
> >>>> > > any
> >>>> > > >> >>way
> >>>> > > >> >> >>at
> >>>> > > >> >> >> >>all
> >>>> > > >> >> >> >> >> >> >>(editing
> >>>> > > >> >> >> >> >> >> >> > "vlan" table and changing to vlan 500 does
> not
> >>>> > work,
> >>>> > > >> >>after
> >>>> > > >> >> >> >> >> >>rebooting
> >>>> > > >> >> >> >> >> >> >>VR
> >>>> > > >> >> >> >> >> >> >> > from ACS gui).
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > :)
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > So, not sure what is roughly expected date
> for
> >>>> > 4.4,
> >>>> > > >> but
> >>>> > > >> >> >>right
> >>>> > > >> >> >> >> >>now,
> >>>> > > >> >> >> >> >> >>I'm
> >>>> > > >> >> >> >> >> >> >> > pretty stuck with a big problem of all VPC
> not
> >>>> > > >> >> >>operational at
> >>>> > > >> >> >> >> >> >>all...
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > Thanks,
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout <
> >>>> > > >> >> >> >> >> >> >> jvanliesh...@schubergphilis.com>
> >>>> > > >> >> >> >> >> >> >> > wrote:
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> Hi Andrija,
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> Daan asked me to have a look at this as
> well.
> >>>> > > Looking
> >>>> > > >> >>at
> >>>> > > >> >> >>you
> >>>> > > >> >> >> >> >> >>issue I
> >>>> > > >> >> >> >> >> >> >> >> recall having seen something similar. Back
> >>>> then
> >>>> > > when
> >>>> > > >> >> >> >>upgrading
> >>>> > > >> >> >> >> >> >>4.2.1
> >>>> > > >> >> >> >> >> >> >>to
> >>>> > > >> >> >> >> >> >> >> >> 4.3 I though it had to do with out own
> custom
> >>>> > build
> >>>> > > >> >>svm
> >>>> > > >> >> >> >> >>template.
> >>>> > > >> >> >> >> >> >> >> >> Let me fire off some questions before
> >>>> explaining
> >>>> > > what
> >>>> > > >> >>the
> >>>> > > >> >> >> >>cause
> >>>> > > >> >> >> >> >> >>was
> >>>> > > >> >> >> >> >> >> >>in
> >>>> > > >> >> >> >> >> >> >> our
> >>>> > > >> >> >> >> >> >> >> >> case. :)
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> - what hypervisor (and version) are you
> >>>> using?
> >>>> > > >> >> >> >> >> >> >> >> - if XS, is the new VR a para-virtualised
> >>>> > instance
> >>>> > > >> >>(PV)
> >>>> > > >> >> >>or
> >>>> > > >> >> >> >> >> >>hardware
> >>>> > > >> >> >> >> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on
> >>>> the VR
> >>>> > > >> uuid
> >>>> > > >> >> >>and
> >>>> > > >> >> >> >> >>check
> >>>> > > >> >> >> >> >> >>that
> >>>> > > >> >> >> >> >> >> >> >> param PV-args is set and HVM-boot-policy
> is
> >>>> > unset.
> >>>> > > >> >> >> >> >> >> >> >> - what is the OS type of the VR in ACS
> >>>> > (guest_os_id
> >>>> > > >> in
> >>>> > > >> >> >> >> >>vm_instance
> >>>> > > >> >> >> >> >> >> >>table
> >>>> > > >> >> >> >> >> >> >> >> and match with table guest_os)
> >>>> > > >> >> >> >> >> >> >> >> - what is the OS type of the SVM template?
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> Now for the explaining. :)
> >>>> > > >> >> >> >> >> >> >> >> In our case the OS type of the new
> template
> >>>> was
> >>>> > not
> >>>> > > >> >> >> >>supported
> >>>> > > >> >> >> >> >>on
> >>>> > > >> >> >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >> >> XenServer version we are running.
> Therefore
> >>>> the
> >>>> > VR
> >>>> > > >> was
> >>>> > > >> >> >> >>started
> >>>> > > >> >> >> >> >>by
> >>>> > > >> >> >> >> >> >>XS
> >>>> > > >> >> >> >> >> >> >>as
> >>>> > > >> >> >> >> >> >> >> a
> >>>> > > >> >> >> >> >> >> >> >> HVM guest. System vms on XS rely on the
> >>>> arguments
> >>>> > > >> >>passed
> >>>> > > >> >> >>to
> >>>> > > >> >> >> >> >>them
> >>>> > > >> >> >> >> >> >>in
> >>>> > > >> >> >> >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >> >> PV-args param (ends up on the guest in
> >>>> > > >> >> >> >>/var/cache/cloud/cmdline
> >>>> > > >> >> >> >> >> >> >>which in
> >>>> > > >> >> >> >> >> >> >> >> turn is used by cloud-early-config) in
> order
> >>>> to
> >>>> > > work.
> >>>> > > >> >> >> >>cmdline
> >>>> > > >> >> >> >> >> >> >>contains
> >>>> > > >> >> >> >> >> >> >> the
> >>>> > > >> >> >> >> >> >> >> >> NIC configuration information.
> >>>> > > >> >> >> >> >> >> >> >> So, long story short, if a VR gets started
> >>>> as a
> >>>> > HVM
> >>>> > > >> it
> >>>> > > >> >> >>will
> >>>> > > >> >> >> >>not
> >>>> > > >> >> >> >> >> >>get
> >>>> > > >> >> >> >> >> >> >>the
> >>>> > > >> >> >> >> >> >> >> >> information needed to configure it's NICs.
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> Workaround
> >>>> > > >> >> >> >> >> >> >> >> We corrected the os_type_id in the DB
> (yes I
> >>>> know
> >>>> > > >> >>editing
> >>>> > > >> >> >> >>the
> >>>> > > >> >> >> >> >>DB
> >>>> > > >> >> >> >> >> >>is
> >>>> > > >> >> >> >> >> >> >> >> something you usually don't want but there
> >>>> is no
> >>>> > > >> other
> >>>> > > >> >> >>way
> >>>> > > >> >> >> >>in
> >>>> > > >> >> >> >> >>this
> >>>> > > >> >> >> >> >> >> >>case)
> >>>> > > >> >> >> >> >> >> >> >> of the existing VR's and of the
> >>>> systemvmtemplate
> >>>> > to
> >>>> > > >> >> >> >>something
> >>>> > > >> >> >> >> >> >> >>supported
> >>>> > > >> >> >> >> >> >> >> by
> >>>> > > >> >> >> >> >> >> >> >> XenServer.
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> Kind regards,
> >>>> > > >> >> >> >> >> >> >> >> Joris van Lieshout
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> Schuberg Philis
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> On 29/05/14 12:18, "Andrija Panic"
> >>>> > > >> >> >><andrija.pa...@gmail.com
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >> >> >>wrote:
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >They are 2 traffic types on 1 physical
> net
> >>>> (that
> >>>> > > is
> >>>> > > >> >>both
> >>>> > > >> >> >> >> >>tagged
> >>>> > > >> >> >> >> >> >>vlan
> >>>> > > >> >> >> >> >> >> >> 500,
> >>>> > > >> >> >> >> >> >> >> >> >and untagged packets travel over same KVM
> >>>> > bridge,
> >>>> > > >> and
> >>>> > > >> >> >>over
> >>>> > > >> >> >> >> >>eth1
> >>>> > > >> >> >> >> >> >>to
> >>>> > > >> >> >> >> >> >> >> outside
> >>>> > > >> >> >> >> >> >> >> >> >world)...
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland
> >>>> > > >> >> >> >><daan.hoogl...@gmail.com>
> >>>> > > >> >> >> >> >> >> wrote:
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >> Are these two traffic types in one
> >>>> physical
> >>>> > net?
> >>>> > > >> or
> >>>> > > >> >> >>two
> >>>> > > >> >> >> >> >> >>physical
> >>>> > > >> >> >> >> >> >> >>nets
> >>>> > > >> >> >> >> >> >> >> >> >> on the same interface (seems wrong).
> >>>> > > >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM,
> Jayapal
> >>>> > Reddy
> >>>> > > >> >>Uradi
> >>>> > > >> >> >> >> >> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote:
> >>>> > > >> >> >> >> >> >> >> >> >> > I don't think editing DB table will
> >>>> work.
> >>>> > > >> >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >> > -Jayapal
> >>>> > > >> >> >> >> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija
> >>>> Panic
> >>>> > > >> >> >> >> >> >> >><andrija.pa...@gmail.com
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >> wrote:
> >>>> > > >> >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >> >> It's like this:
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> I have public subnet /24.
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> half is dedicated for Guest traffic
> >>>> (vlan
> >>>> > > 500)
> >>>> > > >> >>and
> >>>> > > >> >> >>the
> >>>> > > >> >> >> >> >> >>second
> >>>> > > >> >> >> >> >> >> >> half is
> >>>> > > >> >> >> >> >> >> >> >> >> >> dedicated to Public traffic/network
> (no
> >>>> > vlan
> >>>> > > >> >>tags,
> >>>> > > >> >> >> >>that
> >>>> > > >> >> >> >> >>is
> >>>> > > >> >> >> >> >> >> >> untagged
> >>>> > > >> >> >> >> >> >> >> >> >> packets)
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> Both vlan500 and untagged packets
> >>>> travel
> >>>> > over
> >>>> > > >> >> >>physical
> >>>> > > >> >> >> >> >>eth1
> >>>> > > >> >> >> >> >> >> >> >> >>interface on
> >>>> > > >> >> >> >> >> >> >> >> >> >> hypervisors and can reach Internet.
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> Thanks,
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland
> >>>> > > >> >> >> >> >> >><daan.hoogl...@gmail.com>
> >>>> > > >> >> >> >> >> >> >> wrote:
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM,
> >>>> Andrija
> >>>> > > >> >>Panic <
> >>>> > > >> >> >> >> >> >> >> >> >> andrija.pa...@gmail.com>
> >>>> > > >> >> >> >> >> >> >> >> >> >>> wrote:
> >>>> > > >> >> >> >> >> >> >> >> >> >>>> 500
> >>>> > > >> >> >> >> >> >> >> >> >> >>>
> >>>> > > >> >> >> >> >> >> >> >> >> >>>
> >>>> > > >> >> >> >> >> >> >> >> >> >>> is 500 the vlan of your
> guestnetwork
> >>>> or
> >>>> > your
> >>>> > > >> >> >>physical
> >>>> > > >> >> >> >> >> >>network?
> >>>> > > >> >> >> >> >> >> >> You
> >>>> > > >> >> >> >> >> >> >> >> >> >>> wouldn't want to have two nets with
> >>>> vlan
> >>>> > > 500!
> >>>> > > >> >> >> >> >> >> >> >> >> >>>
> >>>> > > >> >> >> >> >> >> >> >> >> >>> --
> >>>> > > >> >> >> >> >> >> >> >> >> >>> Daan
> >>>> > > >> >> >> >> >> >> >> >> >> >>>
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> --
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> >> Andrija Panić
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> --------------------------------------
> >>>> > > >> >> >> >> >> >> >> >> >> >> http://admintweets.com
> >>>> > > >> >> >> >> >> >> >> >> >> >>
> --------------------------------------
> >>>> > > >> >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >> --
> >>>> > > >> >> >> >> >> >> >> >> >> Daan
> >>>> > > >> >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >--
> >>>> > > >> >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >> >Andrija Panić
> >>>> > > >> >> >> >> >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> >> >> >> >> > http://admintweets.com
> >>>> > > >> >> >> >> >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > --
> >>>> > > >> >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >> > Andrija Panić
> >>>> > > >> >> >> >> >> >> >> > --------------------------------------
> >>>> > > >> >> >> >> >> >> >> > http://admintweets.com
> >>>> > > >> >> >> >> >> >> >> > --------------------------------------
> >>>> > > >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >> --
> >>>> > > >> >> >> >> >> >> >> Daan
> >>>> > > >> >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >--
> >>>> > > >> >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >> >Andrija Panić
> >>>> > > >> >> >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> >> >> > http://admintweets.com
> >>>> > > >> >> >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >>
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >--
> >>>> > > >> >> >> >> >> >
> >>>> > > >> >> >> >> >> >Andrija Panić
> >>>> > > >> >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> >> > http://admintweets.com
> >>>> > > >> >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >>
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >--
> >>>> > > >> >> >> >> >
> >>>> > > >> >> >> >> >Andrija Panić
> >>>> > > >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >> > http://admintweets.com
> >>>> > > >> >> >> >> >--------------------------------------
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >>
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >--
> >>>> > > >> >> >> >
> >>>> > > >> >> >> >Andrija Panić
> >>>> > > >> >> >> >--------------------------------------
> >>>> > > >> >> >> > http://admintweets.com
> >>>> > > >> >> >> >--------------------------------------
> >>>> > > >> >> >>
> >>>> > > >> >> >>
> >>>> > > >> >> >
> >>>> > > >> >> >
> >>>> > > >> >> >--
> >>>> > > >> >> >
> >>>> > > >> >> >Andrija Panić
> >>>> > > >> >> >--------------------------------------
> >>>> > > >> >> > http://admintweets.com
> >>>> > > >> >> >--------------------------------------
> >>>> > > >> >>
> >>>> > > >> >>
> >>>> > > >> >
> >>>> > > >> >
> >>>> > > >> >--
> >>>> > > >> >
> >>>> > > >> >Andrija Panić
> >>>> > > >> >--------------------------------------
> >>>> > > >> > http://admintweets.com
> >>>> > > >> >--------------------------------------
> >>>> > > >>
> >>>> > > >>
> >>>> > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > >
> >>>> > > > Andrija Panić
> >>>> > > > --------------------------------------
> >>>> > > > http://admintweets.com
> >>>> > > > --------------------------------------
> >>>> > > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > >
> >>>> > > Andrija Panić
> >>>> > > --------------------------------------
> >>>> > > http://admintweets.com
> >>>> > > --------------------------------------
> >>>> > >
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Andrija Panić
> >>>> --------------------------------------
> >>>> http://admintweets.com
> >>>> --------------------------------------
> >>>>
> >>>
> >>>
> >>
>
>
>
> --
> Daan
>
