CLOUDSTACK-5505 looks alright. As for the solution; Isn't 'untagged' a
valid uri in itself? I would expect it would have always the value
without the 'vlan://'. That said a solution is better then no
solution, maybe the db upgrade path is best.
Andrija, can you try as Marcus suggests, editing the db to change
'untagged' to 'vlan://untagged'?
thanks,
Daan
On Fri, May 30, 2014 at 6:03 PM, Marcus <shadow...@gmail.com> wrote:
> If that works, then I think the fix is better to update the database
> upgrade script to look for this and change the vlan_id. That way new
> installs and upgrades have consistent data. We could fix it in the code by
> filtering it through BroadcastDomainType.Vlan.toUri(ipAddr.getVlanTag()),
> but then later when someone expects the vlan id to be in URI format
> somewhere else it will randomly break on people who did an upgrade. The fix
> to CLOUDSTACK-5505 did this, avoiding the mismatch in format in the DB by
> fetching the data from elsewhere that filtered it into URI format, and then
> we just ended up hitting it again here.
>
>
> On Fri, May 30, 2014 at 9:57 AM, Marcus <shadow...@gmail.com> wrote:
>
>> Actually, if you're in the position to play a bit... New deployments seem
>> to work, and I believe it's because that broadcastUri is stored in the db
>> in the new format:
>>
>> mysql> select id,vlan_id from vlan where network_id = (select id from
>> networks where traffic_type="Public");
>> +----+-----------------+
>> | id | vlan_id |
>> +----+-----------------+
>> | 1 | vlan://untagged |
>> +----+-----------------+
>> 1 row in set (0.00 sec)
>>
>>
>> I believe 4.2 and earlier would say just 'untagged' there. If you want to
>> attempt changing that value to include vlan:// (if it is in fact missing),
>> then restarting everything, that may fix the issue.
>>
>>
>>
>> On Fri, May 30, 2014 at 9:38 AM, Marcus <shadow...@gmail.com> wrote:
>>
>>> I thinnk the commit that caused the change was this or related to it. Is
>>> there any way you could test a fix? Do you need me to build 4.3 RPMs or can
>>> I just provide a patch? What works for you?
>>>
>>> commit 53d09c6f1843f04c5f1ab76be9419f5584302d1e
>>> Date: Mon Aug 5 11:52:40 2013 +0200
>>>
>>> uri code per broadcast/isolation type , default is to accept anything
>>> as uri , vlan and lswitch need some extra tlc
>>>
>>>
>>>
>>> On Fri, May 30, 2014 at 9:36 AM, Marcus <shadow...@gmail.com> wrote:
>>>
>>>> Note the differences in broadcastUri, here is your plug command:
>>>>
>>>> {
>>>> "com.cloud.agent.api.PlugNicCommand": {
>>>> "nic": {
>>>> "deviceId": 1,
>>>> "networkRateMbps": 99999,
>>>> "defaultNic": true,
>>>> "uuid": "6c782af3-2071-4543-acdc-cb30096e89ff",
>>>> "ip": "46.232.xxx.246",
>>>> "netmask": "255.255.255.0",
>>>> "gateway": "46.232.xxx.1",
>>>> "mac": "06:53:82:00:00:25",
>>>> "broadcastType": "Vlan",
>>>> "type": "Public",
>>>> "broadcastUri": "vlan://untagged",
>>>> "isolationUri": "vlan://untagged",
>>>> "isSecurityGroupEnabled": false,
>>>> "name": "breth1-500"
>>>> },
>>>> "instanceName": "r-801-VM",
>>>> "vmType": "DomainRouter",
>>>> "wait": 0
>>>> }
>>>> }
>>>>
>>>> and here is your ip associate command:
>>>>
>>>> {
>>>> "com.cloud.agent.api.routing.IpAssocVpcCommand": {
>>>> "ipAddresses": [
>>>> {
>>>> "accountId": 11,
>>>> "publicIp": "46.232.xxx.246",
>>>> "sourceNat": true,
>>>> "add": true,
>>>> "oneToOneNat": false,
>>>> "firstIP": false,
>>>> "broadcastUri": "untagged",
>>>> "vlanGateway": "46.232.xxx.1",
>>>> "vlanNetmask": "255.255.255.0",
>>>> "vifMacAddress": "06:53:82:00:00:25",
>>>> "networkRate": 99999,
>>>> "trafficType": "Public",
>>>> "networkName": "breth1-500"
>>>> }
>>>> ],
>>>> "accessDetails": {
>>>> "router.guest.ip": "46.232.xxx.246",
>>>> "zone.network.type": "Advanced",
>>>> "router.name": "r-801-VM",
>>>> "router.ip": "169.254.0.52"
>>>> },
>>>> "wait": 0
>>>> }
>>>> }
>>>>
>>>>
>>>> On Fri, May 30, 2014 at 9:27 AM, Andrija Panic <andrija.pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> yes, correct, eth1 is present, and can be started by static IP
>>>>> configuration...
>>>>>
>>>>>
>>>>> On 30 May 2014 17:25, Marcus <shadow...@gmail.com> wrote:
>>>>>
>>>>> > Let me make sure I understand... the 'Plug' of the nic works fine, as
>>>>> it
>>>>> > seems you do have an eth1 and can manually assign the IP to get it to
>>>>> work?
>>>>> > If that's the case then there's probably not an issue in
>>>>> BridgeVifDriver or
>>>>> > the XML. It is definitely in fetching/matching the eth device here
>>>>> >
>>>>> > vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c ethnull -g
>>>>> > 46.232.xxx.1 -m 24 -n 46.232.xxx.0
>>>>> >
>>>>> >
>>>>> > On Fri, May 30, 2014 at 8:33 AM, Andrija Panic <
>>>>> andrija.pa...@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> > > Nope, started, did check, it is reported now as Debian 5 VM, but
>>>>> still
>>>>> > > doesn't work...
>>>>> > > Rebooted VPC (destroyed VR, new one created...)
>>>>> > >
>>>>> > > $ virsh dumpxml r-812-VM
>>>>> > > ...
>>>>> > > <description>Debian GNU/Linux 5(64-bit)</description>
>>>>> > > ...
>>>>> > > <os>
>>>>> > > <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
>>>>> > > <boot dev='cdrom'/>
>>>>> > > <boot dev='hd'/>
>>>>> > > </os>
>>>>> > >
>>>>> > > :(
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > On 30 May 2014 16:24, Andrija Panic <andrija.pa...@gmail.com>
>>>>> wrote:
>>>>> > >
>>>>> > > > I confirm, the highest is Debian 5 64bit
>>>>> > > >
>>>>> > > > Per docs
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-2-x-to-4-3
>>>>> > > > , you should use Debian 7.0 64bit as OS type for system-kvm-4.3
>>>>> > > template...
>>>>> > > > (same for xen and vmware templatest)
>>>>> > > >
>>>>> > > > Will change now DB to point to debian 5, and let you know.
>>>>> > > >
>>>>> > > > This is guest_os_id: 184 | 2 | NULL |
>>>>> > > > 986a0e98-39d6-11e3-8f93-0025904e4412 | Debian GNU/Linux 7(64-bit)
>>>>> > > >
>>>>> > > >
>>>>> > > > On 30 May 2014 15:45, Joris van Lieshout <
>>>>> > > jvanliesh...@schubergphilis.com>
>>>>> > > > wrote:
>>>>> > > >
>>>>> > > >> Andrija,
>>>>> > > >>
>>>>> > > >> The thing is I don't know who the os matching on KVM works.
>>>>> There must
>>>>> > > be
>>>>> > > >> a way to list supported os types.
>>>>> > > >>
>>>>> > > >> I also did some queuing on the guest_os_hypervisor table (ACS
>>>>> 4.3)
>>>>> > and I
>>>>> > > >> don't see Debian 7 for KVM listed.
>>>>> > > >>
>>>>> > > >> select * from guest_os join guest_os_hypervisor on
>>>>> > > >> guest_os.id=guest_os_hypervisor.guest_os_id where
>>>>> > > >> guest_os_hypervisor.hypervisor_type='KVM' and
>>>>> > > >> guest_os_hypervisor.guest_os_name like '%Debian%';
>>>>> > > >>
>>>>> > > >> What was the guest_os_id you where using? Could you try id 72
>>>>> (Debian
>>>>> > 5
>>>>> > > >> 64-bit)? Adjust both os_type_id in vm_instance and vm_template
>>>>> (where
>>>>> > > >> type='SYSTEM' and hypervisor_type='KVM').
>>>>> > > >>
>>>>> > > >> Kind regards,
>>>>> > > >> Joris van Lieshout
>>>>> > > >>
>>>>> > > >> Schuberg Philis
>>>>> > > >>
>>>>> > > >>
>>>>> > > >>
>>>>> > > >>
>>>>> > > >>
>>>>> > > >> On 30/05/14 15:33, "Andrija Panic" <andrija.pa...@gmail.com>
>>>>> wrote:
>>>>> > > >>
>>>>> > > >> >Joris,
>>>>> > > >> >
>>>>> > > >> >do you have recommendation on how in particular to try ? I'm
>>>>> not sure
>>>>> > > how
>>>>> > > >> >to fix that, except playing with editing systemvm-4.3 template
>>>>> to
>>>>> > > define
>>>>> > > >> >it
>>>>> > > >> >as another OS type... ?
>>>>> > > >> >
>>>>> > > >> >Thanks again,
>>>>> > > >> >Andrija
>>>>> > > >> >
>>>>> > > >> >
>>>>> > > >> >On 30 May 2014 15:30, Joris van Lieshout <
>>>>> > > >> jvanliesh...@schubergphilis.com>
>>>>> > > >> >wrote:
>>>>> > > >> >
>>>>> > > >> >> I've read back a bit in the code and if you look at
>>>>> > > >> BridgeVifDriver.java
>>>>> > > >> >> (this is where the log message with the nic profile is
>>>>> generated)
>>>>> > you
>>>>> > > >> >>can
>>>>> > > >> >> see that the nic information might be off already once ACS
>>>>> hits the
>>>>> > > >> >> LibvirtVMDef.InterfaceDef plug function. This leads be to
>>>>> believer
>>>>> > > that
>>>>> > > >> >> the HVM/PV OS mismatch issue might still be related. Try
>>>>> fixing
>>>>> > that
>>>>> > > >> >> first. At least it will allow us to exclude this from the
>>>>> list.
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >> >> Kind regards,
>>>>> > > >> >> Joris van Lieshout
>>>>> > > >> >>
>>>>> > > >> >> Schuberg Philis
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >> >> On 30/05/14 15:26, "Andrija Panic" <andrija.pa...@gmail.com>
>>>>> > wrote:
>>>>> > > >> >>
>>>>> > > >> >> >OK, thanks Joris.
>>>>> > > >> >> >
>>>>> > > >> >> >I will try playing with OS version option, on the
>>>>> systemvm-kvm-4.3
>>>>> > > >> >> >template...
>>>>> > > >> >> >
>>>>> > > >> >> >Let me know if I can help with anything more.
>>>>> > > >> >> >
>>>>> > > >> >> >Thanks.
>>>>> > > >> >> >Andrija
>>>>> > > >> >> >
>>>>> > > >> >> >
>>>>> > > >> >> >On 30 May 2014 15:19, Joris van Lieshout
>>>>> > > >> >><jvanliesh...@schubergphilis.com
>>>>> > > >> >> >
>>>>> > > >> >> >wrote:
>>>>> > > >> >> >
>>>>> > > >> >> >> Hi Andrija,
>>>>> > > >> >> >>
>>>>> > > >> >> >> That does sound familiar and in the start xml of KVM you
>>>>> can see
>>>>> > > >> >>"<type
>>>>> > > >> >> >> arch='x86_64' machine='pc'>hvm</type>". I don't know
>>>>> KVM+ACS
>>>>> > well
>>>>> > > >> >>enough
>>>>> > > >> >> >> to judge if this is the cause but I thing focusing on
>>>>> getting
>>>>> > the
>>>>> > > VR
>>>>> > > >> >> >> started as PV guest might be worth trying. On the other
>>>>> hand I
>>>>> > do
>>>>> > > >> see
>>>>> > > >> >> >> patchviasocket.pl being executed successfully...
>>>>> > > >> >> >>
>>>>> > > >> >> >> The other thing I see is, and now we're getting into java
>>>>> code,
>>>>> > is
>>>>> > > >> >>this:
>>>>> > > >> >> >>
>>>>> > > >> >> >> 2014-05-30 14:41:01,386{GMT} DEBUG
>>>>> > [kvm.resource.BridgeVifDriver]
>>>>> > > >> >> >> (agentRequest-Handler-3:)
>>>>> > > >> >> >>nic=[Nic:Public-46.232.xxx.246-vlan://untagged]
>>>>> > > >> >> >> 2014-05-30 14:41:01,502{GMT} DEBUG [cloud.agent.Agent]
>>>>> > > >> >> >> (agentRequest-Handler-3:) Processing command:
>>>>> > > >> >> >> com.cloud.agent.api.routing.IpAssocVpcCommand
>>>>> > > >> >> >> 2014-05-30 14:41:01,506{GMT} DEBUG
>>>>> > > >> >> >> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> > > >> >> >>(agentRequest-Handler-3:)
>>>>> > > >> >> >> Executing:
>>>>> > > >> >> >>
>>>>> > /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>>> > > >> >> >> vpc_ipassoc.sh 169.254.0.52 -A -l 46.232.xxx.246 -c
>>>>> ethnull -g
>>>>> > > >> >> >> 46.232.xxx.1 -m 24 -n 46.232.xxx.0
>>>>> > > >> >> >>
>>>>> > > >> >> >> My suspicion is that somewhere in the translation from the
>>>>> nic
>>>>> > > >> >>profile
>>>>> > > >> >> >>to
>>>>> > > >> >> >> the actual route_proxy.sh command ACS failes to find the
>>>>> nic id
>>>>> > > and
>>>>> > > >> >> >> returns null.
>>>>> > > >> >> >>
>>>>> > > >> >> >> Let me dig a bit deeper and see what I can find but this is
>>>>> > where
>>>>> > > we
>>>>> > > >> >> >>might
>>>>> > > >> >> >> need some help from someone with knowledge of this pice of
>>>>> the
>>>>> > > code.
>>>>> > > >> >>:)
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >> Kind regards,
>>>>> > > >> >> >> Joris van Lieshout
>>>>> > > >> >> >>
>>>>> > > >> >> >> Schuberg Philis
>>>>> > > >> >> >> Boeingavenue 271
>>>>> > > >> >> >> 1119 PD Schiphol-Rijk
>>>>> > > >> >> >> schubergphilis.com
>>>>> > > >> >> >>
>>>>> > > >> >> >> +31 20-7506672
>>>>> > > >> >> >> +31 6-51428188
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >> On 30/05/14 14:49, "Andrija Panic" <
>>>>> andrija.pa...@gmail.com>
>>>>> > > wrote:
>>>>> > > >> >> >>
>>>>> > > >> >> >> >Hi Joris,
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >I have turned on DEBUG loging in agent.log on cs1.xxx/net
>>>>> host:
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >So, management logs again: http://pastebin.com/F6BRf7Y9
>>>>> > > >> >> >> >Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >Not playing smart, but there is some error:
>>>>> > > >> >> >> >[kvm.resource.KVMGuestOsMapper]
>>>>> > > >> >> >> >(agentRequest-Handler-3:) Can't find the mapping of guest
>>>>> os:
>>>>> > > >> Debian
>>>>> > > >> >> >> >GNU/Linux 7(64-bit)
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >Best,
>>>>> > > >> >> >> >Andrija
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >On 30 May 2014 14:26, Joris van Lieshout
>>>>> > > >> >> >><jvanliesh...@schubergphilis.com
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >wrote:
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >> Hi Andrija,
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> Bold formatting does not come trough on the dev list. :)
>>>>> > > >> >> >> >> But u might need a bit more info.
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> At a certain point I see this line
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request]
>>>>> > > >> >> >> >> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
>>>>> 1-609104082:
>>>>> > > >> >>Sending
>>>>> > > >> >> >> {
>>>>> > > >> >> >> >> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net),
>>>>> Ver: v1,
>>>>> > > >> Flags:
>>>>> > > >> >> >> >>100111,
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM
>>>>> > > >> >>>>>>".
>>>>> > > >> >> >>>>..
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> This is where the information is passed on to the agent
>>>>> > > handles.
>>>>> > > >> >>For
>>>>> > > >> >> >>XS
>>>>> > > >> >> >> >> this would initiate an agent handler on the management
>>>>> server
>>>>> > > but
>>>>> > > >> >>for
>>>>> > > >> >> >> >>KVM,
>>>>> > > >> >> >> >> if I remember correctly, it passed the command on to the
>>>>> > > >> >>cloudstack
>>>>> > > >> >> >> >>agent
>>>>> > > >> >> >> >> service on the hypervisor.
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> Can you check the cloud service log on the KVM
>>>>> hypervisor
>>>>> > > >> >>executing
>>>>> > > >> >> >>the
>>>>> > > >> >> >> >> request? it's this server cs1.xxxxx.net and then
>>>>> search top
>>>>> > > down
>>>>> > > >> >>for
>>>>> > > >> >> >> >> 609104082 in the log. See if you can provide the log
>>>>> from the
>>>>> > > >> >>agent
>>>>> > > >> >> >> >> handler thread started by that sequence.
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> Kind regards,
>>>>> > > >> >> >> >> Joris van Lieshout
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> Schuberg Philis
>>>>> > > >> >> >> >> Boeingavenue 271
>>>>> > > >> >> >> >> 1119 PD Schiphol-Rijk
>>>>> > > >> >> >> >> schubergphilis.com
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> +31 20-7506672
>>>>> > > >> >> >> >> +31 6-51428188
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> On 30/05/14 14:08, "Andrija Panic" <
>>>>> andrija.pa...@gmail.com>
>>>>> > > >> >>wrote:
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >> >Hi Joris,
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >here is the management log:
>>>>> http://pastebin.com/zxnKxFhk
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >Interesting parts (to me): in bold
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >2014-05-30 13:56:21,899 DEBUG
>>>>> > > >> >>[o.a.c.s.m.AncientDataMotionStrategy]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync
>>>>> > > inspecting
>>>>> > > >> >>src
>>>>> > > >> >> >> >>type
>>>>> > > >> >> >> >> >TEMPLATE copyAsync inspecting dest type VOLUME
>>>>> > > >> >> >> >> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
>>>>> > 4-1248669612:
>>>>> > > >> >> >>Sending
>>>>> > > >> >> >> >>{
>>>>> > > >> >> >> >> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net),
>>>>> Ver: v1,
>>>>> > > >> >>Flags:
>>>>> > > >> >> >> >> 100011,
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org
>>>>> > > >> >>>>>>>.a
>>>>> > > >> >> >>>>>pa
>>>>> > > >> >> >> >>>ch
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0
>>>>> > > >> >>>>>>>f-
>>>>> > > >> >> >>>>>b0
>>>>> > > >> >> >> >>>b4
>>>>> > > >> >> >> >> >-5e351e7cae55","origUrl":"
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mas
>>>>> > > >> >> >> >>t
>>>>> > > >> >> >> >> >er-kvm.qcow2.bz2
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"Q
>>>>> > > >> >>>>>>>CO
>>>>> > > >> >> >>>>>W2
>>>>> > > >> >> >> >>>",
>>>>> > > >> >> >> >>
>>>>> > >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","
>>>>> > > >> >> >> >> >*hvm":true*
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.clou
>>>>> > > >> >>>>>>>ds
>>>>> > > >> >> >>>>>ta
>>>>> > > >> >> >> >>>ck
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-220
>>>>> > > >> >>>>>>>3f
>>>>> > > >> >> >>>>>79
>>>>> > > >> >> >> >>>b1
>>>>> > > >> >> >> >> >dc6","id":209,"poolType":"RBD","host":"
>>>>> > > >> >> >> >> >cephmon.xxxxx.net
>>>>> > > >> ","path":"cloudstack","port":6789,"url":"RBD://
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8
>>>>> > > >> >> >> >>8
>>>>> > > >> >> >> >> >a8-2203f79b1dc6
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorT
>>>>> > > >> >>>>>>>yp
>>>>> > > >> >> >>>>>e"
>>>>> > > >> >> >> >>>:"
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
>>>>> > > >> >>>>>>>"u
>>>>> > > >> >> >>>>>ui
>>>>> > > >> >> >> >>>d"
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataSto
>>>>> > > >> >>>>>>>re
>>>>> > > >> >> >>>>>":
>>>>> > > >> >> >> >>>{"
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b9342
>>>>> > > >> >>>>>>>2e
>>>>> > > >> >> >>>>>-1
>>>>> > > >> >> >> >>>a6
>>>>> > > >> >> >> >>
>>>>> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>>> > > >> >> >> >> >cephmon.xxxxx.net
>>>>> > > >> ","path":"cloudstack","port":6789,"url":"RBD://
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8
>>>>> > > >> >> >> >>8
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":10
>>>>> > > >> >>>>>>>64
>>>>> > > >> >> >>>>>,"
>>>>> > > >> >> >> >>>vm
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":
>>>>> > > >> >>>>>>>0,
>>>>> > > >> >> >>>>>"h
>>>>> > > >> >> >> >>>yp
>>>>> > > >> >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}
>>>>> > > >> >>>>>]
>>>>> > > >> >> >> >> >}
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
>>>>> > > >> >> >> >> >(AgentManager-Handler-12:null) Seq 4-1248669612:
>>>>> Processing:
>>>>> > > {
>>>>> > > >> >> >>Ans: ,
>>>>> > > >> >> >> >> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10,
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
>>>>> > > >> >>>>>>>"o
>>>>> > > >> >> >>>>>rg
>>>>> > > >> >> >> >>>.a
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path
>>>>> > > >> >>>>>>>":
>>>>> > > >> >> >>>>>"9
>>>>> > > >> >> >> >>>c4
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id"
>>>>> > > >> >>>>>>>:0
>>>>> > > >> >> >>>>>}}
>>>>> > > >> >> >> >>>,"
>>>>> > > >> >> >> >> >result":true,"wait":0}}]
>>>>> > > >> >> >> >> >}
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
>>>>> > 4-1248669612:
>>>>> > > >> >> >> >>Received: {
>>>>> > > >> >> >> >> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags:
>>>>> 10, {
>>>>> > > >> >> >> >>CopyCmdAnswer
>>>>> > > >> >> >> >> >} }
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
>>>>> nic
>>>>> > > >> >>NicProfile[
>>>>> > > >> >> >> >> >*1092-801-null*-46.232.xxx.246-vlan://untagged of type
>>>>> > Public
>>>>> > > >> >>from
>>>>> > > >> >> >>the
>>>>> > > >> >> >> >> >nics
>>>>> > > >> >> >> >> >passed on vm start. The nic will be plugged later
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
>>>>> nic
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vl
>>>>> > > >> >>>>>>>an
>>>>> > > >> >> >>>>>:/
>>>>> > > >> >> >> >>>/4
>>>>> > > >> >> >> >> >4
>>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
>>>>> nic will
>>>>> > > be
>>>>> > > >> >> >>plugged
>>>>> > > >> >> >> >> >later
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
>>>>> nic
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vl
>>>>> > > >> >>>>>>>an
>>>>> > > >> >> >>>>>:/
>>>>> > > >> >> >> >>>/4
>>>>> > > >> >> >> >> >3
>>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
>>>>> nic will
>>>>> > > be
>>>>> > > >> >> >>plugged
>>>>> > > >> >> >> >> >later
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
>>>>> nic
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl
>>>>> > > >> >>>>>>>an
>>>>> > > >> >> >>>>>:/
>>>>> > > >> >> >> >>>/3
>>>>> > > >> >> >> >> >004
>>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
>>>>> nic will
>>>>> > > be
>>>>> > > >> >> >>plugged
>>>>> > > >> >> >> >> >later
>>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing
>>>>> nic
>>>>> > > >> >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >>
>>>>> > > >>
>>>>> > > >>
>>>>> > >
>>>>> >
>>>>> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl
>>>>> > > >> >>>>>>>an
>>>>> > > >> >> >>>>>:/
>>>>> > > >> >> >> >>>/3
>>>>> > > >> >> >> >> >004
>>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >Thanks,
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >On 30 May 2014 13:54, Joris van Lieshout
>>>>> > > >> >> >> >><jvanliesh...@schubergphilis.com
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >wrote:
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >> Hi Andrija,
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >> Just the start of the VR should be sufficient.
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >> Kind regards,
>>>>> > > >> >> >> >> >> Joris van Lieshout
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >> Schuberg Philis
>>>>> > > >> >> >> >> >> Boeingavenue 271
>>>>> > > >> >> >> >> >> 1119 PD Schiphol-Rijk
>>>>> > > >> >> >> >> >> schubergphilis.com
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >> +31 20-7506672
>>>>> > > >> >> >> >> >> +31 6-51428188
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >> On 30/05/14 13:48, "Andrija Panic" <
>>>>> > andrija.pa...@gmail.com
>>>>> > > >
>>>>> > > >> >> >>wrote:
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >> >Hi Joris,
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >just to be sure - you want me to capture the log
>>>>> from the
>>>>> > > >> >>moment
>>>>> > > >> >> >>I
>>>>> > > >> >> >> >> >>reboot
>>>>> > > >> >> >> >> >> >router - or you want me to stop it, then start
>>>>> capturing
>>>>> > > log,
>>>>> > > >> >>and
>>>>> > > >> >> >> >> >>start it
>>>>> > > >> >> >> >> >> >(and continue capture untill ethnull errors inside
>>>>> VR) ?
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >Thanks,
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >On 30 May 2014 13:39, Joris van Lieshout
>>>>> > > >> >> >> >> >><jvanliesh...@schubergphilis.com
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >wrote:
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> Hi Andrija,
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> Thanks for the answers. In deed your situation is
>>>>> > > different
>>>>> > > >> >>so
>>>>> > > >> >> >> >> >>PV/HVM is
>>>>> > > >> >> >> >> >> >> not the issue.
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> When reading back the log output you have
>>>>> provided I
>>>>> > > noted
>>>>> > > >> >>that
>>>>> > > >> >> >> >>the
>>>>> > > >> >> >> >> >>VR
>>>>> > > >> >> >> >> >> >> messages log indicates that it's waiting for
>>>>> ethnull to
>>>>> > > be
>>>>> > > >> >>up.
>>>>> > > >> >> >> >>This
>>>>> > > >> >> >> >> >> >>raises
>>>>> > > >> >> >> >> >> >> the question where null was introduced instead of
>>>>> 1.
>>>>> > The
>>>>> > > >> ACS
>>>>> > > >> >> >> >> >>management
>>>>> > > >> >> >> >> >> >> log output you send was, what I think, later down
>>>>> the
>>>>> > > road
>>>>> > > >> >> >>where
>>>>> > > >> >> >> >>ACS
>>>>> > > >> >> >> >> >> >>gives
>>>>> > > >> >> >> >> >> >> up trying to wait for the VR to come up. If you
>>>>> would
>>>>> > > >> >>capture
>>>>> > > >> >> >>the
>>>>> > > >> >> >> >> >> >> job-executor in the management log from
>>>>> startCommand
>>>>> > till
>>>>> > > >> >>the
>>>>> > > >> >> >> >> >>exception,
>>>>> > > >> >> >> >> >> >> do you see anywhere a mention of ethnull? You
>>>>> might
>>>>> > need
>>>>> > > to
>>>>> > > >> >> >>reed
>>>>> > > >> >> >> >>into
>>>>> > > >> >> >> >> >> >>the
>>>>> > > >> >> >> >> >> >> DirectAgent executing the startCommand to find a
>>>>> clue.
>>>>> > > The
>>>>> > > >> >> >>thing
>>>>> > > >> >> >> >>is
>>>>> > > >> >> >> >> >> >>that I
>>>>> > > >> >> >> >> >> >> only have experience with XS based environment so
>>>>> I
>>>>> > > cannot
>>>>> > > >> >> >>point
>>>>> > > >> >> >> >>you
>>>>> > > >> >> >> >> >>to
>>>>> > > >> >> >> >> >> >> the exact output to look for. On XS, at least, it
>>>>> is
>>>>> > > >> >> >> >> >> >> "[c.c.h.x.r.CitrixResourceBase]
>>>>> > > >> >>(DirectAgent-351:ctx-4a51bb9e)
>>>>> > > >> >> >> >> >>Created a
>>>>> > > >> >> >> >> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1"
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> Kind regards,
>>>>> > > >> >> >> >> >> >> Joris van Lieshout
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> Schuberg Philis
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> On 30/05/14 10:48, "Andrija Panic" <
>>>>> > > >> andrija.pa...@gmail.com>
>>>>> > > >> >> >> >>wrote:
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >Hi Deen,
>>>>> > > >> >> >> >> >> >> >no, in DB there is field "vlan_id" with value
>>>>> > > "untagged" -
>>>>> > > >> >> >>that
>>>>> > > >> >> >> >> >> >> >"vlan://untagged" is shown from ACS gui, and is
>>>>> used
>>>>> > in
>>>>> > > >> API
>>>>> > > >> >> >>call
>>>>> > > >> >> >> >>(or
>>>>> > > >> >> >> >> >> >> >better
>>>>> > > >> >> >> >> >> >> >said commands that are seen in management server
>>>>> > logs).
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >Best,
>>>>> > > >> >> >> >> >> >> >Andrija
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >On 30 May 2014 10:37, Daan Hoogland
>>>>> > > >> >><daan.hoogl...@gmail.com>
>>>>> > > >> >> >> >> wrote:
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> Andrija,
>>>>> > > >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> Do not just assign a second net vlan://500 You
>>>>> have
>>>>> > > one
>>>>> > > >> >>like
>>>>> > > >> >> >> >>that
>>>>> > > >> >> >> >> >>and
>>>>> > > >> >> >> >> >> >> >> you don't want conflicting nets using the same
>>>>> > vlan. I
>>>>> > > >> am
>>>>> > > >> >> >> >> >>wondering
>>>>> > > >> >> >> >> >> >> >> why 'untagged' comes out as 'vlan://untagged'.
>>>>> I
>>>>> > think
>>>>> > > >> >>that
>>>>> > > >> >> >>is
>>>>> > > >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >> bug. Did you find the string 'vlan://untagged'
>>>>> in
>>>>> > your
>>>>> > > >> >>db?
>>>>> > > >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic
>>>>> > > >> >> >> >> >> >> >><andrija.pa...@gmail.com>
>>>>> > > >> >> >> >> >> >> >> wrote:
>>>>> > > >> >> >> >> >> >> >> > Hi Joris,
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > thank you for taking time to address this
>>>>> issue :)
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > So...:
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by
>>>>> Inktank
>>>>> > > for
>>>>> > > >> >>CEPH
>>>>> > > >> >> >> >> >> >>support),
>>>>> > > >> >> >> >> >> >> >>OS
>>>>> > > >> >> >> >> >> >> >> is
>>>>> > > >> >> >> >> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled.
>>>>> > > >> >> >> >> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine
>>>>> > > >> >> >> >> >> >> >> > - not XS, so I guess no answers for this
>>>>> part :)
>>>>> > > >> >> >> >> >> >> >> > - guest_os_id is 184 = Debian 7 x64
>>>>> > > >> >> >> >> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 =
>>>>> Debian 7
>>>>> > > x64
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > This worked previously on 4.2.1 = template
>>>>> was
>>>>> > > >> ofcourse
>>>>> > > >> >> >> >> >> >> >>systemvm-kvm-4.2
>>>>> > > >> >> >> >> >> >> >> -
>>>>> > > >> >> >> >> >> >> >> > but that was also Debian 7 x64 type... so
>>>>> this
>>>>> > > should
>>>>> > > >> >>not
>>>>> > > >> >> >>be
>>>>> > > >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >>issues
>>>>> > > >> >> >> >> >> >> >> > (guest not supported by host...)
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > The only thing that might be out of
>>>>> "standard" =
>>>>> > all
>>>>> > > >> >>SVMs
>>>>> > > >> >> >> >>are on
>>>>> > > >> >> >> >> >> >>CEPH
>>>>> > > >> >> >> >> >> >> >>-
>>>>> > > >> >> >> >> >> >> >> > there are official docs on altering database
>>>>> to
>>>>> > make
>>>>> > > >> >>some
>>>>> > > >> >> >>new
>>>>> > > >> >> >> >> >> >>System
>>>>> > > >> >> >> >> >> >> >> > Offering as default for SSVM and CPVM - what
>>>>> I
>>>>> > did,
>>>>> > > I
>>>>> > > >> >>also
>>>>> > > >> >> >> >>have
>>>>> > > >> >> >> >> >> >>done
>>>>> > > >> >> >> >> >> >> >>same
>>>>> > > >> >> >> >> >> >> >> > config in DB, to make VR use another System
>>>>> > Offering
>>>>> > > >> as
>>>>> > > >> >> >> >>default
>>>>> > > >> >> >> >> >>-
>>>>> > > >> >> >> >> >> >> >>which
>>>>> > > >> >> >> >> >> >> >> is
>>>>> > > >> >> >> >> >> >> >> > NOT explained in the docs - you could use
>>>>> "Change
>>>>> > > >> >> >> >>Offering..."
>>>>> > > >> >> >> >> >> >>button
>>>>> > > >> >> >> >> >> >> >>on
>>>>> > > >> >> >> >> >> >> >> > exiting, shutdown VR to change it per docs...
>>>>> > > >> >> >> >> >> >> >> > But still this worked all fine on 4.2.1...
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > - regarding /var/cache/cloud/cmdline the
>>>>> content
>>>>> > is
>>>>> > > >> >> >>folowing
>>>>> > > >> >> >> >>at
>>>>> > > >> >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >> moment
>>>>> > > >> >> >> >> >> >> >> > root@r-801-VM:~# cat
>>>>> /var/cache/cloud/cmdline
>>>>> > > >> >> >> >> >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal
>>>>> > > >> dns1=8.8.8.8
>>>>> > > >> >> >> dns2=
>>>>> > > >> >> >> >> >> >> >> template=domP
>>>>> > > >> >> >> >> >> >> >> > name=r-801-VM eth0ip=169.254.0.75
>>>>> > > eth0mask=255.255.0.0
>>>>> > > >> >> >> >> >> >>type=vpcrouter
>>>>> > > >> >> >> >> >> >> >> > disable_rp_filter=true
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > Also please note that only eth1 does not
>>>>> have IP
>>>>> > > info,
>>>>> > > >> >> >>eth0
>>>>> > > >> >> >> >> >> >>(control
>>>>> > > >> >> >> >> >> >> >> > 169.xxx) and all other eh2 and up that are
>>>>> used
>>>>> > for
>>>>> > > >> >>Tiers
>>>>> > > >> >> >> >>get IP
>>>>> > > >> >> >> >> >> >>info
>>>>> > > >> >> >> >> >> >> >> fine.
>>>>> > > >> >> >> >> >> >> >> > I could also manually add IP for eth1
>>>>> (public NIC)
>>>>> > > and
>>>>> > > >> >> >>start
>>>>> > > >> >> >> >> >>ifup
>>>>> > > >> >> >> >> >> >> >>eth1 -
>>>>> > > >> >> >> >> >> >> >> > and it works fine, but adding new IP Port
>>>>> > Forwarding
>>>>> > > >> >>etc
>>>>> > > >> >> >>does
>>>>> > > >> >> >> >> >>not
>>>>> > > >> >> >> >> >> >> >>work...
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > Daan or somebody said it could be realted to
>>>>> my
>>>>> > > >> >>"Public"
>>>>> > > >> >> >> >>network
>>>>> > > >> >> >> >> >> >>(in
>>>>> > > >> >> >> >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >> > Zones, Physical Network, eth1 listing) is NOT
>>>>> > tagged
>>>>> > > >> >> >> >> >> >> >>(vlan://untagged)...
>>>>> > > >> >> >> >> >> >> >> > Interestingly the only VR that does work
>>>>> fine is
>>>>> > the
>>>>> > > >> VR
>>>>> > > >> >> >>used
>>>>> > > >> >> >> >>in
>>>>> > > >> >> >> >> >> >>Shared
>>>>> > > >> >> >> >> >> >> >> > network, but that VR is using IP from Guest
>>>>> IP
>>>>> > range
>>>>> > > >> >>(also
>>>>> > > >> >> >> >> >> >>efectively
>>>>> > > >> >> >> >> >> >> >> > public IPs but on vlan 500)
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > I was instructed to try to change Public IP
>>>>> range
>>>>> > > from
>>>>> > > >> >> >> >>untagged
>>>>> > > >> >> >> >> >>to
>>>>> > > >> >> >> >> >> >> >>vlan
>>>>> > > >> >> >> >> >> >> >> > 500, but I'm not sure how to do this, if
>>>>> there is
>>>>> > > any
>>>>> > > >> >>way
>>>>> > > >> >> >>at
>>>>> > > >> >> >> >>all
>>>>> > > >> >> >> >> >> >> >>(editing
>>>>> > > >> >> >> >> >> >> >> > "vlan" table and changing to vlan 500 does
>>>>> not
>>>>> > work,
>>>>> > > >> >>after
>>>>> > > >> >> >> >> >> >>rebooting
>>>>> > > >> >> >> >> >> >> >>VR
>>>>> > > >> >> >> >> >> >> >> > from ACS gui).
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > :)
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > So, not sure what is roughly expected date
>>>>> for
>>>>> > 4.4,
>>>>> > > >> but
>>>>> > > >> >> >>right
>>>>> > > >> >> >> >> >>now,
>>>>> > > >> >> >> >> >> >>I'm
>>>>> > > >> >> >> >> >> >> >> > pretty stuck with a big problem of all VPC
>>>>> not
>>>>> > > >> >> >>operational at
>>>>> > > >> >> >> >> >> >>all...
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > Thanks,
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout <
>>>>> > > >> >> >> >> >> >> >> jvanliesh...@schubergphilis.com>
>>>>> > > >> >> >> >> >> >> >> > wrote:
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> Hi Andrija,
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> Daan asked me to have a look at this as
>>>>> well.
>>>>> > > Looking
>>>>> > > >> >>at
>>>>> > > >> >> >>you
>>>>> > > >> >> >> >> >> >>issue I
>>>>> > > >> >> >> >> >> >> >> >> recall having seen something similar. Back
>>>>> then
>>>>> > > when
>>>>> > > >> >> >> >>upgrading
>>>>> > > >> >> >> >> >> >>4.2.1
>>>>> > > >> >> >> >> >> >> >>to
>>>>> > > >> >> >> >> >> >> >> >> 4.3 I though it had to do with out own
>>>>> custom
>>>>> > build
>>>>> > > >> >>svm
>>>>> > > >> >> >> >> >>template.
>>>>> > > >> >> >> >> >> >> >> >> Let me fire off some questions before
>>>>> explaining
>>>>> > > what
>>>>> > > >> >>the
>>>>> > > >> >> >> >>cause
>>>>> > > >> >> >> >> >> >>was
>>>>> > > >> >> >> >> >> >> >>in
>>>>> > > >> >> >> >> >> >> >> our
>>>>> > > >> >> >> >> >> >> >> >> case. :)
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> - what hypervisor (and version) are you
>>>>> using?
>>>>> > > >> >> >> >> >> >> >> >> - if XS, is the new VR a para-virtualised
>>>>> > instance
>>>>> > > >> >>(PV)
>>>>> > > >> >> >>or
>>>>> > > >> >> >> >> >> >>hardware
>>>>> > > >> >> >> >> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on
>>>>> the VR
>>>>> > > >> uuid
>>>>> > > >> >> >>and
>>>>> > > >> >> >> >> >>check
>>>>> > > >> >> >> >> >> >>that
>>>>> > > >> >> >> >> >> >> >> >> param PV-args is set and HVM-boot-policy is
>>>>> > unset.
>>>>> > > >> >> >> >> >> >> >> >> - what is the OS type of the VR in ACS
>>>>> > (guest_os_id
>>>>> > > >> in
>>>>> > > >> >> >> >> >>vm_instance
>>>>> > > >> >> >> >> >> >> >>table
>>>>> > > >> >> >> >> >> >> >> >> and match with table guest_os)
>>>>> > > >> >> >> >> >> >> >> >> - what is the OS type of the SVM template?
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> Now for the explaining. :)
>>>>> > > >> >> >> >> >> >> >> >> In our case the OS type of the new template
>>>>> was
>>>>> > not
>>>>> > > >> >> >> >>supported
>>>>> > > >> >> >> >> >>on
>>>>> > > >> >> >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >> >> XenServer version we are running. Therefore
>>>>> the
>>>>> > VR
>>>>> > > >> was
>>>>> > > >> >> >> >>started
>>>>> > > >> >> >> >> >>by
>>>>> > > >> >> >> >> >> >>XS
>>>>> > > >> >> >> >> >> >> >>as
>>>>> > > >> >> >> >> >> >> >> a
>>>>> > > >> >> >> >> >> >> >> >> HVM guest. System vms on XS rely on the
>>>>> arguments
>>>>> > > >> >>passed
>>>>> > > >> >> >>to
>>>>> > > >> >> >> >> >>them
>>>>> > > >> >> >> >> >> >>in
>>>>> > > >> >> >> >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >> >> PV-args param (ends up on the guest in
>>>>> > > >> >> >> >>/var/cache/cloud/cmdline
>>>>> > > >> >> >> >> >> >> >>which in
>>>>> > > >> >> >> >> >> >> >> >> turn is used by cloud-early-config) in
>>>>> order to
>>>>> > > work.
>>>>> > > >> >> >> >>cmdline
>>>>> > > >> >> >> >> >> >> >>contains
>>>>> > > >> >> >> >> >> >> >> the
>>>>> > > >> >> >> >> >> >> >> >> NIC configuration information.
>>>>> > > >> >> >> >> >> >> >> >> So, long story short, if a VR gets started
>>>>> as a
>>>>> > HVM
>>>>> > > >> it
>>>>> > > >> >> >>will
>>>>> > > >> >> >> >>not
>>>>> > > >> >> >> >> >> >>get
>>>>> > > >> >> >> >> >> >> >>the
>>>>> > > >> >> >> >> >> >> >> >> information needed to configure it's NICs.
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> Workaround
>>>>> > > >> >> >> >> >> >> >> >> We corrected the os_type_id in the DB (yes
>>>>> I know
>>>>> > > >> >>editing
>>>>> > > >> >> >> >>the
>>>>> > > >> >> >> >> >>DB
>>>>> > > >> >> >> >> >> >>is
>>>>> > > >> >> >> >> >> >> >> >> something you usually don't want but there
>>>>> is no
>>>>> > > >> other
>>>>> > > >> >> >>way
>>>>> > > >> >> >> >>in
>>>>> > > >> >> >> >> >>this
>>>>> > > >> >> >> >> >> >> >>case)
>>>>> > > >> >> >> >> >> >> >> >> of the existing VR's and of the
>>>>> systemvmtemplate
>>>>> > to
>>>>> > > >> >> >> >>something
>>>>> > > >> >> >> >> >> >> >>supported
>>>>> > > >> >> >> >> >> >> >> by
>>>>> > > >> >> >> >> >> >> >> >> XenServer.
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> Kind regards,
>>>>> > > >> >> >> >> >> >> >> >> Joris van Lieshout
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> Schuberg Philis
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> On 29/05/14 12:18, "Andrija Panic"
>>>>> > > >> >> >><andrija.pa...@gmail.com
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >> >> >>wrote:
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >They are 2 traffic types on 1 physical net
>>>>> (that
>>>>> > > is
>>>>> > > >> >>both
>>>>> > > >> >> >> >> >>tagged
>>>>> > > >> >> >> >> >> >>vlan
>>>>> > > >> >> >> >> >> >> >> 500,
>>>>> > > >> >> >> >> >> >> >> >> >and untagged packets travel over same KVM
>>>>> > bridge,
>>>>> > > >> and
>>>>> > > >> >> >>over
>>>>> > > >> >> >> >> >>eth1
>>>>> > > >> >> >> >> >> >>to
>>>>> > > >> >> >> >> >> >> >> outside
>>>>> > > >> >> >> >> >> >> >> >> >world)...
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland
>>>>> > > >> >> >> >><daan.hoogl...@gmail.com>
>>>>> > > >> >> >> >> >> >> wrote:
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >> Are these two traffic types in one
>>>>> physical
>>>>> > net?
>>>>> > > >> or
>>>>> > > >> >> >>two
>>>>> > > >> >> >> >> >> >>physical
>>>>> > > >> >> >> >> >> >> >>nets
>>>>> > > >> >> >> >> >> >> >> >> >> on the same interface (seems wrong).
>>>>> > > >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal
>>>>> > Reddy
>>>>> > > >> >>Uradi
>>>>> > > >> >> >> >> >> >> >> >> >> <jayapalreddy.ur...@citrix.com> wrote:
>>>>> > > >> >> >> >> >> >> >> >> >> > I don't think editing DB table will
>>>>> work.
>>>>> > > >> >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >> > -Jayapal
>>>>> > > >> >> >> >> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija
>>>>> Panic
>>>>> > > >> >> >> >> >> >> >><andrija.pa...@gmail.com
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >> wrote:
>>>>> > > >> >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >> >> It's like this:
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> I have public subnet /24.
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> half is dedicated for Guest traffic
>>>>> (vlan
>>>>> > > 500)
>>>>> > > >> >>and
>>>>> > > >> >> >>the
>>>>> > > >> >> >> >> >> >>second
>>>>> > > >> >> >> >> >> >> >> half is
>>>>> > > >> >> >> >> >> >> >> >> >> >> dedicated to Public traffic/network
>>>>> (no
>>>>> > vlan
>>>>> > > >> >>tags,
>>>>> > > >> >> >> >>that
>>>>> > > >> >> >> >> >>is
>>>>> > > >> >> >> >> >> >> >> untagged
>>>>> > > >> >> >> >> >> >> >> >> >> packets)
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> Both vlan500 and untagged packets
>>>>> travel
>>>>> > over
>>>>> > > >> >> >>physical
>>>>> > > >> >> >> >> >>eth1
>>>>> > > >> >> >> >> >> >> >> >> >>interface on
>>>>> > > >> >> >> >> >> >> >> >> >> >> hypervisors and can reach Internet.
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> Thanks,
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland
>>>>> > > >> >> >> >> >> >><daan.hoogl...@gmail.com>
>>>>> > > >> >> >> >> >> >> >> wrote:
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM,
>>>>> Andrija
>>>>> > > >> >>Panic <
>>>>> > > >> >> >> >> >> >> >> >> >> andrija.pa...@gmail.com>
>>>>> > > >> >> >> >> >> >> >> >> >> >>> wrote:
>>>>> > > >> >> >> >> >> >> >> >> >> >>>> 500
>>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>> > > >> >> >> >> >> >> >> >> >> >>> is 500 the vlan of your guestnetwork
>>>>> or
>>>>> > your
>>>>> > > >> >> >>physical
>>>>> > > >> >> >> >> >> >>network?
>>>>> > > >> >> >> >> >> >> >> You
>>>>> > > >> >> >> >> >> >> >> >> >> >>> wouldn't want to have two nets with
>>>>> vlan
>>>>> > > 500!
>>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>> > > >> >> >> >> >> >> >> >> >> >>> --
>>>>> > > >> >> >> >> >> >> >> >> >> >>> Daan
>>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> --
>>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> >> Andrija Panić
>>>>> > > >> >> >> >> >> >> >> >> >> >> --------------------------------------
>>>>> > > >> >> >> >> >> >> >> >> >> >> http://admintweets.com
>>>>> > > >> >> >> >> >> >> >> >> >> >> --------------------------------------
>>>>> > > >> >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >> --
>>>>> > > >> >> >> >> >> >> >> >> >> Daan
>>>>> > > >> >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >--
>>>>> > > >> >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >> >Andrija Panić
>>>>> > > >> >> >> >> >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> >> >> >> >> > http://admintweets.com
>>>>> > > >> >> >> >> >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > --
>>>>> > > >> >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >> > Andrija Panić
>>>>> > > >> >> >> >> >> >> >> > --------------------------------------
>>>>> > > >> >> >> >> >> >> >> > http://admintweets.com
>>>>> > > >> >> >> >> >> >> >> > --------------------------------------
>>>>> > > >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >> --
>>>>> > > >> >> >> >> >> >> >> Daan
>>>>> > > >> >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >--
>>>>> > > >> >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >> >Andrija Panić
>>>>> > > >> >> >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> >> >> > http://admintweets.com
>>>>> > > >> >> >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >>
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >--
>>>>> > > >> >> >> >> >> >
>>>>> > > >> >> >> >> >> >Andrija Panić
>>>>> > > >> >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> >> > http://admintweets.com
>>>>> > > >> >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >>
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >--
>>>>> > > >> >> >> >> >
>>>>> > > >> >> >> >> >Andrija Panić
>>>>> > > >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >> > http://admintweets.com
>>>>> > > >> >> >> >> >--------------------------------------
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >>
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >--
>>>>> > > >> >> >> >
>>>>> > > >> >> >> >Andrija Panić
>>>>> > > >> >> >> >--------------------------------------
>>>>> > > >> >> >> > http://admintweets.com
>>>>> > > >> >> >> >--------------------------------------
>>>>> > > >> >> >>
>>>>> > > >> >> >>
>>>>> > > >> >> >
>>>>> > > >> >> >
>>>>> > > >> >> >--
>>>>> > > >> >> >
>>>>> > > >> >> >Andrija Panić
>>>>> > > >> >> >--------------------------------------
>>>>> > > >> >> > http://admintweets.com
>>>>> > > >> >> >--------------------------------------
>>>>> > > >> >>
>>>>> > > >> >>
>>>>> > > >> >
>>>>> > > >> >
>>>>> > > >> >--
>>>>> > > >> >
>>>>> > > >> >Andrija Panić
>>>>> > > >> >--------------------------------------
>>>>> > > >> > http://admintweets.com
>>>>> > > >> >--------------------------------------
>>>>> > > >>
>>>>> > > >>
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > >
>>>>> > > > Andrija Panić
>>>>> > > > --------------------------------------
>>>>> > > > http://admintweets.com
>>>>> > > > --------------------------------------
>>>>> > > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > >
>>>>> > > Andrija Panić
>>>>> > > --------------------------------------
>>>>> > > http://admintweets.com
>>>>> > > --------------------------------------
>>>>> > >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Andrija Panić
>>>>> --------------------------------------
>>>>> http://admintweets.com
>>>>> --------------------------------------
>>>>>
>>>>
>>>>
>>>
>>
--
Daan
