Hello
I have tried this and set expunge.interval and expunge.delay to 120 on weekend.
But it wasn't helpful. Old VMs (not all but some of them) still present in
VMware environment.
Looks like I am need to create a bug.
Is the any way to correctly remove them from CloudStack?
Just several examples:
mysql> select vm.name,vm.state AS VM_state,v.id as VOLUME_id,v.size,v.state AS
VOLUME_state,v.removed,d.root_disk_tag,v.chain_info from volumes v, vm_instance
vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id and
vm.instance_name like 'i-132-1967-VM' \G
*************************** 1. row ***************************
name: Test4UbuntuDesktop14-04
VM_state: Expunging
VOLUME_id: 2013
size: 21474836480
VOLUME_state: Expunged
removed: 2014-09-29 20:33:12
root_disk_tag: auto
chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[AUTO1]
i-132-1967-VM/4e34b5f5d1214296a762c633507acfd8.vmdk"]}
1 row in set (0.03 sec)
mysql>
mysql> select vm.name,vm.state AS VM_state,v.id as VOLUME_id,v.size,v.state AS
VOLUME_state,v.removed,d.root_disk_tag,v.chain_info from volumes v, vm_instance
vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id and
vm.instance_name like 'i-161-1922-VM' \G
*************************** 1. row ***************************
name: FCtest
VM_state: Expunging
VOLUME_id: 1970
size: 42949672960
VOLUME_state: Expunged
removed: 2014-09-22 20:33:01
root_disk_tag: auto
chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[AUTO2]
i-161-1922-VM/ROOT-1922.vmdk"]}
1 row in set (0.02 sec)
mysql>
mysql> select vm.name,vm.state AS VM_state,v.id as VOLUME_id,v.size,v.state AS
VOLUME_state,v.removed,d.root_disk_tag,v.chain_info from volumes v, vm_instance
vm, vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id and
vm.instance_name like 'i-2-1630-VM' \G
*************************** 1. row ***************************
name: myserver
VM_state: Expunging
VOLUME_id: 1676
size: 10737418240
VOLUME_state: Expunged
removed: 2014-08-14 11:37:24
root_disk_tag: sata
chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1]
i-2-1630-VM/ROOT-1630.vmdk"]}
1 row in set (0.03 sec)
mysql>
On 10/17/2014 03:11 PM, GopalaKrishnan wrote:
> Hi Denis,
>
> Try to reduce expunge interval values to 120 and expunge delay. When you
> delete VMs or disk, etc., CloudStack will keep its details in the database
> and disks on the primary storage for 24 hours.
>
> You can shorten this period to expunge 120 seconds.
>
> expunge.interval = 120
> expunge.delay=120
>
> Restart cloudstack once these changes completed.
>
> Thank You.
> Gopalakrishnan.S
> Cloud Consultant - Fogpanel.
>
>
>
> ----- Original Message ----- From: "Denis Finko" <denis.fi...@ecommerce.com>
> To: <us...@cloudstack.apache.org>; <dev@cloudstack.apache.org>
> Sent: Friday, October 17, 2014 5:21 PM
> Subject: Re: Primary Storage Allocated issue
>
>
>> GopalaKrishnan, I am appreciate you for quick reply!
>>
>> But unfortunately "expunge.interval" is not helpful. I have several time
>> updated it recently and it wasn't removed old VMs from VMware.
>>
>> Currently I have:
>> expunge.interval = 86400
>>
>>> There is no need to push manually to remove expunged volumes.
>> Yes, I am agree with you that it's supposed to be. But unfortunately I still
>> have a lot of volumes in 'Expunged' state and don't know how correctly
>> remove them from CloudStack side. It looks like a bug in CloudStack.
>>
>> On 10/17/2014 01:35 PM, GopalaKrishnan wrote:
>>> Hi Denis,
>>>
>>> I think, it should be calculated disk size also from 'Expunged' volumes.
>>> Try to update "expunge.interval" in cloudstack global settings.
>>>
>>> There is no need to push manually to remove expunged volumes.
>>>
>>> Thank You.
>>> Gopalakrishnan.S
>>> Cloud Consultant.
>>> ----- Original Message ----- From: "Denis Finko" <denis.fi...@ecommerce.com>
>>> To: <dev@cloudstack.apache.org>; <us...@cloudstack.apache.org>
>>> Sent: Friday, October 17, 2014 1:47 PM
>>> Subject: Primary Storage Allocated issue
>>>
>>>
>>>> Hello CloudStack community,
>>>>
>>>> Could you please take a look to following issue. The similar was described
>>>> it that topic
>>>> http://comments.gmane.org/gmane.comp.apache.cloudstack.user/13848 but
>>>> looks like wasn't resolved.
>>>>
>>>> In my environment I have:
>>>> CloudStack v. 4.2.1
>>>> VMware v. 5.1
>>>>
>>>> Recently I have found that CloudStack UI provide following information for
>>>> SATA storage:
>>>> Disk Total 5.00 TB
>>>> Disk Allocated 7.26 TB
>>>>
>>>> I have looked to VMware side and found absolutely different data:
>>>> Capacity: 5.00 TB
>>>> Provisioned Space: 4.27 TB
>>>>
>>>> As you can see CloudStack show allocated disk space on 3 TB more.
>>>>
>>>> My Global Settings that could be helpful for investigation:
>>>> storage.overprovisioning.factor = 4
>>>> storage.cleanup.enabled = true
>>>> storage.cleanup.interval = 86400
>>>> storage.stats.interval = 60000
>>>> capacity.check.period = 60000
>>>> expunge.delay = 60
>>>> expunge.interval = 86400
>>>> expunge.workers = 10
>>>>
>>>> I have looked to CloudStack database and found that some Volumes state in
>>>> 'Destroy' but also I have found a lot of in 'Expunged':
>>>> mysql> select v.state, sum(v.size) from volumes v, vm_instance vm,
>>>> vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id and
>>>> d.root_disk_tag like 'sata' group by v.state;
>>>> +-----------+---------------+
>>>> | state | sum(v.size) |
>>>> +-----------+---------------+
>>>> | Destroy | 664646189056 |
>>>> | Expunged | 5267777388544 |
>>>> | Expunging | 2147483648 |
>>>> | Ready | 3833258311680 |
>>>> +-----------+---------------+
>>>> 4 rows in set (0.00 sec)
>>>>
>>>> And a lot of VMs that have state=Expunged (in volume table) still present
>>>> in VMware and on storage. For example all these VMs still present in
>>>> VMware:
>>>>
>>>> mysql> select vm.name,vm.instance_name,vm.state AS
>>>> VM_state,v.id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag
>>>> from volumes v, vm_instance vm, vm_root_disk_tags d where
>>>> v.instance_id=vm.id and d.vm_id=vm.id and v.state like 'Expunged' and
>>>> d.root_disk_tag like 'sata'; (Names have been
>>>> changed)
>>>> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
>>>> | name | instance_name | VM_state | id | size |
>>>> VOLUME_state | removed | root_disk_tag |
>>>> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
>>>> | name1 | i-43-2147-VM | Expunging | 2195 | 53687091200 |
>>>> Expunged | 2014-10-16 08:12:04 | sata |
>>>> | name2 | i-17-2123-VM | Expunging | 2173 | 10737418240 |
>>>> Expunged | 2014-10-15 19:14:30 | sata |
>>>> | name3 | i-43-2052-VM | Expunging | 2098 | 10737418240 |
>>>> Expunged | 2014-10-07 11:57:49 | sata |
>>>> | name4 | i-191-1954-VM | Expunging | 2000 | 42949672960 |
>>>> Expunged | 2014-10-01 15:15:04 | sata |
>>>> | name5 | i-132-1966-VM | Expunging | 2012 | 21474836480 |
>>>> Expunged | 2014-09-29 20:33:12 | sata |
>>>> | name6 | i-132-1967-VM | Expunging | 2013 | 21474836480 |
>>>> Expunged | 2014-09-29 20:33:12 | sata |
>>>> | name7 | i-132-1962-VM | Expunging | 2008 | 10737418240 |
>>>> Expunged | 2014-09-29 20:33:11 | sata |
>>>> | name8 | i-132-1964-VM | Expunging | 2010 | 10737418240 |
>>>> Expunged | 2014-09-29 20:33:11 | sata |
>>>>
>>>> Not all VMs with 'Expunged' state still present in VMware but a lot of
>>>> from them still weren't removed!
>>>>
>>>> mysql> select
>>>> id,uuid,name,status,capacity_bytes,capacity_iops,tag,disk_used_capacity,disk_reserved_capacity
>>>> from storage_pool_view;
>>>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>>>> | id | uuid | name | status | capacity_bytes | capacity_iops
>>>> | tag | disk_used_capacity | disk_reserved_capacity |
>>>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>>>> | 2 | 096b3b6e-481d-33b4-9b34-7178612a2535 | SATA | Up |
>>>> 5497289703424 | NULL | sata | 7913594810761 | 0 |
>>>> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
>>>> 5 rows in set (0.02 sec)
>>>>
>>>> Also I have found that Expunged volumes have 'chain_info' field:
>>>>
>>>> mysql> select name,removed,state,chain_info from volumes where id=2112 \G
>>>> *************************** 1. row ***************************
>>>> name: ROOT-2066
>>>> removed: 2014-10-09 11:57:52
>>>> state: Expunged
>>>> chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1]
>>>> i-238-2066-VM/ROOT-2066.vmdk"]}
>>>> 1 row in set (0.00 sec)
>>>>
>>>>
>>>> BUT Destroyed VMs don't have it:
>>>> mysql> select name,removed,state,chain_info from volumes where id=2192 \G
>>>> *************************** 1. row ***************************
>>>> name: ROOT-2143
>>>> removed: 2014-10-15 18:42:48
>>>> state: Destroy
>>>> chain_info: NULL
>>>> 1 row in set (0.00 sec)
>>>>
>>>>
>>>> So, the questions are:
>>>> - How CloudStack calculate allocated disk size for storage? Is it use disk
>>>> size from 'Expunged' volumes?
>>>> - Can I manually push CloudStack to remove 'Expunged' volumes from VMware?
>>>>
>>>> I haven't created bug in Jira but I can if it's needed.
>>>>
>>>> Thank you!
>>>>
>>>> --
>>>> Best regards,
>>>> Denis
>>>>
>>>
>>>
>>> ---
>>> This email is free from viruses and malware because avast! Antivirus
>>> protection is active.
>>> http://www.avast.com
>>>
>>>
>>
>> --
>> Best regards,
>> Denis
>>
>
>
> ---
> This email is free from viruses and malware because avast! Antivirus
> protection is active.
> http://www.avast.com
>
>
--
Best regards,
Denis
