GopalaKrishnan, I am appreciate you for quick reply! But unfortunately "expunge.interval" is not helpful. I have several time updated it recently and it wasn't removed old VMs from VMware.
Currently I have: expunge.interval = 86400 > There is no need to push manually to remove expunged volumes. Yes, I am agree with you that it's supposed to be. But unfortunately I still have a lot of volumes in 'Expunged' state and don't know how correctly remove them from CloudStack side. It looks like a bug in CloudStack. On 10/17/2014 01:35 PM, GopalaKrishnan wrote: > Hi Denis, > > I think, it should be calculated disk size also from 'Expunged' volumes. Try > to update "expunge.interval" in cloudstack global settings. > > There is no need to push manually to remove expunged volumes. > > Thank You. > Gopalakrishnan.S > Cloud Consultant. > ----- Original Message ----- From: "Denis Finko" <denis.fi...@ecommerce.com> > To: <dev@cloudstack.apache.org>; <us...@cloudstack.apache.org> > Sent: Friday, October 17, 2014 1:47 PM > Subject: Primary Storage Allocated issue > > >> Hello CloudStack community, >> >> Could you please take a look to following issue. The similar was described >> it that topic >> http://comments.gmane.org/gmane.comp.apache.cloudstack.user/13848 but looks >> like wasn't resolved. >> >> In my environment I have: >> CloudStack v. 4.2.1 >> VMware v. 5.1 >> >> Recently I have found that CloudStack UI provide following information for >> SATA storage: >> Disk Total 5.00 TB >> Disk Allocated 7.26 TB >> >> I have looked to VMware side and found absolutely different data: >> Capacity: 5.00 TB >> Provisioned Space: 4.27 TB >> >> As you can see CloudStack show allocated disk space on 3 TB more. >> >> My Global Settings that could be helpful for investigation: >> storage.overprovisioning.factor = 4 >> storage.cleanup.enabled = true >> storage.cleanup.interval = 86400 >> storage.stats.interval = 60000 >> capacity.check.period = 60000 >> expunge.delay = 60 >> expunge.interval = 86400 >> expunge.workers = 10 >> >> I have looked to CloudStack database and found that some Volumes state in >> 'Destroy' but also I have found a lot of in 'Expunged': >> mysql> select v.state, sum(v.size) from volumes v, vm_instance vm, >> vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id and >> d.root_disk_tag like 'sata' group by v.state; >> +-----------+---------------+ >> | state | sum(v.size) | >> +-----------+---------------+ >> | Destroy | 664646189056 | >> | Expunged | 5267777388544 | >> | Expunging | 2147483648 | >> | Ready | 3833258311680 | >> +-----------+---------------+ >> 4 rows in set (0.00 sec) >> >> And a lot of VMs that have state=Expunged (in volume table) still present in >> VMware and on storage. For example all these VMs still present in VMware: >> >> mysql> select vm.name,vm.instance_name,vm.state AS >> VM_state,v.id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag from >> volumes v, vm_instance vm, vm_root_disk_tags d where v.instance_id=vm.id and >> d.vm_id=vm.id and v.state like 'Expunged' and d.root_disk_tag like 'sata'; >> (Names have been >> changed) >> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+ >> | name | instance_name | VM_state | id | size | >> VOLUME_state | removed | root_disk_tag | >> +----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+ >> | name1 | i-43-2147-VM | Expunging | 2195 | 53687091200 | Expunged >> | 2014-10-16 08:12:04 | sata | >> | name2 | i-17-2123-VM | Expunging | 2173 | 10737418240 | Expunged >> | 2014-10-15 19:14:30 | sata | >> | name3 | i-43-2052-VM | Expunging | 2098 | 10737418240 | Expunged >> | 2014-10-07 11:57:49 | sata | >> | name4 | i-191-1954-VM | Expunging | 2000 | 42949672960 | Expunged >> | 2014-10-01 15:15:04 | sata | >> | name5 | i-132-1966-VM | Expunging | 2012 | 21474836480 | Expunged >> | 2014-09-29 20:33:12 | sata | >> | name6 | i-132-1967-VM | Expunging | 2013 | 21474836480 | Expunged >> | 2014-09-29 20:33:12 | sata | >> | name7 | i-132-1962-VM | Expunging | 2008 | 10737418240 | Expunged >> | 2014-09-29 20:33:11 | sata | >> | name8 | i-132-1964-VM | Expunging | 2010 | 10737418240 | Expunged >> | 2014-09-29 20:33:11 | sata | >> >> Not all VMs with 'Expunged' state still present in VMware but a lot of from >> them still weren't removed! >> >> mysql> select >> id,uuid,name,status,capacity_bytes,capacity_iops,tag,disk_used_capacity,disk_reserved_capacity >> from storage_pool_view; >> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+ >> | id | uuid | name | status | capacity_bytes | capacity_iops | >> tag | disk_used_capacity | disk_reserved_capacity | >> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+ >> | 2 | 096b3b6e-481d-33b4-9b34-7178612a2535 | SATA | Up | >> 5497289703424 | NULL | sata | 7913594810761 | 0 | >> +----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+ >> 5 rows in set (0.02 sec) >> >> Also I have found that Expunged volumes have 'chain_info' field: >> >> mysql> select name,removed,state,chain_info from volumes where id=2112 \G >> *************************** 1. row *************************** >> name: ROOT-2066 >> removed: 2014-10-09 11:57:52 >> state: Expunged >> chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1] >> i-238-2066-VM/ROOT-2066.vmdk"]} >> 1 row in set (0.00 sec) >> >> >> BUT Destroyed VMs don't have it: >> mysql> select name,removed,state,chain_info from volumes where id=2192 \G >> *************************** 1. row *************************** >> name: ROOT-2143 >> removed: 2014-10-15 18:42:48 >> state: Destroy >> chain_info: NULL >> 1 row in set (0.00 sec) >> >> >> So, the questions are: >> - How CloudStack calculate allocated disk size for storage? Is it use disk >> size from 'Expunged' volumes? >> - Can I manually push CloudStack to remove 'Expunged' volumes from VMware? >> >> I haven't created bug in Jira but I can if it's needed. >> >> Thank you! >> >> -- >> Best regards, >> Denis >> > > > --- > This email is free from viruses and malware because avast! Antivirus > protection is active. > http://www.avast.com > > -- Best regards, Denis
