Yes, that is what is happening. Thanks for pointing that out.
On Mon, Nov 13, 2017 at 3:14 AM, Anshul Gangwar < anshul.gang...@accelerite.com> wrote: > Rafael, you seem to combine two things. Optimization and my fix are two > different commits. I mentioned about optimization (done in commit “8caf52c” > not by me) because that was broken by commit “8a3fd10”. Commit “ > 2c4ea503f92bcf9c611f409d5cdecb” only fixed the issue for newer versions > of XenServer. Probably you are just going through bug description and > missing the commit “8caf52c”. > > Regards, > Anshul > > On 10/11/17, 5:25 PM, "Rafael Weingärtner" <rafaelweingart...@gmail.com> > wrote: > > Anshul, > > I will not touch the merit of documentation in code, for now, this is > another discussion. And regarding the “optimization”, you brought it > up, > whenever I saw you discussing this issue, you would talk about some > sort of > “optimization”… see > https://github.com/apache/cloudstack/pull/1124#issuecomment-326550060. > At > least for me, that was quite confusing, and I was not seeing what you > were > talking about. > > If you diff the files, what broke your “solution” was the removal of > the > following lines: > > > EndPoint ep = endPointSelector.selectHypervisorHost(new > ZoneScope(host. > > getDataCenterId())); > > host = hostDao.findById(ep.getId()); > > > > The conditional checks were maintained. We only removed something that > we > seemed logicless. > > With PR https://github.com/apache/cloudstack/pull/2315, I did not add > anything extra. I only re-worked the method to make it clear and more > readable (removed the if-inception). The only point I changed is that, > instead of using “selectOneHypervisorHostByZone”, I create a specific > method for that, which selects a host according to the volume of the > snapshot. > > On Fri, Nov 10, 2017 at 8:21 AM, Anshul Gangwar < > anshul.gang...@accelerite.com> wrote: > > > Rafael, you are still misunderstanding the fix “ > > 2c4ea503f92bcf9c611f409d5cdecb”. That has only fixed the issue due > to > > check for specific version i.e. before “ > 2c4ea503f92bcf9c611f409d5cdecb” > > things were fine for xenserver 6.2 but not working for 6.5. Commit > doesn’t > > mention anything about optimization. Other than version change it > reverts > > the previous commit which is introducing hypervisor specific check > in core > > code. > > > > Regarding documentation, “git history”is the best documentation. Code > > documentation may not make sense after some changes as it is often > ignored. > > > > See the PR for more comments. > > > > Regards, > > Anshul > > > > On 10/11/17, 3:14 AM, "Rafael Weingärtner" < > rafaelweingart...@gmail.com> > > wrote: > > > > Anshul and Lotic, after debugging and inspecting code for quite a > > while I > > understood the complete picture and created a solution for it. > The PR > > with > > the solution is found athttps://github.com/apache/ > cloudstack/pull/2315 > > . > > This was a clear example of what I constantly tell people here, > code > > needs > > to be clear, concise, well tested and documented. Here follows > some > > explanation regarding the removal of some code introduced with > commit > > “2c4ea503f92bcf9c611f409d5cdecb”. > > > > With commit “8caf52c” Daan and I removed a small piece of code > > introduced > > by “2c4ea503f92bcf9c611f409d5cdecb” because it did not seem to > make > > much > > sense. We did not remove everything; we only removed the part > that was > > looking for a random host in the zone to execute the command. As > we > > (Anshul) exchanged some messages in some other PR, the code > introduced > > in > > “2c4ea503f92bcf9c611f409d5cdecb” would only work for zones that > do > > not have > > other hypervisor types if they are deployed with XenServer > clusters.It > > was > > being created a limitation in ACS that should not exist. I > stress again > > that this only happened for the lack of documentation and clear > coding. > > For instance, when I read the commit “ > 2c4ea503f92bcf9c611f409d5cdecb”, > > it > > says it introduced an “optimization”, then I assumed that the > process > > executed with our code base before commit “ > > 2c4ea503f92bcf9c611f409d5cdecb” > > is working, but not as fast or with as much quality as the code > with > > “2c4ea503f92bcf9c611f409d5cdecb”. However, that is not the > case; the > > code > > “2c4ea503f92bcf9c611f409d5cdecb” is not optimizing anything; it > is, in > > fact, fixing/creating a workflow to create templates from > snapshots in > > XenServer deployments. > > > > The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only > > tackling > > the problem for CloudStack deployments that use single > hypervisor types > > (restricted to XenServers host in the same zone; this means, it > was not > > expecting to have multiple hypervisors types in the same zone). > > Additionally, the lack of information regarding that solution > > (documentation, test cases and description in PRs and Jira > ticket) led > > the > > code to be removed in #1124 after a long discussion and analysis > in > > #1056. > > That piece of code seemed logicless. It would receive a hostId > and > > then > > change that hostId for other hostId of the zone without doing any > > check; it > > was not even checking the hypervisor and storage in which the > host was > > connected to. > > > > The problem reported in #CLOUDSTACK-9025 is caused by partial > snapshots > > that are taken in XenServer. This means, we do not take a > complete > > snapshot, but a partial one that contains only the modified > data. This > > requires rebuilding the VHD hierarchy when creating a template > out of > > the > > snapshot. The point is that the first hostId received is not a > hostId, > > but > > a system VM ID(SSVM). That is why the code in #1176 fixed the > problem > > for > > some deployment scenarios, but would cause problems for scenarios > > where we > > have multiple hypervisors in the same zone. We need to execute > the > > creation > > of the VHD that represents the template in the hypervisor, so > the VHD > > chain > > can be built using the parent links. > > > > The PR #2315 changes the behavior of > > com.cloud.hypervisor.XenServerGuru. > getCommandHostDelegation(long, > > Command). > > From now on we replace the hostId that is intended to execute > the “copy > > command” that will create the VHD of the template according to > some > > conditions that were already in place. The idea is that starting > with > > XenServer 6.2.0 hotFix ESP1004 we need to execute the command in > the > > hypervisor host and not from the SSVM. Moreover, the method was > > improved > > making it more readable; it was also created test cases assuring > that > > from > > XenServer 6.2.0 hotFix ESP1004 and upward versions we change the > hostId > > that will be used to execute the “copy command”. > > > > Furthermore, we are not selecting a random host from a zone > anymore. A > > new > > method was introduced in the HostDao object called > > “findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using > this > > method > > we look for a host that is in the cluster that is using the > storage > > pool > > where the volume from which the Snapshot is taken of resides. By > doing > > this, we guarantee that the host that is connected to the primary > > storage > > where all of the snapshots parent VHDs are stored is used to > create the > > template. > > > > Also, I opened the PR only against master. If someone needs the > fix in > > any > > other version we can cherry pick to 4.9.2.0. > > > > On Wed, Nov 8, 2017 at 9:41 AM, Anshul Gangwar < > > anshul.gang...@accelerite.com> wrote: > > > > > Rafael, In ACS 4.5.2 you are facing issue due to XenServer > version > > (was > > > only working on xenserver 6.2 with some patches) which got > fixed in > > 4.6 > > > with commit 2c4ea503f92bcf9c611f409d5cdecb42b0115b69. Now > with code > > clean > > > up in commit “8a3fd10” it’s broken. During storage refactoring > > changes in > > > commit “8caf52c” optimization was done to send CopyCommand to > > Hypervisor > > > resource instead of SSVM. Now with commit “8a3fd10” it’s again > going > > to > > > SSVM so it’s working fine for first snapshot but broken for > linked > > snapshot. > > > > > > Regards, > > > Anshul > > > > > > On 04/11/17, 7:20 PM, "Rafael Weingärtner" < > > rafaelweingart...@gmail.com> > > > wrote: > > > > > > Anshul, I tried with ACS 4.5.2 where commit “8a3fd10” is > not > > present. > > > This > > > is the commit where Daan and I removed some code that > seemed > > (is? at > > > least > > > for me) logicless (https://github.com/apache/ > > cloudstack/pull/1124 and > > > https://github.com/apache/cloudstack/pull/1056). > > > > > > I tested with ACS 4.5.2, which I have in a test > environment. I > > have ACS > > > 4.5.2, NFS as the primary storage, and XenServer 6.5 as > hosts. I > > > managed to > > > reproduce the problem as Lotic is having. > > > > > > I spin up a VM, then I created a file of 2GB in this VM, > and > > later I > > > took a > > > snapshot of the root volume of it (the VM only has the root > > volume). > > > The > > > VHD “687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd” is the > first > > snapshot > > > of the > > > root volume. If I create a template from this snapshot, > > everything > > > works. I > > > created a new VM from the template of the first snapshot, > and it > > > worked, > > > and the 2GB file was there. Then, I returned for the first > VM > > (from > > > which I > > > took the snapshot), I created another 1GB file. Then I > stopped > > the VM > > > again > > > and took the second snapshot (27d37d21-13a6-488d-b2f0- > > > 0ac54236b83a.vhd). > > > The interesting thing here is that the file size reduced!? > And I > > also > > > noticed that there is no hierarchy in the VHDs files. I > tried to > > start > > > a VM > > > from a template that I created with the second snapshot, > and it > > did not > > > work as the problem Lotic is having. > > > > > > [root@xenserver-1 ~]# ls -lah > > > > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/ > > snapshots/2/8/ > > > > total 6.9G > > > > drwxr-xr-x 2 root root 4.0K Nov 4 10:33 . > > > > drwxr-xr-x 3 root root 4.0K Nov 4 09:49 .. > > > > -rw-r--r-- 1 root root 3.2G Nov 4 10:38 > > > > 27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd > > > > -rw-r--r-- 1 root root 3.7G Nov 4 09:55 > > > > 687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd > > > > [root@xenserver-1 ~]# vhd-util scan -f -a -p > > > > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/ > > > snapshots/2/8/687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd > > > > vhd=687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd > > capacity=21474836480 > > > > size=3916767744 hidden=0 parent=none > > > > [root@xenserver-1 ~]# vhd-util scan -f -a -p > > > > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/ > > > snapshots/2/8/27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd > > > > vhd=27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd > > capacity=21474836480 > > > > size=3420873216 hidden=0 parent=none > > > > > > > > > > > > > Was it not working way before “8a3fd10”?! I think we will > need > > help > > > from > > > more folks here to see if this is a problem that only > affects > > XenServer > > > deployments. I will also test on my 4.9.2.0 environment > again to > > see > > > if I > > > miss anything. > > > > > > On Thu, Nov 2, 2017 at 6:30 AM, Anshul Gangwar < > > > anshul.gang...@accelerite.com> wrote: > > > > > > > Hi Lotic, > > > > > > > > Can you try with any release between 4.6 to 4.8 and see > if > > this bug > > > is > > > > there? > > > > > > > > Basically, any release which contains commit > > > 2c4ea503f92bcf9c611f409d5cdecb42b0115b69 > > > > and missing commit 8a3fd10. > > > > > > > > On 02/11/17, 1:14 AM, "Lotic Lists" <li...@lotic.com.br> > > wrote: > > > > > > > > Rafael, I know and appreciate your help :) > > > > > > > > I recognize all effort of CloudStack community. > > > > I think all admins use snapshot to backup your > virtual > > disks, I > > > think > > > > it is not a isolated problem. Imagine a crash on your > storage > > and > > > you need > > > > convert your snapshots to templates. > > > > > > > > Thanks > > > > Marcelo > > > > > > > > -----Original Message----- > > > > From: Rafael Weingärtner [mailto:rafaelweingartner@ > > gmail.com] > > > > Sent: quarta-feira, 1 de novembro de 2017 17:17 > > > > To: dev@cloudstack.apache.org > > > > Subject: Re: Strange size of template from snapshot > on > > XenServer > > > > > > > > Well, I could not reproduce the problem in ACS > 4.9.2.0. So, > > > before > > > > fixing anything, the person that might take on this > should > > first > > > find the > > > > root of the problem. > > > > > > > > I understand what you are saying that it is a > critical > > problem, > > > but > > > > you have to understand that this is an open-source and > free > > software. > > > > > > > > Having said that, ALL of my work here has been > pro-bono; > > even > > > though I > > > > am a committer and PMC, I am not willing to dive deeper > into > > this > > > issue now > > > > as it does not affect me. It does not mean that I would > not > > help > > > people if > > > > I have time, as you can notice whenever I have some free > time > > I try > > > to be > > > > around and guide or check things for other people in > this list. > > > > > > > > If this is a huge problem for you right now, I > suggest you > > > looking for > > > > companies that provide enterprise support for ACS. > > > > > > > > On Wed, Nov 1, 2017 at 7:35 PM, Lotic Lists < > > li...@lotic.com.br> > > > > wrote: > > > > > > > > > Hi Rafael > > > > > > > > > > Tested on ACS 4.5.2.2, 4.9.2.0, 4.9.3.0 and 4.10. > All > > versions > > > with > > > > > same issue > > > > > > > > > > Description and steps to reproduce > > > > > https://issues.apache.org/ > jira/browse/CLOUDSTACK-10128 > > > > > > > > > > Again, it is a critical problem for production > > environments. > > > > > > > > > > Regards > > > > > Marcelo > > > > > > > > > > -----Original Message----- > > > > > From: Rafael Weingärtner [mailto: > rafaelweingartner@ > > gmail.com] > > > > > Sent: terça-feira, 31 de outubro de 2017 17:10 > > > > > To: dev@cloudstack.apache.org > > > > > Subject: Re: Strange size of template from > snapshot on > > > XenServer > > > > > > > > > > Yes, it is a big problem. I have not checked the > code > > that > > > creates > > > > the > > > > > template, but I am not sure if the problem you are > > > experiencing is > > > > > related to 9025. It might be a problem in some > other > > place. > > > > > > > > > > My suggestion is for you to create a Jira ticket > > detailing the > > > > > situation and wait until someone fixes it. Have you > > tested on > > > ACS > > > > > 4.10? If 4.9.2.0 works, it might be something that > went > > only > > > into > > > > 4.9.3.0. > > > > > > > > > > On Tue, Oct 31, 2017 at 8:04 PM, Lotic Lists < > > > li...@lotic.com.br> > > > > wrote: > > > > > > > > > > > I tested on a fresh environment, acs 4.9.3.0, > > xenserver 6.5 > > > with > > > > all > > > > > > patches and primary storage NFS, we have same > problem > > > > > > > > > > > > - Snapshot files > > > > > > [root@acs01 203]# ls -lh > /exports/secondary/snapshots/ > > 2/6/ > > > total > > > > > > 2.8G > > > > > > -rw-r--r-- 1 root root 1.7G Oct 31 16:48 > > > > > > cad96f0c-de49-4255-b0a0-9aea5e2297cb.vhd > > > > > > <-- snap-base > > > > > > -rw-r--r-- 1 root root 1.1G Oct 31 16:49 > > > > > > fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd > > > > > > <-- snap-diff > > > > > > https://i.imgur.com/bWupsAS.png > > > > > > > > > > > > [root@acs01 203]# md5sum > /exports/secondary/snapshots/ > > > > > > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd > > > > > > 17eec8d4f6d4c128b34b9e2e1876ceb7 > > > /exports/secondary/snapshots/ > > > > > > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd > > > > > > > > > > > > - Template files > > > > > > [root@acs01 203]# ls -lh > /exports/secondary/template/ > > > tmpl/2/203 > > > > > > total 2.0G > > > > > > -rw-r--r-- 1 root root 1.1G Oct 31 16:50 > > > e9def45d-1b9e-4c2e-b843- > > > > > > 852fe40f00b2.vhd > > > > > > -rw-r--r-- 1 root root 303 Oct 31 16:50 > > template.properties > > > > > > > > > > > > [root@acs01 203]# md5sum > /exports/secondary/template/ > > > > > > tmpl/2/203/e9def45d-1b9e-4c2e- > b843-852fe40f00b2.vhd > > > > > > 17eec8d4f6d4c128b34b9e2e1876ceb7 > > > /exports/secondary/template/ > > > > > > tmpl/2/203/e9def45d-1b9e-4c2e- > b843-852fe40f00b2.vhd > > > > > > > > > > > > It's a huge problem!!! I think issue 9025 is not > > resolved > > > > > > > > > > > > Regards > > > > > > Marcelo > > > > > > > > > > > > -----Original Message----- > > > > > > From: Rafael Weingärtner [mailto: > rafaelweingartner@ > > gmail.com > > > ] > > > > > > Sent: terça-feira, 31 de outubro de 2017 16:00 > > > > > > To: us...@cloudstack.apache.org > > > > > > Subject: Re: Strange size of template from > snapshot on > > > XenServer > > > > > > > > > > > > Well, reading the description of the issue it > seems to > > be > > > related > > > > to > > > > > > the problem you are describing. However, I > compared the > > > classes > > > > that > > > > > > were changed with PR > > > > > > (https://github.com/apache/cloudstack/pull/1176) > > > > > > and they are the same in ACS 4.9.2.0 and 4.9.3.0. > > > > > > > > > > > > On Tue, Oct 31, 2017 at 6:36 PM, Lotic Lists < > > > li...@lotic.com.br> > > > > wrote: > > > > > > > > > > > > > Hi Rafael, thanks for test. > > > > > > > > > > > > > > I found the issue > > > > > > > https://issues.apache.org/ > > jira/browse/CLOUDSTACK-9025 > > > > > > > I think is a same case. I will up a new lab > > environment > > > now for > > > > > > > simulate the issue > > > > > > > > > > > > > > Regards. > > > > > > > Marcelo > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Rafael Weingärtner [mailto: > rafaelweingartner@ > > > gmail.com] > > > > > > > Sent: segunda-feira, 30 de outubro de 2017 > 12:08 > > > > > > > To: us...@cloudstack.apache.org > > > > > > > Subject: Re: Strange size of template from > snapshot > > on > > > XenServer > > > > > > > > > > > > > > I just did, and the size of the template is > the size > > of > > > the root > > > > > > > disk > > > > > > > + the 1GB file I created in the test vm. > > > > > > > > > > > > > > The system I tested this is an ACS 4.9.2, > XenServer > > 6.5, > > > Primary > > > > > > > Stg ISCSI > > > > > > > > > > > > > > On Mon, Oct 30, 2017 at 10:20 AM, Lotic Lists < > > > > li...@lotic.com.br> > > > > > > wrote: > > > > > > > > > > > > > > > Good morning > > > > > > > > > > > > > > > > Guys, who could execute the test below? > > > > > > > > > > > > > > > > 1. Create a manual snapshot of a volume 2. > Create > > a file > > > (1GB) > > > > > > > > with dd on VM 3. Create second snapshot of a > > volume 4. > > > Convert > > > > > > > > latest snapshot to a template 5. Verify the > size > > of vhd > > > of new > > > > > > > > template in secondary storage. > > > > > > > > > > > > > > > > Here the template have 1GB, exactly the size > of > > latest > > > > snapshot. > > > > > > > > > > > > > > > > ACS 4.9.3, XenServer 6.5, Primary Stg ISCSI > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Lotic Lists [mailto:li...@lotic.com.br > ] > > > > > > > > Sent: quinta-feira, 26 de outubro de 2017 > 15:03 > > > > > > > > To: us...@cloudstack.apache.org > > > > > > > > Subject: RE: Strange size of template from > > snapshot on > > > > XenServer > > > > > > > > > > > > > > > > I test the same case on other environment, > create > > > template from > > > > > > > > snapshot, acs 4.9.2.0 and XenServer 6.5/iscsi > > > > > > > > > > > > > > > > 1. Was created two snapshots > > > > > > > > > > > > > > > > # ls -lh /var/cloud_mount/e5921d9a- > > > > 70b0-3501-9fda-6097e721fffb/ > > > > > > > > snapshots/131/617/ > > > > > > > > total 3.0G > > > > > > > > -rw-r--r-- 1 4294967294 root 13G Oct 26 > 13:46 > > > > > > > > 42c7c51d-4aa7-40d2-b5eb- cb56fc963974.vhd > > > > > > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 > 14:42 > > > > > > > > 53b33a5a-a7d9-4efa-b126- 38eec5482b05.vhd > > > > > > > > > > > > > > > > https://i.imgur.com/cSHZAJ3.png > > > > > > > > > > > > > > > > 2. create a template from latest snapshot > > > > > > > > > > > > > > > > # ls -lh > > > > > > > > total 37M > > > > > > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 > 14:43 > > > > > > > > 637604a6-3085-444e-ba09- b9aab11a2c16.vhd > > > > > > > > -rw-r--r-- 1 4294967294 root 303 Oct 26 > 14:43 > > > > > > > > template.properties > > > > > > > > > > > > > > > > # cat template.properties > > > > > > > > #Thu Oct 26 16:43:54 UTC 2017 > > > > > > > > filename=637604a6-3085-444e- > ba09-b9aab11a2c16.vhd > > > > > > > > id=1 > > > > > > > > vhd=true > > > > > > > > vhd.filename=637604a6-3085- > > 444e-ba09-b9aab11a2c16.vhd > > > > > > > > public=true > > > > > > > > uniquename=637604a6-3085-444e- > ba09-b9aab11a2c16 > > > > > > > > vhd.virtualsize=21474836480 > > > > > > > > virtualsize=21474836480 > > > > > > > > hvm= > > > > > > > > vhd.size=1197752832 > > > > > > > > size=1197752832 > > > > > > > > > > > > > > > > # md5sum /var/cloud_mount/e5921d9a- > > > > 70b0-3501-9fda-6097e721fffb/ > > > > > > > > snapshots/131/617/53b33a5a- > > a7d9-4efa-b126-38eec5482b05. > > > vhd > > > > > > > > 9d2078a6f41ca5e1eab824848c06df53 > > > /var/cloud_mount/e5921d9a- > > > > > > > > 70b0-3501-9fda-6097e721fffb/ > > snapshots/131/617/53b33a5a- > > > > > > > > a7d9-4efa-b126-38eec5482b05.vhd > > > > > > > > > > > > > > > > # md5sum /var/cloud_mount/e5921d9a- > > > > 70b0-3501-9fda-6097e721fffb/ > > > > > > > > template/tmpl/2/294/637604a6- > > > 3085-444e-ba09-b9aab11a2c16.vhd > > > > > > > > 9d2078a6f41ca5e1eab824848c06df53 > > > /var/cloud_mount/e5921d9a- > > > > > > > > 70b0-3501-9fda-6097e721fffb/ > > > template/tmpl/2/294/637604a6- > > > > > > > > 3085-444e-ba09-b9aab11a2c16.vhd > > > > > > > > > > > > > > > > ACS not merge files of snapshots in a single > file > > for > > > template, > > > > > > > > I think it's a bug. > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Rafael Weingärtner [mailto: > > rafaelweingartner@ > > > gmail.com] > > > > > > > > Sent: quarta-feira, 25 de outubro de 2017 > 22:51 > > > > > > > > To: us...@cloudstack.apache.org > > > > > > > > Subject: Re: Strange size of template from > > snapshot on > > > > XenServer > > > > > > > > > > > > > > > > It is easy to test, get one of these files > and try > > to > > > use them > > > > > > > > in Virtualbox. If they run, they do not > require > > anything > > > else. > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 10:49 PM, Lotic Lists > > > > > > > > <li...@lotic.com.br> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Sure but files on secondary storage don't > have > > > parents, I > > > > > > > > > don't know if ACS can connect standalone > files. > > > > > > > > > > > > > > > > > > vhd-util scan -f -m'*.vhd' -p > > > > > > > > > vhd=14818597-55bb-49be-9ace- > 761e9e01c074.vhd > > > > > > > > > capacity=311385128960 > > > > > > > > > size=304124744192 hidden=0 parent=none > > > > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe- > f262396a2d84.vhd > > > > > > > > > capacity=311385128960 > > > > > > > > > size=17770869248 hidden=0 parent=none > > > > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e- > fd0ea061cb2d.vhd > > > > > > > > > capacity=311385128960 > > > > > > > > > size=52199817728 hidden=0 parent=none > > > > > > > > > vhd=4b450217-c9ad-45a2-946b- > de3cb323469b.vhd > > > > > > > > > capacity=311385128960 > > > > > > > > > size=51615670784 hidden=0 parent=none > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Rafael Weingärtner [mailto: > > rafaelweingartner@ > > > gmail.com > > > > ] > > > > > > > > > Sent: quarta-feira, 25 de outubro de 2017 > 22:35 > > > > > > > > > To: us...@cloudstack.apache.org > > > > > > > > > Subject: Re: Strange size of template from > > snapshot on > > > > > > > > > XenServer > > > > > > > > > > > > > > > > > > If it was not executing coalesce you would > see > > the > > > parent > > > > > > > > > reference when you listed the hierarchy, > right? > > > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 10:25 PM, Lotic > Lists > > > > > > > > > <li...@lotic.com.br> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I think ACS preserve all VHDs for > coalesce but > > > coalesce not > > > > > > > > > > occur when create a template or before > > starting next > > > > > > > > > > snapshot > > > > > > scheduled. > > > > > > > > > > If I create a template from any > snapshots (2) > > > showing in > > > > > > > > > > GUI, the template have 49GB. 49 GB is > the size > > of > > > the 2 > > > > > > > > > > latest files on secondary > > > > > > > > > storage. > > > > > > > > > > > > > > > > > > > > What do you think? Bug? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Rafael Weingärtner > > > > > > > > > > [mailto:rafaelweingart...@gmail.com] > > > > > > > > > > Sent: quarta-feira, 25 de outubro de > 2017 22:07 > > > > > > > > > > To: us...@cloudstack.apache.org > > > > > > > > > > Subject: Re: Strange size of template > from > > snapshot > > > on > > > > > > > > > > XenServer > > > > > > > > > > > > > > > > > > > > Aha, The two destroyed entries are > showing " > > removed: > > > > NULL". > > > > > > > > > > > > > > > > > > > > In ACS one thing is the "destroying" of a > > resource, > > > another > > > > > > > > > > is the removal of this resource from the > > system. I > > > believe > > > > > > > > > > there is something to do with the expunge > > interval, > > > but I > > > > am > > > > > > > > > > not sure if the snapshots complete > removal also > > > happens > > > > > > > > > > within the expunge interval of user VMs > or if > > it is > > > > > > > > > > configured by something > > > > > else. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 10:03 PM, Lotic > Lists > > > > > > > > > > <li...@lotic.com.br> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I not start manual snapshot, just > scheduled > > with > > > keep=2 > > > > > > > > > > > > > > > > > > > > > > GUI showing 2 snapshots, database > showing 4 > > > entries, 2 > > > > > > > > > > > Destroyed and > > > > > > > > > > > 2 BackedUp > > > > > > > > > > > > > > > > > > > > > > *************************** 132. row > > > > > *************************** > > > > > > > > > > > id: 6380 > > > > > > > > > > > data_center_id: 2 > > > > > > > > > > > account_id: 64 > > > > > > > > > > > domain_id: 49 > > > > > > > > > > > volume_id: 2102 > > > > > > > > > > > disk_offering_id: 230 > > > > > > > > > > > status: Destroyed > > > > > > > > > > > path: NULL > > > > > > > > > > > name: FullCRM_ROOT-542_ > > 20171022030205 > > > > > > > > > > > uuid: > 1294cdb7-01bf-4e76-b4c0- > > > 11b3c6b395e9 > > > > > > > > > > > snapshot_type: 4 > > > > > > > > > > > type_description: DAILY > > > > > > > > > > > size: 311385128960 > > > > > > > > > > > created: 2017-10-22 03:02:05 > > > > > > > > > > > removed: NULL > > > > > > > > > > > backup_snap_id: NULL > > > > > > > > > > > swift_id: NULL > > > > > > > > > > > sechost_id: NULL > > > > > > > > > > > prev_snap_id: NULL > > > > > > > > > > > hypervisor_type: XenServer > > > > > > > > > > > version: 2.2 > > > > > > > > > > > s3_id: NULL > > > > > > > > > > > min_iops: NULL > > > > > > > > > > > max_iops: NULL > > > > > > > > > > > *************************** 133. row > > > > > *************************** > > > > > > > > > > > id: 6416 > > > > > > > > > > > data_center_id: 2 > > > > > > > > > > > account_id: 64 > > > > > > > > > > > domain_id: 49 > > > > > > > > > > > volume_id: 2102 > > > > > > > > > > > disk_offering_id: 230 > > > > > > > > > > > status: Destroyed > > > > > > > > > > > path: NULL > > > > > > > > > > > name: FullCRM_ROOT-542_ > > 20171023030205 > > > > > > > > > > > uuid: > 0a8067a3-ad15-40b6-8b16- > > > c1881d7d9f41 > > > > > > > > > > > snapshot_type: 4 > > > > > > > > > > > type_description: DAILY > > > > > > > > > > > size: 311385128960 > > > > > > > > > > > created: 2017-10-23 03:02:05 > > > > > > > > > > > removed: NULL > > > > > > > > > > > backup_snap_id: NULL > > > > > > > > > > > swift_id: NULL > > > > > > > > > > > sechost_id: NULL > > > > > > > > > > > prev_snap_id: NULL > > > > > > > > > > > hypervisor_type: XenServer > > > > > > > > > > > version: 2.2 > > > > > > > > > > > s3_id: NULL > > > > > > > > > > > min_iops: NULL > > > > > > > > > > > max_iops: NULL > > > > > > > > > > > *************************** 134. row > > > > > *************************** > > > > > > > > > > > id: 6455 > > > > > > > > > > > data_center_id: 2 > > > > > > > > > > > account_id: 64 > > > > > > > > > > > domain_id: 49 > > > > > > > > > > > volume_id: 2102 > > > > > > > > > > > disk_offering_id: 230 > > > > > > > > > > > status: BackedUp > > > > > > > > > > > path: NULL > > > > > > > > > > > name: FullCRM_ROOT-542_ > > 20171024030207 > > > > > > > > > > > uuid: > 1ca646e5-a155-4a59-aa31- > > > 5889eec7536f > > > > > > > > > > > snapshot_type: 4 > > > > > > > > > > > type_description: DAILY > > > > > > > > > > > size: 311385128960 > > > > > > > > > > > created: 2017-10-24 03:02:07 > > > > > > > > > > > removed: NULL > > > > > > > > > > > backup_snap_id: NULL > > > > > > > > > > > swift_id: NULL > > > > > > > > > > > sechost_id: NULL > > > > > > > > > > > prev_snap_id: NULL > > > > > > > > > > > hypervisor_type: XenServer > > > > > > > > > > > version: 2.2 > > > > > > > > > > > s3_id: NULL > > > > > > > > > > > min_iops: NULL > > > > > > > > > > > max_iops: NULL > > > > > > > > > > > *************************** 135. row > > > > > *************************** > > > > > > > > > > > id: 6497 > > > > > > > > > > > data_center_id: 2 > > > > > > > > > > > account_id: 64 > > > > > > > > > > > domain_id: 49 > > > > > > > > > > > volume_id: 2102 > > > > > > > > > > > disk_offering_id: 230 > > > > > > > > > > > status: BackedUp > > > > > > > > > > > path: NULL > > > > > > > > > > > name: FullCRM_ROOT-542_ > > 20171025030208 > > > > > > > > > > > uuid: > 759d61dc-e3db-4815-b68b- > > > b90904ccbda5 > > > > > > > > > > > snapshot_type: 4 > > > > > > > > > > > type_description: DAILY > > > > > > > > > > > size: 311385128960 > > > > > > > > > > > created: 2017-10-25 03:02:08 > > > > > > > > > > > removed: NULL > > > > > > > > > > > backup_snap_id: NULL > > > > > > > > > > > swift_id: NULL > > > > > > > > > > > sechost_id: NULL > > > > > > > > > > > prev_snap_id: NULL > > > > > > > > > > > hypervisor_type: XenServer > > > > > > > > > > > version: 2.2 > > > > > > > > > > > s3_id: NULL > > > > > > > > > > > min_iops: NULL > > > > > > > > > > > max_iops: NULL > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: Rafael Weingärtner > > > > > > > > > > > [mailto:rafaelweingart...@gmail.com] > > > > > > > > > > > Sent: quarta-feira, 25 de outubro de > 2017 > > 21:50 > > > > > > > > > > > To: us...@cloudstack.apache.org > > > > > > > > > > > Subject: Re: Strange size of template > from > > > snapshot on > > > > > > > > > > > XenServer > > > > > > > > > > > > > > > > > > > > > > The file "14818597-55bb-49be-9ace- > > > 761e9e01c074.vhd" > > > > seems > > > > > > > > > > > more like a manual snapshot. It was > created 2 > > > hours after > > > > > > > > > > > the > > > > > "normal" > > > > > > > > > > > time. I would check these snapshots in > ACS > > and see > > > how > > > > > > > > > > > they are > > > > > > > > presented. > > > > > > > > > > > BTW, how many snapshots is ACS showing > for > > this > > > volume? I > > > > > > > > > > > would also check the "snapshots" table > for > > the > > > volume you > > > > > > > > > > > have configured > > > > > > > > them. > > > > > > > > > > > select * from snapshots where > volume_id = ? > > > > > > > > > > > > > > > > > > > > > > you may also filter for removed: > > > > > > > > > > > select * from snapshots where > volume_id = ? > > and > > > removed > > > > is > > > > > > > > > > > null > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 9:39 PM, Lotic > Lists > > > > > > > > > > > <li...@lotic.com.br> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Rafael, I check > > > > > > > > > > > > > > > > > > > > > > > > # ls -lrt > > > > > > > > > > > > total 285368382 > > > > > > > > > > > > -rw-r--r-- 1 root root 304124744192 > Oct 22 > > 03:28 > > > > > > > > > > > > 14818597-55bb-49be-9ace- > 761e9e01c074.vhd > > > > > > > > > > > > -rw-r--r-- 1 root root 17770869248 > Oct 23 > > 01:10 > > > > > > > > > > > > bbd9e9b5-53fb-44a2-afbe- > f262396a2d84.vhd > > > > > > > > > > > > -rw-r--r-- 1 root root 52199817728 > Oct 24 > > 01:30 > > > > > > > > > > > > 3b90bbb2-7ce5-41f6-9f7e- > fd0ea061cb2d.vhd > > > > > > > > > > > > -rw-r--r-- 1 root root 51615670784 > Oct 25 > > 01:28 > > > > > > > > > > > > 4b450217-c9ad-45a2-946b- > de3cb323469b.vhd > > > > > > > > > > > > > > > > > > > > > > > > # vhd-util scan -f -m'*.vhd' -p > > > > > > > > > > > > vhd=14818597-55bb-49be-9ace- > > 761e9e01c074.vhd > > > > > > > > > > > > capacity=311385128960 > > > > > > > > > > > > size=304124744192 hidden=0 > parent=none > > > > > > > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe- > > f262396a2d84.vhd > > > > > > > > > > > > capacity=311385128960 > > > > > > > > > > > > size=17770869248 hidden=0 parent=none > > > > > > > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e- > > fd0ea061cb2d.vhd > > > > > > > > > > > > capacity=311385128960 > > > > > > > > > > > > size=52199817728 hidden=0 parent=none > > > > > > > > > > > > vhd=4b450217-c9ad-45a2-946b- > > de3cb323469b.vhd > > > > > > > > > > > > capacity=311385128960 > > > > > > > > > > > > size=51615670784 hidden=0 parent=none > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > From: Rafael Weingärtner > > > > > > > > > > > > [mailto:rafaelweingart...@gmail.com] > > > > > > > > > > > > Sent: quarta-feira, 25 de outubro de > 2017 > > 20:44 > > > > > > > > > > > > To: us...@cloudstack.apache.org > > > > > > > > > > > > Subject: Re: Strange size of > template from > > > snapshot on > > > > > > > > > > > > XenServer > > > > > > > > > > > > > > > > > > > > > > > > Did you check the hierarchy of those > > snapshot > > > files in > > > > > > > > > > > > your secondary storage? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 7:44 PM, > Lotic > > Lists > > > > > > > > > > > > <li...@lotic.com.br> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anyone identify problems with > templates > > from > > > > snapshots > > > > > > > > > > > > > on > > > > > > > > > XenServer? > > > > > > > > > > > > > > > > > > > > > > > > > > I created a recurrent snapshot, > first > > VHD on > > > > secondary > > > > > > > > > > > > > storage have a similar size of > volume. > > If I > > > create a > > > > > > > > > > > > > template from latest snapshot, the > size > > of VHD > > > of > > > > > > > > > > > > > template is smaller equal from last > > snapshot, > > > I think > > > > > > > > > > > > > CloudStack/Xenserver not coalesce > VHD > > files of > > > > > > > > > > > > > secondary storage for create the > template > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I keep two recurrent snapshots, on > > secondary > > > storage > > > > > > > > > > > > > have four > > > > > > > > > files. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Environment is ACS 4.9.3.0 and > XenServer > > 6.5, > > > primary > > > > > > > > > > > > > storage is PreSetup with ISCSI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > Marcelo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > -- > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > DISCLAIMER > > > > ========== > > > > This e-mail may contain privileged and confidential > information > > > which is > > > > the property of Accelerite, a Persistent Systems > business. It > > is > > > intended > > > > only for the use of the individual or entity to which it > is > > > addressed. If > > > > you are not the intended recipient, you are not > authorized to > > read, > > > retain, > > > > copy, print, distribute or use this message. If you have > > received > > > this > > > > communication in error, please notify the sender and > delete all > > > copies of > > > > this message. Accelerite, a Persistent Systems business > does > > not > > > accept any > > > > liability for virus infected mails. > > > > > > > > > > > > > > > > -- > > > Rafael Weingärtner > > > > > > > > > DISCLAIMER > > > ========== > > > This e-mail may contain privileged and confidential information > > which is > > > the property of Accelerite, a Persistent Systems business. It > is > > intended > > > only for the use of the individual or entity to which it is > > addressed. If > > > you are not the intended recipient, you are not authorized to > read, > > retain, > > > copy, print, distribute or use this message. If you have > received > > this > > > communication in error, please notify the sender and delete all > > copies of > > > this message. Accelerite, a Persistent Systems business does > not > > accept any > > > liability for virus infected mails. > > > > > > > > > > > -- > > Rafael Weingärtner > > > > > > DISCLAIMER > > ========== > > This e-mail may contain privileged and confidential information > which is > > the property of Accelerite, a Persistent Systems business. It is > intended > > only for the use of the individual or entity to which it is > addressed. If > > you are not the intended recipient, you are not authorized to read, > retain, > > copy, print, distribute or use this message. If you have received > this > > communication in error, please notify the sender and delete all > copies of > > this message. Accelerite, a Persistent Systems business does not > accept any > > liability for virus infected mails. > > > > > > -- > Rafael Weingärtner > > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is > the property of Accelerite, a Persistent Systems business. It is intended > only for the use of the individual or entity to which it is addressed. If > you are not the intended recipient, you are not authorized to read, retain, > copy, print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Accelerite, a Persistent Systems business does not accept any > liability for virus infected mails. > -- Rafael Weingärtner
