Re: Snapshots only on Primary Storage feature

[Simon Weller]

yeah, I like that idea Mike.


Also, I want to be clear that breaking out the backup to secondary into a 
separate thread is a great feature, so kudos to those that developed it.  That 
along with the ability to turn it off completely makes for a nice overall 
improvement to snapshots.


- Si


From: Tutkowski, Mike 
Sent: Saturday, May 19, 2018 8:40 PM
To: dev@cloudstack.apache.org
Subject: Re: Snapshots only on Primary Storage feature

Perhaps instead of renaming the setting, we can note in its description the 
hypervisors it currently pertains to.

> On May 19, 2018, at 7:03 PM, Glen Baars  wrote:
>
> Based on the responses, I think it is a worthy feature to be retained. Maybe 
> the following changes?
>
> Rename the setting to something like kvmxen.snapshot.primaryonly ( I have no 
> idea of the naming scheme that Cloudstack uses )
> Ensure the code for vmware snapshots does not get impacted by the setting
> Record in the DB that the snapshot is only on the primary storage
> When the create template or download template features are used, use the 
> primary storage as the source.
>
> Kind regards,
> Glen Baars
>
> -Original Message-
> From: Will Stevens 
> Sent: Saturday, 19 May 2018 12:57 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Snapshots only on Primary Storage feature
>
> I think reverting the change in 4.11.1 is probably a good idea.
>
> *Will Stevens*
> Chief Technology Officer
> c 514.826.0190
>
> 
>
>
> On Fri, May 18, 2018 at 2:52 PM ilya musayev 
> wrote:
>
>> Perhaps bring it back into 4.11.1?
>>
>> On Fri, May 18, 2018 at 9:28 AM Suresh Kumar Anaparti <
>> sureshkumar.anapa...@gmail.com> wrote:
>>
>>> Si / Will,
>>>
>>> That is just FYI, if anyone uses VMware with that flag set to false.
>>> I'm neither against the feature nor telling to rip that out.
>>>
>>> You are correct, the PR 2081 supports KVM and Xen as the volume
>>> snapshots are directly supported on them and backup operation is not
>>> tightly
>> coupled
>>> with the create operation.
>>>
>>> -Suresh
>>>
>>> On Fri, May 18, 2018 at 7:38 PM, Simon Weller
>>> 
>>> wrote:
>>>
 There are plenty of features in ACS that are particular to a
 certain hypervisor (or hypervisor set), including VMware specific items.

 It was never claimed this feature worked across all hypervisors.
 In addition to that, the default was to leave the existing
 functionality exactly the way it was originally implemented and if
 a user wished to change the functionality they could via a global config 
 variable.

 Your original spec for PR 2081 in confluence states that the PR
 was targeted towards KVM and Xen, so I'm confused as to why VMware
 is even being mentioned here.


 This is a major feature regression that a number of
>> organizations/service
 providers are relying on and it wasn't called out when the PR was
>>> submitted.


 
 From: Will Stevens 
 Sent: Friday, May 18, 2018 6:12 AM
 To: dev@cloudstack.apache.org
 Subject: Re: Snapshots only on Primary Storage feature

 Just because it does not work for VMware should not a reason to
 rip out
>>> the
 functionality for other hypervisors where it is being used though.

 I know we also have the requirement that snapshots are not
>> automatically
 replicated to secondary storage, so this feature is useful to us.

 I don't understand the rational for removing the feature just
 because
>> it
 does not work on VMware.

 On Fri, May 18, 2018, 6:27 AM Suresh Kumar Anaparti, <
 sureshkumar.anapa...@gmail.com> wrote:

> Si,
>
> The PR# 1697 with the global setting
> *snapshot.backup.rightafter** -
> false* doesn't
> work for VMware as create snapshot never takes a snapshot in
> Primary
 pool,
> it just returns the snapshot uuid. The backup snapshot does the
>>> complete
> job - creates a VM snapshot with the uuid, extracts and exports
> the
 target
> volume to secondary. On demand backup snapshot doesn't work as
> there
>> is
 no
> snapshot in primary. Also, there'll be only one entry with
> Primary
>>> store
> role in snapshot_store_ref, which is the latest snapshot taken
> for
>> that
> volume.
>
> -Suresh
>
> On Fri, May 18, 2018 at 1:03 AM, Simon Weller
>> >>>
> wrote:
>
>> The whole point of the original PR was to optionally disable
>> this functionality.
>>
>> We don't expose views of the backup state to our customers (we
>> have
>>> our
>> own customer interfaces) and it's a large waste of space for
>> us to
>> be
>> backing up tons of VM images when we have a solid primary
>> storage infrastructure that already has lots of resiliency.
>>
>>
>> I guess we're going to have to revisit this again before we

Can't build master

[Tutkowski, Mike]

Hi,

Did I miss an e-mail or something? I’m having trouble building master (below).

Thanks!
Mike

Running org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest
log4j:WARN No appenders could be found for logger 
(org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.296 sec <<< 
FAILURE! - in org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest
testConfigDriveBuild(org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest)
  Time elapsed: 0.278 sec  <<< ERROR!
com.cloud.utils.exception.CloudRuntimeException: Unable to create iso file: 
i-x-y.iso due to java.io.IOException: Cannot run program 
"/usr/bin/genisoimage": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at com.cloud.utils.script.Script.execute(Script.java:215)
at com.cloud.utils.script.Script.execute(Script.java:183)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder.buildConfigDrive(ConfigDriveBuilder.java:152)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest.testConfigDriveBuild(ConfigDriveBuilderTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 28 more

at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder.buildConfigDrive(ConfigDriveBuilder.java:156)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest.testConfigDriveBuild(ConfigDriveBuilderTest.java:56)


Results :

Tests in error:
  ConfigDriveBuilderTest.testConfigDriveBuild:56 » CloudRuntime Unable to 
create...

Tests run: 2, Failures: 0, Errors: 1, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CloudStack Developer Tools - Checkstyle Configuration SUCCESS [  
1.644 s]
[INFO] Apache CloudStack .. SUCCESS [  1.714 s]
[INFO] Apache CloudStack Framework - Managed Context .. SUCCESS [  5.043 s]
[INFO] Apache CloudStack Framework  SUCCESS [  0.213 s]
[INFO] Apache CloudStack Framework - Certificate Authority  SUCCESS [  1.50

Re: Can't build master

[Tutkowski, Mike]

Just an FYI that this is on OS X Version 10.11.6.

From: "Tutkowski, Mike" 
Date: Monday, May 21, 2018 at 2:42 PM
To: "dev@cloudstack.apache.org" 
Subject: Can't build master

Hi,

Did I miss an e-mail or something? I’m having trouble building master (below).

Thanks!
Mike

Running org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest
log4j:WARN No appenders could be found for logger 
(org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.296 sec <<< 
FAILURE! - in org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest
testConfigDriveBuild(org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest)
  Time elapsed: 0.278 sec  <<< ERROR!
com.cloud.utils.exception.CloudRuntimeException: Unable to create iso file: 
i-x-y.iso due to java.io.IOException: Cannot run program 
"/usr/bin/genisoimage": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at com.cloud.utils.script.Script.execute(Script.java:215)
at com.cloud.utils.script.Script.execute(Script.java:183)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder.buildConfigDrive(ConfigDriveBuilder.java:152)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest.testConfigDriveBuild(ConfigDriveBuilderTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 28 more

at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder.buildConfigDrive(ConfigDriveBuilder.java:156)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest.testConfigDriveBuild(ConfigDriveBuilderTest.java:56)


Results :

Tests in error:
  ConfigDriveBuilderTest.testConfigDriveBuild:56 » CloudRuntime Unable to 
create...

Tests run: 2, Failures: 0, Errors: 1, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CloudStack Developer Tools - Checkstyle Configuration SUCCESS [  
1.644 s]
[INFO] Apache CloudStack .. SUCCESS [  1.714 s]
[INFO] Apache CloudStack Framework - Managed Context ..

Re: Can't build master

[Rohit Yadav]

Hi Mike,


Is genisoimage or mkisofs available on osx? This is usually installed at 
/usr/bin/ on CentOS6/CentOS7/Ubuntu Linux. Can you try brew or something else 
to install it?

They are also used by injectkeys.sh/.py when the management server starts. The 
change is part of a recent PR I did and added a unit test for it where it tries 
to build a config drive ISO file. If genisoimage is not availabe on OSX, we can 
add some environment check to the unit test to skip on non-Linux environments.


- Rohit






From: Tutkowski, Mike 
Sent: Tuesday, May 22, 2018 2:13:23 AM
To: dev@cloudstack.apache.org
Subject: Re: Can't build master

Just an FYI that this is on OS X Version 10.11.6.

From: "Tutkowski, Mike" 
Date: Monday, May 21, 2018 at 2:42 PM
To: "dev@cloudstack.apache.org" 
Subject: Can't build master

Hi,

Did I miss an e-mail or something? I’m having trouble building master (below).

Thanks!
Mike

Running org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest
log4j:WARN No appenders could be found for logger 
(org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.296 sec <<< 
FAILURE! - in org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest
testConfigDriveBuild(org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest)
  Time elapsed: 0.278 sec  <<< ERROR!
com.cloud.utils.exception.CloudRuntimeException: Unable to create iso file: 
i-x-y.iso due to java.io.IOException: Cannot run program 
"/usr/bin/genisoimage": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at com.cloud.utils.script.Script.execute(Script.java:215)
at com.cloud.utils.script.Script.execute(Script.java:183)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilder.buildConfigDrive(ConfigDriveBuilder.java:152)
at 
org.apache.cloudstack.storage.configdrive.ConfigDriveBuilderTest.testConfigDriveBuild(ConfigDriveBuilderTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 28 more

at 
org.apache.cloudstack.storage.configdrive.Co