RE: [PROPOSAL] Storage Subsystem API Interface Additions

[Alex Huang]

That's my read on the proposal also but, Chris, please clarify.  I don't think 
the end user will see the change.  It's an optimization for interfacing with 
the storage backend.

--Alex

> -Original Message-
> From: Marcus Sorensen [mailto:shadow...@gmail.com]
> Sent: Wednesday, September 18, 2013 9:22 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Storage Subsystem API Interface Additions
> 
> Perhaps he needs to elaborate on the use case and what he means by more
> efficient.  He may be referring to multiple volumes in the sense of
> snapshotting the ROOT disks for 10 different VMs.
> 
> On Wed, Sep 18, 2013 at 10:10 AM, Darren Shepherd
>  wrote:
> > Here's my general concern about multiple volume snapshots at once.
> > Giving such a feature leads the user to believe that snapshotting
> > multiple volumes at once will give them consistency across the volumes in
> the snapshot.
> > This is not true, and difficult to do with many hypervisors, and
> > typically requires an agent in the VM.  A single snapshot, as exists
> > today, is really crash consistent, meaning that there is may exist
> > unsync'd data.  To do a true multi volume snapshot requires a "quiesce"
> functionality in the VM.
> > So you do pause I/O queues, fsync, fsync, snapshot, snapshot, unpause I/O.
> >
> > I'm might be fine with the option of allowing multiple volumeId's to
> > be specified in the snapshot API, but it needs to be clear that those
> > snapshots may be taken sequentially and they are all independently
> > crash consistent.  But, if you make that clear, then why even have the API.
> > Essentially it is the same as doing multiple snapshot API commands.
> >
> > So really I would lean towards having the multiple snapshotting
> > supported in the driver or storage subsystem, but not exposed to the
> > user.  You can easy accomplish it by having a timed window on
> > snapshotting.  So every 10 seconds you do snapshots, if 5 requests
> > have queued in the last 10 seconds, you do them all at once.  This could be
> implemented as a framework thing.
> > If your provider implements "SnapshotBatching" interface and that has
> > a getBatchWindowTime(), then the framework can detect that it should
> > try to queue up some snapshot requests and send them to the driver in
> > a batch.  Or that could be implemented in the driver itself.  I would
> > lean toward doing it in the driver and if that goes well, we look at
> > pulling the functionality into core ACS.
> >
> > Darren
> >
> >
> > On Wed, Sep 18, 2013 at 5:22 AM, SuichII, Christopher <
> > chris.su...@netapp.com> wrote:
> >
> >> I would like to raise for discussion the idea of adding a couple
> >> methods to the Storage Subsystem API interface. Currently,
> >> takeSnapshot() and
> >> revertSnapshot() only support single VM volumes. We have a use case
> >> for snapshotting multiple VM volumes at the same time. For us, it is
> >> more efficient to snapshot them all at once rather than snapshot VM
> >> Volumes individually and this seems like a more elegant solution than
> >> queueing the requests within our plugin.
> >>
> >> Base on my investigation, this should require:
> >> -Two additional API to be invoked from the UI -Two additional methods
> >> added to the Storage Subsystem API interface -Changes in between the
> >> API level and invoking the Storage Subsystem API implementations (I
> >> know this is broad and vague), mainly around the SnapshotManger/Impl
> >>
> >> There are a couple topics we would like discussion on:
> >> -Would this be beneficial/detrimental/neutral to other storage providers?
> >> -How should we handle the addition of new methods to the Storage
> >> Subsystem API interface? Default them to throw an
> UnsupportedOperationException?
> >> Default to calling the single VM volume version multiple times?
> >> -Does anyone see any issues with allowing multiple snapshots to be
> >> taken at the same time or letting storage providers have a list of
> >> all the requested volumes to backup?
> >>
> >> Please let me know if I've missed any major topics for discussion or
> >> if anything needs clarification.
> >>
> >> Thanks,
> >> Chris
> >> --
> >> Chris Suich
> >> chris.su...@netapp.com
> >> NetApp Software Engineer
> >> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat
> >>
> >>

RE: [DISCUSS] Java 7, tomcat 7 and further upgrades

[Alex Huang]

I don't have much problem with switching to jdk1.7.  My eclipse is running with 
jdk1.7 as the builder and it can't find any problems in cs code.  The main 
question I think will come from the Linux variants.  Are all of them shipping 
with jdk1.7 now?

--Alex

> -Original Message-
> From: Trippie [mailto:trip...@gmail.com] On Behalf Of Hugo Trippaers
> Sent: Wednesday, September 18, 2013 5:10 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] Java 7, tomcat 7 and further upgrades
> 
> Hey all,
> 
> Sorry for the threadomancy, but the discussion have become relevant again
> with the current issues with the libvirt library. Of course this could also be
> solved by updating the libvirt library with a jdk6 version. Still it might be 
> good
> to revisit this topic.
> 
> It appears not to be possible to switch code style to 1.7 and produce a 1.6
> compatible binary. I remember this working with olders versions, but didn't
> dig to much into this.
> 
> So the new question in this thread will be, should we switch CloudStack to
> jdk 1.7?
> 
> Cheers,
> 
> Hugo
> 
> On Jul 1, 2013, at 12:45 AM, Prasanna Santhanam  wrote:
> 
> > On Thu, Jun 27, 2013 at 04:18:40PM -0400, John Burwell wrote:
> >> All,
> >>
> >> I am +1 for Java7.  However, I would like to propose ridding
> >> ourselves of Tomcat entirely and embedding a network stack such as
> >> Netty (http://netty.io) with a servlet bridge.  We have one JSP in
> >> the system that generates JSON resources.  It could be easily
> >> eliminated with a simple servlet that generates JSON from a
> >> ResourceBundle.  Outside of this JSP. I don't see any other
> >> requirements for a JEE container besides hosting a servlet.  We would
> >> gain a far simpler, self-contained deployment model (a single jar).
> >> Additionally, we would gain greater control of the startup and
> >> shutdown lifecycle, as well as, threading dynamics.  If there is
> >> interest in this approach, I have thoughts on how to achieve this
> >> embedding and create a lightweight daemon framework that could be
> >> used for all CloudStack daemons.
> >>
> >> As an aside, I also think we should replace our hand-rolled NIO code
> >> with Netty as well.
> >>
> >
> > John - could you break this and other thoughts down a little more in
> > what's involved?  Perhaps into its own thread. I don't know Netty. And
> > my J2EE is shaky at best.
> >
> > It's been a previous wish on this list to have the packaging of
> > cloudstack into a single easily deployable war instead of all the
> > complicated packaging we do. So I'd like to hear more of that and
> > other issues you describe.
> >
> > --
> > Prasanna.,
> >
> > 
> > Powered by BigRock.com
> >

RE: Apache CloudStack 4.2.0 (fifth round)

[Alex Huang]

+1(binding) followed release procedure and tested on devCloud.

--Alex

> -Original Message-
> From: Animesh Chaturvedi [mailto:animesh.chaturv...@citrix.com]
> Sent: Friday, September 13, 2013 4:13 PM
> To: dev@cloudstack.apache.org
> Subject: Apache CloudStack 4.2.0 (fifth round)
> 
> 
> I've created a 4.2.0 release, with the following artifacts up for a vote:
> 
> Git Branch and Commit SH:
> https://git-wip-
> us.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/4.2
> Commit: c1e24ff89f6d14d6ae74d12dbca108c35449030f
> 
> List of changes:
> https://git-wip-
> us.apache.org/repos/asf?p=cloudstack.git;a=blob_plain;f=CHANGES;hb=4.2
> 
> Source release (checksums and signatures are available at the same
> location):
> https://dist.apache.org/repos/dist/dev/cloudstack/4.2.0/
> 
> PGP release keys (signed using 94BE0D7C):
> https://dist.apache.org/repos/dist/release/cloudstack/KEYS
> 
> Testing instructions are here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+test+pr
> ocedure
> 
> Vote will be open for 72 hours (Wednesday 9/18 End of Day PST).
> 
> For sanity in tallying the vote, can PMC members please be sure to indicate
> "(binding)" with their vote?
> 
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)

RE: Removal of cloud-plugin-netapp

[Alex Huang]

I think it should be removed.

--Alex

> -Original Message-
> From: SuichII, Christopher [mailto:chris.su...@netapp.com]
> Sent: Wednesday, September 18, 2013 5:26 AM
> To: 
> Subject: Removal of cloud-plugin-netapp
> 
> I remember seeing/hearing some discussion about removing the cloud-
> plugin-netapp component since it is not used or really managed any more. Is
> this still in the works? One of the dependencies (manageontap.jar) will
> conflict with one of the dependencies of our (in development) plugin. The
> version of manageontap.jar used by CloudStack is different than what we
> require in our plugin and would likely cause runtime issues if still included 
> in
> the MS classpath.
> 
> Any thoughts?
> 
> Thanks,
> Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms - Cloud Solutions
> Citrix, Cisco & Red Hat

RE: conflicting dependencies between CloudStack and Whirr

[Alex Huang]

I actually did a quick try to update cloudstack to use newest gson version 
about 3 months back.  Had to roll it back but I didn't try very hard though.  
Part of the reason why I decided to rollback is due to gson is used differently 
by various components in CloudStack and I didn't have time to get into each 
component to see if I break anything.  The other is also I thought using 
Jackson would be much better and thought our next attempt should be with 
Jackson.  

Not that any of these helps you.  Just know updating CloudStack to latest gson 
is not simple. 

--Alex

> -Original Message-
> From: Andrei Savu [mailto:savu.and...@gmail.com]
> Sent: Wednesday, September 18, 2013 10:53 AM
> To: d...@whirr.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: conflicting dependencies between CloudStack and Whirr
> 
> It's easy to usr jclouds and whirr inside an OSGi container - just add the
> feature url. Bonus: you can also use jclouds shell interface (part of jclouds 
> cli).
> 
> Another option is to upgrade the CloudStack API to use the new version.
> On Sep 18, 2013 5:14 AM, "Han,Meng"  wrote:
> 
> > Dear all,
> >
> > I am adding an API to CloudStack which utilizes Whirr to launch
> > various clusters on CloudStack. Now I am facing a dependency conflicting
> issue.
> >
> > Whirr 0.8.2 requires gson 2.2.2 while CloudStack API requires gson 1.7.1.
> > If I use gson 1.7.1 for Whirr, the following error will happen:
> >
> > com.google.common.util.**concurrent.ExecutionError:
> java.lang.**NoClassDefFoundError:
> > com/google/gson/TypeAdapter
> >
> > This TypeAdapter class can be found inside gson-2.2.2.jar. However if
> > I modify CloudStack to use gson 2.2.2 CloudStack will not build
> > successfully due to the following test error.
> >
> > [INFO] Building Apache CloudStack Core 4.1.1 [INFO]
> > --**--**
> > 
> > [INFO]
> > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ cloud-core
> > --- [INFO] Deleting /home/meng/cloudstack/core/**target
> > [INFO]
> > [INFO] --- maven-remote-resources-plugin:**1.3:process (default) @
> > cloud-core --- [INFO] [INFO] ---
> > maven-resources-plugin:2.5:**resources (default-resources) @
> > cloud-core --- [debug] execute contextualize [INFO] Using 'UTF-8'
> > encoding to copy filtered resources.
> > [INFO] skip non existing resourceDirectory
> > /home/meng/cloudstack/core/** src/main/resources [INFO] Copying 3
> > resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:**compile
> > (default-compile) @ cloud-core --- [INFO] Compiling 156 source files
> > to /home/meng/cloudstack/core/** target/classes [INFO] [INFO] ---
> > maven-resources-plugin:2.5:**testResources
> > (default-testResources) @ cloud-core --- [debug] execute contextualize
> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
> > [INFO] skip non existing resourceDirectory
> > /home/meng/cloudstack/core/** src/test/resources [INFO] Copying 3
> > resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:**testCompile
> > (default-testCompile) @ cloud-core --- [INFO] Compiling 1 source file
> > to /home/meng/cloudstack/core/** target/test-classes [INFO] [INFO] ---
> > maven-surefire-plugin:2.12:**test (default-test) @ cloud-core
> > ---
> > [INFO] Surefire report directory: /home/meng/cloudstack/core/**
> > target/surefire-reports
> >
> > --**-
> >  T E S T S
> > --**-
> > Running com.cloud.agent.transport.**RequestTest
> > log4j:WARN No appenders could be found for logger
> > (com.cloud.agent.transport.**RequestTest).
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN See
> http://logging.apache.org/**log4j/1.2/faq.html#noconfig che.org/log4j/1.2/faq.html#noconfig>for more info.
> > Tests run: 4, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 3.054
> > sec <<< FAILURE!
> >
> > Results :
> >
> > Failed tests:   testSerDeser(com.cloud.agent.**transport.RequestTest)
> >
> > Tests in error:
> >   testLogging(com.cloud.agent.**transport.RequestTest)
> >
> > Tests run: 4, Failures: 1, Errors: 1, Skipped: 0
> >
> >
> > I ran "mvn -P developer clean install -DskipTests" , the CloudStack
> > building process went through. But when I issued an API
> > --launchCluster to CloudStack, the following error related to gson
> > popped up on CloudStack Management Server:
> >
> > ERROR [agent.transport.Request] (AgentManager-Handler-2:) Caught
> > problem with
> > [{"StartupRoutingCommand":{"**cpus":2,"speed":3000,"memory":**
> > 3844370432,"dom0MinMemory":**384437043,"poolSync":false,"**
> > vms":{"v-2-VM":{"state":"**Running"},"s-1-VM":{"state":"**
> > Running"},"r-4-VM":{"state":"**Running"}},"caps":"hvm,**
> > snapshot","pool":"/root","**hypervisorType":"KVM","**
> >
> hostDetails":{"com.cloud.**network.Networks.**RouterPrivateIpStrategy"
> > :"**
> > HostLocal","Host.OS":"CentOS

RE: DatabaseUpgradeChecker and running from eclipse

[Alex Huang]

Around 4.1 timeframe, Rohit added DatabaseCreator to call 
DatabaseUpgradeChecker to create and upgrade the database but forgot to remove 
it from the server configuration.  We should simply add a DatabaseVesionChecker 
for the server configuration to check if the database is at the version 
expected for the code.  That was supposed to happen in 4.2 but never happened.  
It might be a good time to add that in now.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Thursday, September 19, 2013 9:35 AM
> To: dev
> Subject: Re: DatabaseUpgradeChecker and running from eclipse
> 
> On Thu, Sep 19, 2013 at 6:12 PM, Darren Shepherd
>  wrote:
> > Only to test builds before commit.  mvn is too slow.
> 
> That's probably it. I run from eclipse but usually do a build first.
> That is going to change one minute ago, thanks. i'll keep an eye on the
> dbupgradechecker

RE: DatabaseUpgradeChecker and running from eclipse

[Alex Huang]

Ah...got it.  Makes sense.

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, September 19, 2013 10:38 AM
> To: dev@cloudstack.apache.org
> Subject: Re: DatabaseUpgradeChecker and running from eclipse
> 
> Its getting the JDK version because of
> this.getClass().getSuperclass().getPackage().getImplementationVersion();
> which it only check if
> this.getClass().getPackage().getImplementationVersion() is null.  So parent is
> java.lang.Object, so it gets the JDK version.  I understand your really 
> looking
> for 4.3.0-SNAPSHOT that is put in the MANIFEST.MF in the mvn build.  The
> patch I put on review board is to remove the getSuperclass() check as that
> seems to be never a good idea.
> 
> Darren
> 
> 
> 
> On Thu, Sep 19, 2013 at 10:27 AM, Alex Huang 
> wrote:
> 
> > Before cloudstack was donated to apache, it had an open source version
> > and a non-open source portion (which included  vmware, f5, and
> > juniper).  Now, everything is open source but that division is
> > sometimes still in the files.  You can basically merge those files now.
> >
> > As for jdk version, that's not what's recording. The version that's
> > supposed to be recorded is CloudStack's version.  Not sure why through
> > eclipse, you're getting the jdk version.
> >
> > --Alex
> >
> > > -Original Message-
> > > From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> > > Sent: Thursday, September 19, 2013 10:11 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: DatabaseUpgradeChecker and running from eclipse
> > >
> > > So I just looked at DatabaseCreator and noticed it calls
> > > PremiumDatabaseUpgradeChecker, but on startup we call
> > > DatabaseUpgradeChecker.  What is this madness?  Why are there two db
> > > upgrade classes?
> > >
> > > Darren
> > >
> > >
> > > On Thu, Sep 19, 2013 at 10:04 AM, Alex Huang 
> > > wrote:
> > >
> > > > Around 4.1 timeframe, Rohit added DatabaseCreator to call
> > > > DatabaseUpgradeChecker to create and upgrade the database but
> > > > forgot to remove it from the server configuration.  We should
> > > > simply add a DatabaseVesionChecker for the server configuration to
> > > > check if the database is at the version expected for the code.
> > > > That was supposed to happen in
> > > > 4.2 but never happened.  It might be a good time to add that in now.
> > > >
> > > > --Alex
> > > >
> > > > > -Original Message-
> > > > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > > > > Sent: Thursday, September 19, 2013 9:35 AM
> > > > > To: dev
> > > > > Subject: Re: DatabaseUpgradeChecker and running from eclipse
> > > > >
> > > > > On Thu, Sep 19, 2013 at 6:12 PM, Darren Shepherd
> > > > >  wrote:
> > > > > > Only to test builds before commit.  mvn is too slow.
> > > > >
> > > > > That's probably it. I run from eclipse but usually do a build first.
> > > > > That is going to change one minute ago, thanks. i'll keep an eye
> > > > > on the dbupgradechecker
> > > >
> >

RE: DatabaseUpgradeChecker and running from eclipse

[Alex Huang]

Before cloudstack was donated to apache, it had an open source version and a 
non-open source portion (which included  vmware, f5, and juniper).  Now, 
everything is open source but that division is sometimes still in the files.  
You can basically merge those files now.

As for jdk version, that's not what's recording. The version that's supposed to 
be recorded is CloudStack's version.  Not sure why through eclipse, you're 
getting the jdk version.

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, September 19, 2013 10:11 AM
> To: dev@cloudstack.apache.org
> Subject: Re: DatabaseUpgradeChecker and running from eclipse
> 
> So I just looked at DatabaseCreator and noticed it calls
> PremiumDatabaseUpgradeChecker, but on startup we call
> DatabaseUpgradeChecker.  What is this madness?  Why are there two db
> upgrade classes?
> 
> Darren
> 
> 
> On Thu, Sep 19, 2013 at 10:04 AM, Alex Huang 
> wrote:
> 
> > Around 4.1 timeframe, Rohit added DatabaseCreator to call
> > DatabaseUpgradeChecker to create and upgrade the database but forgot
> > to remove it from the server configuration.  We should simply add a
> > DatabaseVesionChecker for the server configuration to check if the
> > database is at the version expected for the code.  That was supposed
> > to happen in
> > 4.2 but never happened.  It might be a good time to add that in now.
> >
> > --Alex
> >
> > > -Original Message-
> > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > > Sent: Thursday, September 19, 2013 9:35 AM
> > > To: dev
> > > Subject: Re: DatabaseUpgradeChecker and running from eclipse
> > >
> > > On Thu, Sep 19, 2013 at 6:12 PM, Darren Shepherd
> > >  wrote:
> > > > Only to test builds before commit.  mvn is too slow.
> > >
> > > That's probably it. I run from eclipse but usually do a build first.
> > > That is going to change one minute ago, thanks. i'll keep an eye on
> > > the dbupgradechecker
> >

RE: cloudstack source code in build dir

[Alex Huang]

Hi黑洞,

CloudStack has a few properties files with tokens within them.  These tokens 
are usually replace during installation of cloudstack.  For developers, 
replace.properties contains replacement values files for these tokens so that 
after you build, cloudstack already have the proper values for your build 
environment.  

Most of the time, these values are things such as mysql user and password for 
your local build.

--Alex

> -Original Message-
> From: 黑洞 [mailto:heids...@sina.com]
> Sent: Thursday, September 12, 2013 7:35 PM
> To: dev
> Subject: cloudstack source code in build dir
> 
> The replace.properties file
> What is the role
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 云计算基础架构师
> Github:https://github.com/heidsoftWeibo: http://weibo.com/liuganbin
> 研究方向：云计算（云安全）、大数据（数据处理分析）
> 主要学习: Linux / C / C++ / JAVA
> Email:heids...@sina.com
> Tell:18601706743
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 某云计算公司
> 
> 
> 
> 
> 
> Jake.liu@heidsoft黑洞
> 
> 
> 
> 
> 云计算系统架构师
> 
> 
> 
> 
> 
> 
> 
> 
>840608792
> 
> 
> 
> 
> 
> heidso...@gmail.com
> 
> 
> 
> 
> 
>  weibo.com/liuganbin
> 
> 
> 
> 
> 
>  上海 徐汇区
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 名片二维码
> 
> 
> 
> 
> 
> 
> 

RE: conflicting dependencies between CloudStack and Whirr

[Alex Huang]

Darren,

When I looked into updating to the latest gson, the problem, IIRC, is things 
that weren't considered cyclical dependencies are suddenly considered cyclical 
with the latest.  I don't recall where though. 

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, September 19, 2013 1:33 PM
> To: dev@cloudstack.apache.org
> Subject: Re: conflicting dependencies between CloudStack and Whirr
> 
> Alright, I looked into this and it will take a bit more work to switch to 
> Jackson.
> The snag I hit was how we serialize commands for the agents.  We use a
> customer typeadatper in gson to produce a format like { "org.MyClass"
> : {...} }.  In jackson I don't see producing a similar format as being so 
> straight
> forward.  I'm sure there's an elegant solution, but I didn't immediately find 
> it.
> The problem is if you use a custom serializer in jackson and do
> writeObjectField(x.getClass().getName(), x), you'll get stuck in a loop
> because it will call your serializer again.  So if somebody can figure out 
> how to
> do this format easily in jackson, the rest looks simple enough.
> 
> So, being that it is not an obvious switch to using jackson, what is the 
> impact
> of moving to the latest gson?  What were the issue encountered when
> people tried it before?
> 
> Darren
> 
> 
> On Wed, Sep 18, 2013 at 4:42 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
> 
> > I can do some analysis on this.  I'm always up for a terribly painful
> > refactor :)
> >
> > Darren
> >
> >
> > On Wed, Sep 18, 2013 at 4:06 PM, Alex Huang 
> wrote:
> >
> >> I actually did a quick try to update cloudstack to use newest gson
> >> version about 3 months back.  Had to roll it back but I didn't try
> >> very hard though.  Part of the reason why I decided to rollback is
> >> due to gson is used differently by various components in CloudStack
> >> and I didn't have time to get into each component to see if I break
> >> anything.  The other is also I thought using Jackson would be much
> >> better and thought our next attempt should be with Jackson.
> >>
> >> Not that any of these helps you.  Just know updating CloudStack to
> >> latest gson is not simple.
> >>
> >> --Alex
> >>
> >> > -Original Message-
> >> > From: Andrei Savu [mailto:savu.and...@gmail.com]
> >> > Sent: Wednesday, September 18, 2013 10:53 AM
> >> > To: d...@whirr.apache.org
> >> > Cc: dev@cloudstack.apache.org
> >> > Subject: Re: conflicting dependencies between CloudStack and Whirr
> >> >
> >> > It's easy to usr jclouds and whirr inside an OSGi container - just
> >> > add
> >> the
> >> > feature url. Bonus: you can also use jclouds shell interface (part
> >> > of
> >> jclouds cli).
> >> >
> >> > Another option is to upgrade the CloudStack API to use the new version.
> >> > On Sep 18, 2013 5:14 AM, "Han,Meng"  wrote:
> >> >
> >> > > Dear all,
> >> > >
> >> > > I am adding an API to CloudStack which utilizes Whirr to launch
> >> > > various clusters on CloudStack. Now I am facing a dependency
> >> conflicting
> >> > issue.
> >> > >
> >> > > Whirr 0.8.2 requires gson 2.2.2 while CloudStack API requires
> >> > > gson
> >> 1.7.1.
> >> > > If I use gson 1.7.1 for Whirr, the following error will happen:
> >> > >
> >> > > com.google.common.util.**concurrent.ExecutionError:
> >> > java.lang.**NoClassDefFoundError:
> >> > > com/google/gson/TypeAdapter
> >> > >
> >> > > This TypeAdapter class can be found inside gson-2.2.2.jar.
> >> > > However if I modify CloudStack to use gson 2.2.2 CloudStack will
> >> > > not build successfully due to the following test error.
> >> > >
> >> > > [INFO] Building Apache CloudStack Core 4.1.1 [INFO]
> >> > > --**--**
> >> > > 
> >> > > [INFO]
> >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
> >> > > cloud-core
> >> > > --- [INFO] Deleting /home/meng/cloudstack/core/**target
> >> > > [INFO]
> >> > > [INFO] --- maven-remote-res

RE: Moved the systemvm to its own maven project

[Alex Huang]

Nice!  Can't wait to look at this!

--Alex

> -Original Message-
> From: Trippie [mailto:trip...@gmail.com] On Behalf Of Hugo Trippaers
> Sent: Friday, September 20, 2013 3:40 AM
> To: dev@cloudstack.apache.org
> Subject: Moved the systemvm to its own maven project
> 
> Hey all,
> 
> Just a heads-up that i pushed this commit to master:
> 
> commit 6c261042821c5597dab8d6be85dc59c948424e13
> Author: Hugo Trippaers 
> Date:   Fri Sep 20 18:08:20 2013 +0800
> 
> Move the system vm to a separate maven project.
> 
> All (almost) files belonging to the systemvm aer now centralize in the
> systemvm directory. The code for the separate functions is still in the
> services directory. This will make the code easier to understa
> 
> 
> This commit effectively separates the code in the console-proxy from the
> actual building of the systemvm.iso. The systemvm profile (mvn -Psystemvm)
> will now control if the systemvm will be build entirely instead of just
> determining if the iso will be built. For slight speed up in the complete
> compile just leave the systemvm profile disabled.
> 
> I think this will make it more clear for everybody where the systemvm is
> located as its current location in console-proxy was a bit confusing.
> 
> Testing done:
>  * Setting up devcloud with new build and new systemvm.iso
>  * Verified ssvm operation (nfs secondary storage)
>  * Verified  cvm operation
> 
> As with all maven changes don't forget to update the maven configuration in
> your IDE. (For eclipse users, right-click on the project, Maven -> Update
> Project)
> 
> Cheers,
> 
> Hugo

Re: Review Request 13891: replaced volatile int values with AtomicInteger

[Alex Huang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13891/#review26324
---

Ship it!


Master: 3e813ce61d308c09ce23f54bd67adc980478640c

- Alex Huang


On Aug. 28, 2013, 7:40 p.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13891/
> ---
> 
> (Updated Aug. 28, 2013, 7:40 p.m.)
> 
> 
> Review request for cloudstack and Alex Huang.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> volatile int ++ and -- operations do not guarantee thread safety, these 
> values are replaced with AtomicInteger's
> 
> 
> Diffs
> -
> 
>   
> framework/jobs/src/org/apache/cloudstack/framework/jobs/impl/AsyncJobMonitor.java
>  3bf3622 
> 
> Diff: https://reviews.apache.org/r/13891/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

RE: VmWare SDK to vijava

[Alex Huang]

So this discussion took a big turn that I did not expect.  I am very strongly 
against creating a "plugin" layer just to interface with VmWare.  I appreciate 
both the effort from Darren and Hugo and each hold some merit.  I think we 
should discuss this out and drive a consensus rather than spending time to 
create "plugin" layers and such.

For me, I prefer going directly to wsdl because the helper layer (both vim25 
and vijava) is so thin that it's not a lot of effort to reproduce.  Going 
directly to wsdl has the following advantages:
  - We don't have to worry about upgrades in the helper libraries.  Look at 
what happened on 4.2 vim25 upgrade and the change in hidden behavior changes 
that caused lots of bugs.  It just doesn't seem to be worth the time to deal 
with third party helper libraries for the small amount of code gain.
  - With wsdl, we can generate with different package names so that using two 
different versions of wsdls can have two different set of stub classes so we 
avoid using java class loaders to distinguish between the two. 

Please, let's discuss on this and drive a consensus together rather than try to 
accommodate all. 

--Alex

> -Original Message-
> From: Kelven Yang [mailto:kelven.y...@citrix.com]
> Sent: Tuesday, September 24, 2013 2:06 PM
> To: dev@cloudstack.apache.org
> Subject: Re: VmWare SDK to vijava
> 
> It is about the interface layer between CloudStack and VMware, Spring can
> only help to what it can, ultimately, we need to refactor the interface layer 
> to
> take in vijava as one of its implementations.
> 
> We also need to consider of another situation, with the latest vSphere 5.1
> API, although VMware claims to support all previous vSphere versions with it,
> we actually found that this is not true. We may have to face the fact that we
> will need to support two versions of vSphere APIs at run time side-by-side.
> I'm leaning towards for us to to have some control in generating the API stub
> for this(the direct WSDL way), if it is to vijava, it depends on whether or 
> not it
> has built-in side-by-side VMware API support. and we will have more things
> to worry and test about it.
> 
> Kelven
> 
> 
> On 9/24/13 1:10 PM, "Frank Zhang"  wrote:
> 
> >Agree. Given current CloudStack code base changing something to another
> >thing is really not a good way.
> >As Darren is working on modular spring, why not construct a new VMWare
> >plugin separately using vijava?
> >Then we can reduce the risk of surprising existing customer and switch
> >to new module when it finally gets mature.
> >
> >
> >> -Original Message-
> >> From: Kelven Yang [mailto:kelven.y...@citrix.com]
> >> Sent: Tuesday, September 24, 2013 11:59 AM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: VmWare SDK to vijava
> >>
> >> We have commercial releases on top of existing code base and there
> >>are lots of  testing efforts behind it, dramatic switch means $ cost,
> >>the major concern for  me is not about how beautiful vijava is or how
> >>bad a direct wsdl approach  would be. it is about to get things move
> >>smoothly.
> >>
> >> It looks like that we should have VMware layer on its own to have a
> >>plugin  structure so that we can replace underlying binding easier, it
> >>should solve the  balance between developer's motivation and carrying
> >>on the legacy with  minimal impacts to the rest of others.
> >>
> >>
> >> Kelven
> >>
> >> On 9/23/13 6:01 PM, "Hugo Trippaers"  wrote:
> >>
> >> >Heya,
> >> >
> >> >This biggest advantage i see in using vijava is that a lot of the
> >> >functionality that we now have in the vmware-base project is already
> >> >supplied with vijava.
> >> >
> >> >There is a lot of code that facilitates calling tasks and other
> >> >stuff in our MO classes. These classes are available in vijava and
> >> >could be used instead of our classes. Basically when using vijava
> >> >correctly you hardly have to work with the ManagedObjectReferences
> >> >anymore. For me this would be a big benefit as it makes programming
> >> >against vmware a lot easier. We also have a lot of duplicate code
> >> >currently in the vmware class and i wouldn't mind getting rid of it,
> >> >which i think is easier with the vijava libraries.
> >> >
> >> >That said, the main driver is getting it into the main build so any
> >> >other efforts towards that goal are ok with me.
> >> >
> >> >I would propose that somebody else puts up a branch with our own
> >> >wdsl wrapper and we open a discussion thread when both branches are
> >> >ready to see which we want to merge in master. Anybody who wants to
> >> >pick that
> >>up?
> >> >
> >> >I'm stubbornly going to continue with converting to vijava, I put
> >> >some effort into it and i want at least to see it running once ;-)
> >> >And the more i work with it the more i'm seeing to benefits of the
> >> >library so i might be able to be more convincing in the end :-)
> >> >
> >> >Cheers,
> >> >
> >> >Hugo
> >> >
> >> >
> >> >On Sep 24, 2

RE: VmWare SDK to vijava

[Alex Huang]

> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: Tuesday, September 24, 2013 4:23 PM
> To: dev@cloudstack.apache.org
> Subject: RE: VmWare SDK to vijava
> 
> So this discussion took a big turn that I did not expect.  I am very strongly
> against creating a "plugin" layer just to interface with VmWare.  I appreciate
> both the effort from Darren and Hugo and each hold some merit.  I think we
> should discuss this out and drive a consensus rather than spending time to
> create "plugin" layers and such.
> 
> For me, I prefer going directly to wsdl because the helper layer (both vim25
> and vijava) is so thin that it's not a lot of effort to reproduce.  Going 
> directly
> to wsdl has the following advantages:
>   - We don't have to worry about upgrades in the helper libraries.  Look at
> what happened on 4.2 vim25 upgrade and the change in hidden behavior
> changes that caused lots of bugs.  It just doesn't seem to be worth the time
> to deal with third party helper libraries for the small amount of code gain.
>   - With wsdl, we can generate with different package names so that using
> two different versions of wsdls can have two different set of stub classes so
> we avoid using java class loaders to distinguish between the two.

I want to elaborate specifically on this point a little bit.  In a production 
deployed cloud that has been around, there's bound to be hypervisors of 
different versions.  We have that problem with XenServer because it is version 
that's been supported by CloudStack the longest.  CloudStack deals with the 
different versions of the same Hypervisor by writing two different 
ServerResources (they might share a common base of course).  However, if the 
client stubs the different ServerResources talk to are also of different 
versions but with the same name, then we have to work on confining different 
versions of the client stubs to different class loaders.  That's a lot of 
effort to do something simple.  Instead, we can simply designate a different 
package name when  generating from the wsdl (assuming the different versions of 
client stubs is due to wsdl versioning differences when interfacing with 
different versions of hypervisor) and have the ServerResources just refer to 
the different client stubs generated.  Hence, I prefer generation directly from 
wsdl.

--Alex

RE: VmWare SDK to vijava

[Alex Huang]

Agreed.  If we can do it is the big question.  Although, AFAIK, vijava is also 
generated from the same set of wsdl.  So if we can't get the license, I have to 
wonder how did vijava get their BSD license?

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Tuesday, September 24, 2013 6:07 PM
> To: dev@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: VmWare SDK to vijava
> 
> My extensive testing consisted of "it compiles!"  So somebody needs to
> validate it, I don't have a vmware setup handy.  The wsdl generation route is
> not feasible unless legal says it's okay.  Somebody want to email legal@?
> 
> Darren
> 
> > On Sep 24, 2013, at 5:27 PM, Hugo Trippaers  wrote:
> >
> > So at this moment we have the following tally,
> >
> > Darren, Alex, Kelven opt for the wsdl route Hugo opts for the vijava
> > route
> >
> > I'm perfectly ok with ditching my work on the vijava in favour of the wsdl
> route. The arguments presented in the last few mails make a lot of sense. So
> i say the wsdl route is the way to go.
> >
> > I do think however that we need to revisit the vmware code anyway. There
> is dead code in there, formatting issues and a general duplication of code.
> Parts of my plan to switch to vijava was to simplify the code as well, but i 
> can
> still do that with the WSDL layer.
> >
> > Darren, did you run your wsdl branch through the BVT test suite? If you did
> we can merge it on short notice and get on with it. If you didn't can you take
> care of that and give an overview of the testing done on that branch besides
> the BVT?
> >
> > Cheers,
> >
> > Hugo
> >
> >
> >> On Sep 25, 2013, at 2:58 AM, Kelven Yang 
> wrote:
> >>
> >> We have commercial releases on top of existing code base and there
> >> are lots of testing efforts behind it, dramatic switch means $ cost,
> >> the major concern for me is not about how beautiful vijava is or how
> >> bad a direct wsdl approach would be. it is about to get things move
> smoothly.
> >>
> >> It looks like that we should have VMware layer on its own to have a
> >> plugin structure so that we can replace underlying binding easier, it
> >> should solve the balance between developer's tho motivation and
> >> carrying on the legacy with minimal impacts to the rest of others.
> >>
> >>
> >> Kelven
> >>
> >>> On 9/23/13 6:01 PM, "Hugo Trippaers"  wrote:
> >>>
> >>> Heya,
> >>>
> >>> This biggest advantage i see in using vijava is that a lot of the
> >>> functionality that we now have in the vmware-base project is already
> >>> supplied with vijava.
> >>>
> >>> There is a lot of code that facilitates calling tasks and other
> >>> stuff in our MO classes. These classes are available in vijava and
> >>> could be used instead of our classes. Basically when using vijava
> >>> correctly you hardly have to work with the ManagedObjectReferences
> >>> anymore. For me this would be a big benefit as it makes programming
> >>> against vmware a lot easier. We also have a lot of duplicate code
> >>> currently in the vmware class and i wouldn't mind getting rid of it,
> >>> which i think is easier with the vijava libraries.
> >>>
> >>> That said, the main driver is getting it into the main build so any
> >>> other efforts towards that goal are ok with me.
> >>>
> >>> I would propose that somebody else puts up a branch with our own
> >>> wdsl wrapper and we open a discussion thread when both branches are
> >>> ready to see which we want to merge in master. Anybody who wants to
> pick that up?
> >>>
> >>> I'm stubbornly going to continue with converting to vijava, I put
> >>> some effort into it and i want at least to see it running once ;-)
> >>> And the more i work with it the more i'm seeing to benefits of the
> >>> library so i might be able to be more convincing in the end :-)
> >>>
> >>> Cheers,
> >>>
> >>> Hugo
> >>>
> >>>
>  On Sep 24, 2013, at 2:18 AM, Kelven Yang 
> wrote:
> 
>  Prior to 5.1, VMware provides java binding in its SDK and this is
>  where CloudStack VMware integration began with. Starting from 5.1,
>  VMware no longer provides the java binding in binary form and
>  recommends its customers to generate directly from its WS WSDL.
> 
>  Since we are not sure if we can distribute VMware wsdl legally or
>  not, therefore, we ended up to generate and distribute the java
>  binding in binary form. If we can get this cleared in lebal, as
>  Darren pointed, we can solve our non-dist issue easily in matter of
>  adding couple of lines in maven.
> 
>  As of vijava, yes, I think it may add some value from developer's
>  point of view, but on the other hand, I don't see immediate
>  benefits to having another layer on top of VMware official WS API,
>  vijava is an open source project for providing convenient java
>  binding to vmware WS API, maybe I'm wrong, but I think VMware
>  vSphere WS API is the only official published

RE: VmWare SDK to vijava

[Alex Huang]

Wow good guess...Hugo had me scratching my head on that oneIs Prussian some 
code name for Apache legalShould I askWould it make me look stupid if I 
askedall sorts of doubts going through my mind.

--Alex

> -Original Message-
> From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com]
> Sent: Tuesday, September 24, 2013 7:01 PM
> To: dev@cloudstack.apache.org
> Subject: Re: VmWare SDK to vijava
> 
> Ha! Prussians == Prasanna?
> Godawful autocorrect.
> 
> On 9/24/13 6:47 PM, "Hugo Trippaers"  wrote:
> 
> >Darren, you can probably work with Prussians to get it through the bvt
> >suite. That would be a nice start.
> >
> >Cheers,
> >
> >Hugo
> >
> >Sent from my iPhone
> >
> >> On 25 sep. 2013, at 09:06, Darren Shepherd
> >> wrote:
> >>
> >> My extensive testing consisted of "it compiles!"  So somebody needs
> >>to validate it, I don't have a vmware setup handy.  The wsdl
> >>generation route is not feasible unless legal says it's okay.
> >>Somebody want to email legal@?
> >>
> >> Darren
> >>
> >>> On Sep 24, 2013, at 5:27 PM, Hugo Trippaers  wrote:
> >>>
> >>> So at this moment we have the following tally,
> >>>
> >>> Darren, Alex, Kelven opt for the wsdl route Hugo opts for the vijava
> >>> route
> >>>
> >>> I'm perfectly ok with ditching my work on the vijava in favour of
> >>>the wsdl route. The arguments presented in the last few mails make a
> >>>lot of sense. So i say the wsdl route is the way to go.
> >>>
> >>> I do think however that we need to revisit the vmware code anyway.
> >>>There is dead code in there, formatting issues and a general
> >>>duplication of code. Parts of my plan to switch to vijava was to
> >>>simplify the code as well, but i can still do that with the WSDL layer.
> >>>
> >>> Darren, did you run your wsdl branch through the BVT test suite? If
> >>>you did we can merge it on short notice and get on with it. If you
> >>>didn't can you take care of that and give an overview of the testing
> >>>done on that branch besides the BVT?
> >>>
> >>> Cheers,
> >>>
> >>> Hugo
> >>>
> >>>
>  On Sep 25, 2013, at 2:58 AM, Kelven Yang 
> wrote:
> 
>  We have commercial releases on top of existing code base and there
> are  lots of testing efforts behind it, dramatic switch means $
> cost, the major  concern for me is not about how beautiful vijava is
> or how bad a direct  wsdl approach would be. it is about to get
> things move smoothly.
> 
>  It looks like that we should have VMware layer on its own to have a
> plugin  structure so that we can replace underlying binding easier,
> it should  solve the balance between developer's tho motivation and
> carrying on the  legacy with minimal impacts to the rest of others.
> 
> 
>  Kelven
> 
> > On 9/23/13 6:01 PM, "Hugo Trippaers"  wrote:
> >
> > Heya,
> >
> > This biggest advantage i see in using vijava is that a lot of the
> > functionality that we now have in the vmware-base project is
> > already supplied with vijava.
> >
> > There is a lot of code that facilitates calling tasks and other
> >stuff in  our MO classes. These classes are available in vijava and
> >could be used  instead of our classes. Basically when using vijava
> >correctly you hardly  have to work with the
> ManagedObjectReferences
> >anymore. For me this would  be a big benefit as it makes
> >programming against vmware a lot easier. We  also have a lot of
> >duplicate code currently in the vmware class and i  wouldn't mind
> >getting rid of it, which i think is easier with the vijava
> >libraries.
> >
> > That said, the main driver is getting it into the main build so
> >any other  efforts towards that goal are ok with me.
> >
> > I would propose that somebody else puts up a branch with our own
> >wdsl  wrapper and we open a discussion thread when both branches
> >are ready to  see which we want to merge in master. Anybody who
> >wants to pick that up?
> >
> > I'm stubbornly going to continue with converting to vijava, I put
> >some  effort into it and i want at least to see it running once ;-)
> >And the  more i work with it the more i'm seeing to benefits of the
> >library so i  might be able to be more convincing in the end :-)
> >
> > Cheers,
> >
> > Hugo
> >
> >
> >> On Sep 24, 2013, at 2:18 AM, Kelven Yang 
> >>wrote:
> >>
> >> Prior to 5.1, VMware provides java binding in its SDK and this is
> >>where  CloudStack VMware integration began with. Starting from
> >>5.1, VMware no  longer provides the java binding in binary form
> >>and recommends its  customers to generate directly from its WS
> >>WSDL.
> >>
> >> Since we are not sure if we can distribute VMware wsdl legally or
> >>not,  therefore, we ended up to generate and distribute the java
> >>binding in  binary form. If we can get this cle

RE: [VOTE] Accept the donation of a Contrail plugin into Apache CloudStack

[Alex Huang]

+1 (binding)

--Alex

> -Original Message-
> From: Chip Childers [mailto:chipchild...@apache.org]
> Sent: Wednesday, September 25, 2013 10:13 AM
> To: dev@cloudstack.apache.org
> Subject: [VOTE] Accept the donation of a Contrail plugin into Apache
> CloudStack
> 
> Hi all!
> 
> As stated in other threads, Juniper is proposing the donation of a Contrail
> plugin to Apache CloudStack.  The code itself has been posted to
> reviewboard [1].  The design has been documented by Pedro [2].
> 
> [1] https://reviews.apache.org/r/14325/
> [2]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Contrail+networ
> k+plugin
> 
> I'm calling a vote here, so that we have a formal consensus on accepting the
> code into the project.  As I've suggested earlier, I'd like us to accept the 
> code
> into a branch, and then work through any technical concerns / reviews /
> changes prior to a master branch merge.
> 
> So...  voting will end in ~72 hours.  As this is a technical decision, 
> committer
> and PMC votes are binding.
> 
> -chip
> 
> 
> Votes please!
> 
> [ ] +1 - Accept the donation
> [ ] +/-0 - No strong opinion
> [ ] -1 - Do not accept the donation

RE: [DISCUSS] Release Managers for future ACS releases - enhacement

[Alex Huang]

Agreed.  If you look at what a release manager has to do today

- triage bugs
- follow up on reviews and ask people to commit them
- cherry-pick fixes

To me it is a lot of work for one person to do for CloudStack.  We can 
certainly divide up the work.  For example,

 - One RM is responsible for overall release 
 - One RM is responsible for following up on review board
 - Two or three RMs is responsible for triaging bugs
 - One is responsible for cherry-pick

I also like to propose that we stop the practice of only assigning bugs to 
yourself.  I know it's there to make sure there's no cookie-licking but really 
let's not make ourselves less efficient just for the sake of appearances.   RMs 
should be able to assign bugs as part of the process to ask for someone to look 
at the bug rather than having to ask privately and have the person assign to 
themselves.  Keeping track of such things with the amount of changes CloudStack 
goes through in a release just makes us less efficient and less enjoyable to 
work on CloudStack.
 
--Alex

> -Original Message-
> From: Musayev, Ilya [mailto:imusa...@webmd.net]
> Sent: Thursday, September 26, 2013 9:02 AM
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS] Release Managers for future ACS releases - enhacement
> 
> I apologize in advance if this is a repeat of something that was previously
> stated.
> 
> As Animesh learned recently with ACS 4.2, RM work for major versions takes
> a lot of effort, to lesser extent the 4.2.x minor release may not be as 
> involved,
> but still decent amount of work.
> 
> What complicates the matter further, is many of us have $dayjobs$ that
> don't emphasize heavy involvement on ACS.
> 
> Perhaps we can revisit the strategy and have 2 -3 release managers for major
> version and 1-2 for minor.
> 
> Obviously, one is going the be a Lead RM, and others will be secondary but
> also involved.
> 
> Any thoughts on this approach?
> 
> Thanks
> ilya

RE: [PROPOSAL] Service monitoring tool in virtual router

[Alex Huang]

Using SNMP for alert notification is not a bad idea though.  I don't see why we 
can't do that instead of posting to the management server.  This is 
specifically referring to the second part of the proposal.  Why reinvent that 
part of it?

--Alex

> -Original Message-
> From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com]
> Sent: Wednesday, September 25, 2013 10:28 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Service monitoring tool in virtual router
> 
> SNMP wouldn't restart a failed process nor would it generate alerts. It is
> simply too generic for the requirements outlined here. The proposal does
> not talk about modifying monit, just using it. That wouldn't trigger the AGPL.
> I think the idea is to have a tight monitoring loop that scales: so executing 
> the
> monitoring loop in-situ makes sense.
> 
> 
> On 9/25/13 9:53 PM, "David Nalley"  wrote:
> 
> >On Wed, Sep 25, 2013 at 9:30 AM, Jayapal Reddy Uradi
> > wrote:
> >> Hi,
> >>
> >> Currently in virtual router there is no way to recover and notify if
> >>some service goes down unexpectedly.
> >>
> >> This feature is about monitoring all the services rendered by the
> >>virtual router, ensure that the services are running through the life
> >>time of the VR.
> >>
> >> On service failure:
> >> 1. Generate an alert and event indicating failure 2. Restart the
> >> service
> >>
> >> Services to be monitored:
> >> DHCP, DNS, haproxy, password server etc.
> >>
> >> As part of monitoring there are two activities
> >>
> >> 1. One is monitoring the services in VR and log the events. Using
> >>monit for monitoring services  2. Second part is pushing alerts from
> >>router to  MS server. Thinking on POST the logs to web server in MS.
> >>
> >> I will be updating more details and FS in this thread.
> >>
> >> I created enhancement bug for this.
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-4736
> >>
> >> Thanks,
> >> Jayapal
> >
> >So several things - why not make this via SNMP? Query processes, and
> >many other things. This should be relatively simple, is well known, can
> >be locked down (or could be monitored for many other things by external
> >monitoring packages) and is the defacto standard for monitoring hosts.
> >Second - monit is Affero GPL licensed - which is a cat-x license.
> >While I expect that we would merely use this and not do any hacking on
> >it - I think its inclusion might be a surprise (and forbidden in many
> >environments) to our users
> >
> >--David

RE: NetworkElement and other *ServiceProviders often have useless return values

[Alex Huang]

Shouldn't be a problem.  They were Boolean previously to mean this 
networkelement does not know how to apply the rule, move on to the next one 
because previously it was really more like a broadcast to all networkelements.  
Whichever networkelment believes they should apply the rule, then applies.  
Now, it is more prescriptive as the service providers are actually specified by 
the admin when they create the zone so the Boolean is no longer needed.  I 
doubt it's checked anywhere.

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Friday, September 27, 2013 11:07 AM
> To: dev@cloudstack.apache.org; Chiradeep Vittal; Alex Huang
> Subject: NetworkElement and other *ServiceProviders often have useless
> return values
> 
> On NetworkElement and many other *ServiceProvider interfaces (ie
> DhcpServiceProvider), the return value for the methods are a boolean, but
> those booleans are never checked.  I would like to clean up all these
> interfaces and put the majority of the methods to return "void."
>  Would there be any objection to this?  I find it quite confusing when
> implementing an interface when I have to look at calling code to determine if
> I should return false or throw an exception in some case.
> 
> Cleaning this up would mean I'd have to update all the implementations of
> the interfaces.
> 
> Darren

RE: [PROPOSAL] Support OVA files with multiple disks for templates and uploaded volumes in VMWare

[Alex Huang]

Likitha,

This is not of value to VmWare only.  We should make sure it works or has 
plugin points for all of the hypervisors.  In general, how we support this 
today is through hypervisor capabilities to determine during orchestration and 
then on the Resource for each hypervisor, it can determine if it can support 
commands introduced for a functionality.

--Alex

> -Original Message-
> From: Likitha Shetty [mailto:likitha.she...@citrix.com]
> Sent: Monday, September 30, 2013 10:30 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] Support OVA files with multiple disks for templates
> and uploaded volumes in VMWare
> 
> True, OVA can contain any disk image type. I specifically called out VMware
> because I was thinking about how today in CS this enhancement is of value
> only to VMware.
> 
> But yes, as you pointed out, during implementation we should ensure this
> support is hypervisor agnostic.
> 
> Thanks,
> Likitha
> 
> >-Original Message-
> >From: Chip Childers [mailto:chipchild...@apache.org]
> >Sent: Monday, September 30, 2013 10:06 PM
> >To: dev@cloudstack.apache.org
> >Subject: Re: [PROPOSAL] Support OVA files with multiple disks for
> >templates and uploaded volumes in VMWare
> >
> >On Mon, Sep 30, 2013 at 03:26:54PM +, Likitha Shetty wrote:
> >> Chip,
> >>
> >> In Cloudstack, for Xen and KVM hypervisors since the files used for
> >> templates
> >and volumes are virtual disks (VHD, QCOW2) the current CS assumption of
> >template and volumes containing a single virtual disk seems fine. But
> >since for VMware the files used for templates and volumes are in OVA
> >format which is an archive that can contain multiple VMDKs this
> >improvement seems appropriate only for VMware?
> >>
> >> Thanks,
> >> Likitha
> >
> >Ok, I see why you are headed down a VMware only path now.  That being
> >said, OVF/OVA are actually envelope specs which can contain any disk
> image type.
> >The key is to either use the native HV OVF/A import capabilities
> >(VMware obviously does this, as does XenServer IIRC), or use a tool
> >like virt-tools to make the embedded meta-data about the included VM
> useful for the target HV.
> >
> >While I get doing this for VMware only, I'd ask that the implementation
> >be designed to potentially work with the other HVs.
> >
> >Make sense?

RE: [DISCUSS] Breaking out Marvin from CloudStack

[Alex Huang]

I don't really understand what purpose would this serve.  Would we ever use 
newer marvin against older CloudStack or vice versa?  What's the benefit?  

I can understand it for cloudmonkey because cloudmonkey is an admin cli tool 
and reving it differently is not a bad idea.  I just don't see it for marvin 
and, especially for the tests.

--Alex

> -Original Message-
> From: Prasanna Santhanam [mailto:t...@apache.org]
> Sent: Wednesday, October 2, 2013 10:14 AM
> To: CloudStack Dev
> Subject: [DISCUSS] Breaking out Marvin from CloudStack
> 
> I would like to seperate marvin from the main cloudstack repo. Much of
> marvin's development has little coupling with CloudStack.
> 
> Similar to CloudMonkey, marvin undergoes rapid changes and it is essential
> to provide a smooth workflow and faster releases for those working with it.
> 
> There are also a small set of people currently looking at marvin for testing
> right now. Often, their reviews and QA effort is mixed with those of
> cloudstack itself. By having a different repo I'd like to be able to provide
> commit access to those working on marvin alone quickly to help with testing.
> 
> After separating marvin
> 0. we will have a separate release cycle for marvin 1. we will have a new
> home for marvin's docs using Sphinx 2. if possible, a different criteria for
> providing commit access to marvin's repos.
> 3. all tests of cloudstack will also move to marvin's repository
> 
> Thoughts?
> 
> --
> Prasanna.,
> 
> 
> Powered by BigRock.com

RE: [DISCUSS] Breaking out Marvin from CloudStack

[Alex Huang]

Yeah...I'm more amendable to this proposal.  I just don't see tests being 
separated from the source release.  In fact, I see a lot of problems with 
matching versions and releases.

I still question the value of separating out a framework where both ends (tests 
and the server it tests) stay in the same repo but I guess there should be no 
harm.  I do think our time can be better spent elsewhere (for example, writing 
the tests) but, if others see it as necessary, I wouldn't be against it.

@Edison
I can do what you said now.  To me that's more or less maven changes and build 
changes.  Not a repo/separate release question.

--Alex

> IMO, we should consider Marvin the "framework" to be the thing to break
> out, and the tests should be different from the framework.
> 
> Now that leads to the question: to test or not to test (in the main repo)?
> 
> I'd suggest that *tests* belong in the main repo, because they are tied to the
> software's capabilities and versions.
> 
> The Marvin framework, on the other hand, since the re-work that Prasanna
> did, is mostly distinct (and uses API discovery).
> 
> Anyone else agree?

RE: [MERGE] spring-modularization to master - Spring Modularization

[Alex Huang]

+1 on running the BVT on it.  We've been through this one once before.  Should 
be careful.

--Alex

> -Original Message-
> From: Kelven Yang [mailto:kelven.y...@citrix.com]
> Sent: Wednesday, October 2, 2013 4:39 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [MERGE] spring-modularization to master - Spring
> Modularization
> 
> Darren,
> 
> This looks really nice. A few questions on Spring AOP replacement.
> 
> 1) Spring AOP is proxy-based, the reason we ended up of using customized
> AOP is mainly due to that inside existing CloudStack codebase, we have many
> places that are doing run-time type-casting, the code in these places
> assumes a real object that implements all interfaces in the semantics context.
> At the time when I initially converted to Spring, I couldn't ensure that
> switching to proxy-based AOP can have 100% coverage for these run-time
> cases. What is your approach to address this run-time type-casting issue?
> 
> 2) We've run into a huge-memory footprint issue that may be caused by
> conflicts of CGLIB usage in Spring AOP and the CGLIB usage in CloudStack Dao
> layer. Do you have a chance to run a memory analysis in the heap after
> management server is started.
> 
> I might be good to run BVT test on the branch before the merge, could
> someone initiate the effort?
> 
> kelven
> 
> 
> 
> On 10/2/13 3:48 PM, "Darren Shepherd" 
> wrote:
> 
> >Not sure how this works...  I would like to merge in the new
> >modularized Spring setup to master. There is info on the wiki about it
> >[1] [2] [3].  The primary change is to break apart the monolithic
> >applicationContext.xml and componentContext.xml files such that each
> >plugin can maintain and contribute its own configuration.
> >
> >In addition to breaking up the configuration we no longer use ACS
> >custom AOP and it is now fully Spring AOP.
> >
> >Now adding/removing a plug-in is a matter of just adding a jar to the
> >classpath (exception being commands.properties, I'll address that in a
> >different thread).  Unfortunately this branch does not have the changes
> >to package things in different RPMs.  So it would be great if somebody
> >could take up the packaging effort to split out all the plugins into
> >different RPMs.
> >
> >Darren
> >
> >[1]
> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Sp
> rin
> >g
> >[2]
> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> ins%2C+Modu
> >les
> >%2C+and+Extensions
> >[3] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions

RE: [Proposal] Improve VR upgrades

[Alex Huang]

+1.  That's the way to do it with no downtime.  Tighter control needs to be 
formed on the VR to VR version compatibility though.  It's one reason I always 
support that we break VR into smaller pieces.  Redundant VRs work great as 
routing but then if you have to handle DHCP, DNS, Load Balancing, and others 
with it then it becomes orders of magnitude harder to get right.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Thursday, October 3, 2013 1:38 PM
> To: dev
> Subject: Re: [Proposal] Improve VR upgrades
> 
> At Schuberg Philis we are looking at supporting HA redundant virtual routers
> to be able to upgrade without any downtime.  I don't think there is any other
> way to go in the future, for multi-tenant corporate environments.
> Upgrading will then mean destroying the redundant pair one by one while
> waiting with destroying the second one till the first is back up.
> 
> Of course this is not usefull for other systemvms then the router vms. Do you
> see a place for this in your design, Kishan?
> 
> As for Chip's concern; another implementation would be to allow for a
> versioned vm/ms interface, instead of a backwards compatible one. Any way
> it is a matter to deal with somehow.
> 
> Daan
> 
> 
> 
> On Thu, Oct 3, 2013 at 5:21 PM, Chip Childers
> wrote:
> 
> > On Thu, Oct 03, 2013 at 11:47:57AM +, Kishan Kavala wrote:
> > > During CS upgrade, VRs are required to be upgraded to use newer
> > > systemVm
> > template .
> > > The current VR upgrade procedure has following limitations:
> > >  - takes 'long' time and the time exponentially increases with the
> > > size
> > of the cloud
> > > - no way to sequence upgrade of different parts of the cloud, i.e.,
> > specific clusters or pods or even zones
> > > - there is no way to determine when a particular customer's services
> > (e.g. VR) will be upgraded with the upgrade interval
> > >
> > > Goals for this feature are to address the above issues
> > >
> > > 1. Give admin control to sequence the upgrade of the cloud by:
> > >- Infrastructure hierarchy: by Cluster, Pod, and Zone etc.
> > >- Administrative hierarchy: by Tenant or Domain 2. Minimize
> > > service interruption to users 3. Improve the speed of the upgrade
> > > time by making as many upgrade
> > operations in parallel as possible
> > >
> > > I've created JIRA ticket:
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-4793
> > >
> > > thanks,
> > > Kishan
> > >
> >
> > This proposal sounds great, but the devil will be in the
> > implementation details.  To do this as rolling, we'd need to ensure
> > backward compat with agent>MS communications, right?
> >
> > -chip
> >

RE: Command sequence logic in agent code

[Alex Huang]

I only took a quick look here.  I think when commands originate from the 
management server, it doesn't go through this code.  This is if the command 
came from the agent which doesn't matter if there's multiple management servers.

--Alex

> -Original Message-
> From: Koushik Das
> Sent: Thursday, October 24, 2013 2:11 AM
> To: 
> Cc: Alex Huang
> Subject: Re: Command sequence logic in agent code
> 
> Created https://issues.apache.org/jira/browse/CLOUDSTACK-4944 to track
> this issue.
> 
> Alex, Any reason for adding requests based on sequence and not doing FIFO?
> Do you see any issues if request always gets added to the end of the queue?
> 
> 
> On 23-Oct-2013, at 6:26 PM, Koushik Das  wrote:
> 
> > I was looking at the command sequencing logic in the agent code.
> >
> > Each agent maintains a sequence that gets initialised based on following
> logic
> >
> >private static final Random s_rand = new
> Random(System.currentTimeMillis());
> >_nextSequence = s_rand.nextInt(Short.MAX_VALUE) << 48;
> >
> > For every command that gets processed by the agent the sequence is
> incremented by 1. If commands are to be executed in sequence then they
> are queued up based on this sequence
> >
> >protected synchronized void addRequest(Request req) {
> >int index = findRequest(req);
> >assert (index < 0) : "How can we get index again? " + index + ":" +
> req.toString();
> >_requests.add(-index - 1, req);
> >}
> >
> > The above works fine in case of a single MS scenario. In case of a clustered
> MS setup things change slightly.
> >
> > The command can originate at any MS and based on the ownership of the
> agent, it gets forwarded to the correct MS which then handles the command.
> Now command sequences are local to individual agents in MS. In this case
> the originating MS agent tags the request with a sequence. This gets
> forwarded to the owning MS and based on if 'executedInSequence' flag is
> set, gets added to the list based on the sequence number. Now here lies the
> problem, commands are not inserted in the order in which they arrive but
> based on the sequence number. In case of a forwarded command the
> sequence is different from the local sequence. If the starting sequence of
> forwarded commands is much less than that of the locally generated
> commands then there is a possibility of local commands getting starved if
> there is a steady arrival of forwarded commands. Similarly it can also happen
> the other way round. Also if the the starting sequence for a agent in local 
> and
> peer MS is not spread far apart then there may be overlaps and a new
> request will override the old one.
> >
> > Not sure if anyone encountered any issues due to this. The correct way
> looks like to implement the queue model rather than doing a add based on
> the above code.
> >
> > Comments?
> >
> > -Koushik
> >
> >
> >
> >
> >

RE: Metadata URL location

[Alex Huang]

In order to use an link local address inside the end user vm, that metadata 
service must be setup on every hypervisor's dom0 or it has to be proxied out of 
the dom0.  That's not doable for VmWare.  Instead, CloudStack uses VR to serve 
the data, which works for all three hypervisors.  

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 8:13 AM
> To: dev@cloudstack.apache.org
> Cc: klindg...@godaddy.com
> Subject: Re: Metadata URL location
> 
> My guess, I don't really know, would be because its hard.  The VR uses link
> local for the control network so 169.254/16 is bound to the wrong nic.  To fix
> this you just need some ip routing magic in linux (credit goes to Kris 
> Lindgren
> who showed me how to do this).  Add the below to a file, substitute eth0 for
> the guest network nic, run "ip -b "
> The below effectively creates a routing table dedicated to the IP
> 169.254.169.254 that sets it default route to go out the guest network nic.
> 
> rule add from 169.254.169.254 table 70
> rule add to 169.254.169.254 dev eth0 table 70 route flush table 70 route add
> default dev eth0 src 169.254.169.254 table 70 route flush cache
> 
> Darren
> 
> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>  wrote:
> > Hi Guys,
> >
> > CloudStack metadata services are on the default gateway while on EC2,
> > its at 169.254.169.254. Am curious to know why CloudStack does not use
> > a link local address for meta data services.
> >
> > A search of the Wiki
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDST
> ACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&
> spaceSearch=true&queryString=metadata) didn't seem to list any doc
> related to the design of this service.
> >
> > TIA.
> >
> > --
> > @shankerbalan
> >
> > M: +91 98860 60539 | O: +91 (80) 67935867 shanker.ba...@shapeblue.com
> > | www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
> >
> > CloudStack Bootcamp Training on 27/28 November, Bangalore
> > http://www.shapeblue.com/cloudstack-training/
> >
> >
> >
> >
> > This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender if
> you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape
> Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
> registered trademark.

RE: Metadata URL location

[Alex Huang]

Darren,

I was under the impression that linklocal doesn't go out of the VM but 
Chiradeep just explained to me that it does as long as the default gateway is 
correct and there's no other routes directing it elsewhere.  Then the changes 
make sense.  

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 2:52 PM
> To: dev@cloudstack.apache.org
> Cc: klindg...@godaddy.com
> Subject: Re: Metadata URL location
> 
> Alex,
> 
> I don't think that's correct.  The instructions that I gave in the earlier 
> email
> was changes to the VR to serve metadata from the VR.
> Regardless of hypervisor, it should work.
> 
> Darren
> 
> On Thu, Oct 24, 2013 at 10:54 AM, Alex Huang 
> wrote:
> > In order to use an link local address inside the end user vm, that metadata
> service must be setup on every hypervisor's dom0 or it has to be proxied out
> of the dom0.  That's not doable for VmWare.  Instead, CloudStack uses VR to
> serve the data, which works for all three hypervisors.
> >
> > --Alex
> >
> >> -Original Message-
> >> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> >> Sent: Thursday, October 24, 2013 8:13 AM
> >> To: dev@cloudstack.apache.org
> >> Cc: klindg...@godaddy.com
> >> Subject: Re: Metadata URL location
> >>
> >> My guess, I don't really know, would be because its hard.  The VR
> >> uses link local for the control network so 169.254/16 is bound to the
> >> wrong nic.  To fix this you just need some ip routing magic in linux
> >> (credit goes to Kris Lindgren who showed me how to do this).  Add the
> >> below to a file, substitute eth0 for the guest network nic, run "ip -b
> "
> >> The below effectively creates a routing table dedicated to the IP
> >> 169.254.169.254 that sets it default route to go out the guest network nic.
> >>
> >> rule add from 169.254.169.254 table 70 rule add to 169.254.169.254
> >> dev eth0 table 70 route flush table 70 route add default dev eth0 src
> >> 169.254.169.254 table 70 route flush cache
> >>
> >> Darren
> >>
> >> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
> >>  wrote:
> >> > Hi Guys,
> >> >
> >> > CloudStack metadata services are on the default gateway while on
> >> > EC2, its at 169.254.169.254. Am curious to know why CloudStack does
> >> > not use a link local address for meta data services.
> >> >
> >> > A search of the Wiki
> >>
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDS
> >> T
> ACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&
> >> spaceSearch=true&queryString=metadata) didn't seem to list any doc
> >> related to the design of this service.
> >> >
> >> > TIA.
> >> >
> >> > --
> >> > @shankerbalan
> >> >
> >> > M: +91 98860 60539 | O: +91 (80) 67935867
> >> > shanker.ba...@shapeblue.com
> >> > | www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India
> >> > | LLP,
> >> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
> >> >
> >> > CloudStack Bootcamp Training on 27/28 November, Bangalore
> >> > http://www.shapeblue.com/cloudstack-training/
> >> >
> >> >
> >> >
> >> >
> >> > This email and any attachments to it may be confidential and are
> >> > intended
> >> solely for the use of the individual to whom it is addressed. Any
> >> views or opinions expressed are solely those of the author and do not
> >> necessarily represent those of Shape Blue Ltd or related companies.
> >> If you are not the intended recipient of this email, you must neither
> >> take any action based upon its contents, nor copy or show it to
> >> anyone. Please contact the sender if you believe you have received
> >> this email in error. Shape Blue Ltd is a company incorporated in
> >> England & Wales. ShapeBlue Services India LLP is a company
> >> incorporated in India and is operated under license from Shape Blue
> >> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> >> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
> registered trademark.

failing unit tests....

[Alex Huang]

I'm getting this failing unit test when building with the latest from master.  
Anyone working on it or know what it is already?  From the stack, it looks like 
it's a problem location the jdbc driver.  This was working just yesterday.

Test set: com.cloud.alert.AlertControlsUnitTest
---
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.175 sec <<< 
FAILURE!
warning(junit.framework.TestSuite$1)  Time elapsed: 0.004 sec  <<< FAILURE!
junit.framework.AssertionFailedError: Exception in constructor: testInjected 
(java.lang.ExceptionInInitializerError
at 
com.cloud.alert.AlertControlsUnitTest.(AlertControlsUnitTest.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at junit.framework.TestSuite.createTest(TestSuite.java:61)
at junit.framework.TestSuite.addTestMethod(TestSuite.java:294)
at junit.framework.TestSuite.addTestsFromTestCase(TestSuite.java:150)
at junit.framework.TestSuite.(TestSuite.java:129)
at 
org.junit.internal.runners.JUnit38ClassRunner.(JUnit38ClassRunner.java:71)
at 
org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder.java:14)
at 
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at 
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at 
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at 
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to 
initialize a connection to the database for locking purposes:
at com.cloud.utils.db.Merovingian2.(Merovingian2.java:74)
at 
com.cloud.utils.db.Merovingian2.createLockMaster(Merovingian2.java:80)
at 
com.cloud.server.LockMasterListener.(LockMasterListener.java:33)
at 
com.cloud.server.ManagementServerImpl.(ManagementServerImpl.java:602)
... 27 more
Caused by: java.sql.SQLException: No suitable driver found for 
jdbc:mysql://localhost:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true&prepStmtCacheSqlLimit=4096
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at 
org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:75)
at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)

--Alex

RE: failing unit tests....

[Alex Huang]

Thanks.  In case this helpsAlena helped me do some digging.  The reason is 
because the unit test creates ManagementServer which causes the JVM to 
initialize all static variables which causes new code that moved the static 
LockMaster to initialize which which causes the requirement on the JDBC 
drivers.  When running the unit tests, there's no jdbc drivers and so it fails.

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Friday, October 25, 2013 2:53 PM
> To: dev@cloudstack.apache.org
> Subject: Re: failing unit tests
> 
> I'll fix that.
> 
> Darren
> 
> On Fri, Oct 25, 2013 at 1:54 PM, Alex Huang  wrote:
> > I'm getting this failing unit test when building with the latest from 
> > master.
> Anyone working on it or know what it is already?  From the stack, it looks 
> like
> it's a problem location the jdbc driver.  This was working just yesterday.
> >
> > Test set: com.cloud.alert.AlertControlsUnitTest
> > --
> > - Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
> > elapsed: 0.175 sec <<< FAILURE!
> > warning(junit.framework.TestSuite$1)  Time elapsed: 0.004 sec  <<<
> FAILURE!
> > junit.framework.AssertionFailedError: Exception in constructor:
> testInjected (java.lang.ExceptionInInitializerError
> > at
> com.cloud.alert.AlertControlsUnitTest.(AlertControlsUnitTest.java:46)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructor
> AccessorImpl.java:57)
> > at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCon
> structorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at junit.framework.TestSuite.createTest(TestSuite.java:61)
> > at junit.framework.TestSuite.addTestMethod(TestSuite.java:294)
> > at
> junit.framework.TestSuite.addTestsFromTestCase(TestSuite.java:150)
> > at junit.framework.TestSuite.(TestSuite.java:129)
> > at
> org.junit.internal.runners.JUnit38ClassRunner.(JUnit38ClassRunner.jav
> a:71)
> > at
> org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder.java:14)
> > at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.j
> ava:57)
> > at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDe
> faultPossibilitiesBuilder.java:29)
> > at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.j
> ava:57)
> > at
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
> > at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.jav
> a:234)
> > at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Prov
> ider.java:134)
> > at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java
> :113)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(Refl
> ectionUtils.java:189)
> > at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(Pr
> oviderFactory.java:165)
> > at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(Provider
> Factory.java:85)
> > at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Forked
> Booter.java:103)
> > at
> >
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:7
> > 4) Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable
> > to initialize a connection to the database for locking purposes:
> > at com.cloud.utils.db.Merovingian2.(Merovingian2.java:74)
> > at
> com.cloud.utils.db.Merovingian2.createLockMaster(Merovingian2.java:80)
> > at
> com.cloud.server.LockMasterListener.(LockMasterListener.java:33)
> > at
> com.cloud.server.ManagementServerImpl.(ManagementServerImpl.
> java:602)
> > ... 27 more
> > Caused by: j

RE: failing unit tests....

[Alex Huang]

Hi Laszlo,

Thanks for the fix.  It does fix this problem.  I took a quick look.

As I understand it from Alena, Darren had to move the initialization of the 
LockMasterListener higher due to some problems Mike experienced.  My guess is 
that it had to be higher because Spring cannot work with the ordered 
initialization that we used to have for the components (new, configure, start) 
so it was moved into static which relies on jvm initialization order.  So 
moving the initialization of LockMasterListener back into start() method of 
ManagementServer basically will break things for Mike again.  I'll leave it to 
Darren to resolve this for now.  I don't know enough about Spring loading to 
resolve this correctly for both cases.

--Alex

> -Original Message-
> From: Laszlo Hornyak [mailto:laszlo.horn...@gmail.com]
> Sent: Friday, October 25, 2013 3:19 PM
> To: dev@cloudstack.apache.org
> Subject: Re: failing unit tests
> 
> I have just sent a fix for that, looks like everything is green now. Please 
> check!
> 
> 
> On Fri, Oct 25, 2013 at 11:52 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
> 
> > I'll fix that.
> >
> > Darren
> >
> > On Fri, Oct 25, 2013 at 1:54 PM, Alex Huang 
> wrote:
> > > I'm getting this failing unit test when building with the latest
> > > from
> > master.  Anyone working on it or know what it is already?  From the
> > stack, it looks like it's a problem location the jdbc driver.  This
> > was working just yesterday.
> > >
> > > Test set: com.cloud.alert.AlertControlsUnitTest
> > >
> > --
> > -
> > > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> > > 0.175
> > sec <<< FAILURE!
> > > warning(junit.framework.TestSuite$1)  Time elapsed: 0.004 sec  <<<
> > FAILURE!
> > > junit.framework.AssertionFailedError: Exception in constructor:
> > testInjected (java.lang.ExceptionInInitializerError
> > > at
> > com.cloud.alert.AlertControlsUnitTest.(AlertControlsUnitTest.jav
> > a:46)
> > > at
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> > rAccessorImpl.java:57)
> > > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> > nstructorAccessorImpl.java:45)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > > at junit.framework.TestSuite.createTest(TestSuite.java:61)
> > > at junit.framework.TestSuite.addTestMethod(TestSuite.java:294)
> > > at
> > junit.framework.TestSuite.addTestsFromTestCase(TestSuite.java:150)
> > > at junit.framework.TestSuite.(TestSuite.java:129)
> > > at
> > org.junit.internal.runners.JUnit38ClassRunner.(JUnit38ClassRunne
> > r.java:71)
> > > at
> > org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder
> > .java:14)
> > > at
> > org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder
> > .java:57)
> > > at
> > org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForCl
> > ass(AllDefaultPossibilitiesBuilder.java:29)
> > > at
> > org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder
> > .java:57)
> > > at
> > org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:2
> > 4)
> > > at
> > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider
> > .java:234)
> > > at
> > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4P
> > rovider.java:134)
> > > at
> > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.
> > java:113)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:57)
> > > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> s
> > orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at
> > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(R
> > eflectionUtils.java:189)
> > > at
> > org.apache.maven.

RE: ifconfig and MacAddress

[Alex Huang]

I can confirm the code's only there because Java didn't have it before.

--Alex

From: Laszlo Hornyak [mailto:laszlo.horn...@gmail.com]
Sent: Monday, October 28, 2013 11:50 PM
To: dev@cloudstack.apache.org
Cc: Alex Huang; Kelven Yang
Subject: Re: ifconfig and MacAddress

Hi,

I noticed other problem with the MacAddress, it appears that it does not handle 
the Windows ipconfig output.
(see related patch) https://reviews.apache.org/r/14514/ I tested it with win8, 
7, and XP outputs
As far as I remember this may be a problem when running CS cluster on windows, 
since each node determines the default as id.

On Sun, Oct 27, 2013 at 1:11 AM, Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>> wrote:
Is there a specific reason that in MacAddress it uses "ifconfig" or
other utilities to grab the Mac Address as opposed to the java API
java.net.NetworkInterface?  There's a comment in that code that says
that code was copied from some public domain utility.  So I'm guessing
its there just because that's what somebody wrote back before Java 6
introduced NetworkInteface.getHardwareAddress().  I'd like to rewrite
this code to use the standard API, grab the Mac deterministically and
also not use ifconfig which was deprecated 3 years ago.

Darren



--

EOF

RE: [MERGE] network-guru-orchestration into master

[Alex Huang]

I agree with Hugo that the current way of ask NetworkGuru to decide itself is 
not desirable.  It makes absolute sense to change that but I wonder why do it 
through capabilities.  Why not just ask the system admin to choose when they 
setup the zone?   When I originally designed NetworkGuru, I figured there's a 
difference in the L2-L3 technology deployed and the network services (L4-L7) 
provided on top of it.  So I separated out NetworkGuru and NetworkElement.  
However, I didn't think it's likely that there would be two or three different 
type of L2-L3 technologies deployed within the same zone.  I figured we can 
leave it to the system admin to just pick which set of NetworkGurus should be 
deployed in a certain zone and that's good enough. 

I do think there should be more tie in between the NetworkElements and 
NetworkGurus.  For example, whether a certain NetworkElement can work with the 
L2-L3 technology deployed.

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Wednesday, October 30, 2013 10:11 AM
> To: dev@cloudstack.apache.org; Alex Huang; Chiradeep Vittal
> Subject: Re: [MERGE] network-guru-orchestration into master
> 
> I don't particular like saying that picking a Guru is based solely on that 
> criteria.
> It should be much looser than that.  If the problem you are trying to fix is 
> two
> Gurus implement the same network then we need to set back a bit.  I'd like
> Alex or Chiradeep to weigh in on this.  Currently setupNetwork returns a list
> of networks that were created.  I've looked at the code and every single
> invocation to setupNetwork expects (and mostly hard code) a response of
> one network.
> 
> So for a more proper fix I'd say that the API of setupNetwork should just
> return Network.  Additionally NetworkGuru should have a
> canHandle/canDesign method on the interface that returns a priority/score
> (similar to what was just done in the storage strategies).  Then the
> orchestrator can find the best match and then use the correct guru.  Now the
> consolidated logic that you've done should be in a abstract base class that
> handles the basic cases of traffic type, network type, etc.
> 
> Darren
> 
> On Wed, Oct 30, 2013 at 6:10 AM, Hugo Trippaers  wrote:
> >
> > On 30 okt. 2013, at 02:09, Darren Shepherd 
> wrote:
> >
> >> First, I like the idea of consolidating logic.  I could see also
> >> implementing this as just an abstract base class that handles this
> >> common logic.  I'm not sure which approach I prefer.  Exactly what is
> >> the problem that you are trying to solve?  Without more details, I'd
> >> lean towards implementing this logic in an abstract base class that
> >> all NetworkGurus can choose to extend.
> >>
> >
> > Not as much a problem as a design choice. It is my intention to use
> > capabilities to select the NetworkGuru instead of asking each network
> > guru in turn if it wants to design the network. The idea here is that
> > new network gurus only need to supply a list of capabilities to be
> > integrated into cloudstack. Like i can handle isolation type X in
> > advanced networks for traffic type Guest. The network orchestrator can
> > make an informed decision on who to give the task (and warn if there
> > is more than one capable)
> >
> >
> >> Darren
> >>
> >> On Tue, Oct 29, 2013 at 9:42 AM, Hugo Trippaers 
> wrote:
> >>> Hey guys,
> >>>
> >>> This is something i had on my wish list for some time. The current way
> network gurus are handled is that each guru is asked to design a network and
> will decide by itself to either do it or don't. Most gurus have sane checks on
> which types of networks it can handle, but we have seen issues there in the
> past.
> >>>
> >>> With these changes the network orchestrator will check the capabilities
> of a guru and based on that ask one or more gurus to design the network.
> With this the power is where is should new, the network orchestrator.
> >>>
> >>> This also means that new networking plugins with gurus will have an
> easier integration, just list the capabilities. It will also save some 
> database
> calls (once i clean out all canHandle functions) as gurus will have to lookup
> less to decide if they should to their job.
> >>>
> >>> What do you guys think?
> >>>
> >>> Current branch is tested with devcloud in a manual test, so there is more
> work to do there. I wanted to get some opinions while performing more
> tests.
> >>>
> >>> Cheers,
> >>>
> >>> Hugo
> >

uploading to maven repository

[Alex Huang]

Hi all,

I have a chance to convince xenserver to upload their java stubs to maven so 
that we don't have to keep a copy of it in ours source code.  They're stuck on 
the whole upload process as they've never dealt with maven before.  Is there 
anyone who can walk me through how to do it?

Reading the maven guide[1], it seems to easiest would be to just upload the 
artifact[2].  Is that the right approach?  

Thanks.

--Alex
[1] http://maven.apache.org/guides/mini/guide-central-repository-upload.html
[2] 
https://docs.sonatype.org/display/Repository/Uploading+3rd-party+Artifacts+to+The+Central+Repository

RE: Eclipse Question

[Alex Huang]

Usually problems like that goes away if you refresh your project or do a clean 
on the project.  Eclipse just get confused.

I never know which one helps for sure but when this gets really bad, these are 
the steps I do.
Project->Turn off Build Automatically
Project->Clean
Highlight all projects -> Right click -> Refresh
Highlight all projects -> Right click -> Maven -> Update Project
Make sure you only highlight cloudstack maven projects, not any source files or 
other projects, inside the project or else the options may not appear. 
Project->Turn on Build automatically

When you do this, then eclipse will rebuild all binaries and it should clear up 
any problems it has recognizing class hierarchies etc.

--Alex

> -Original Message-
> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> Sent: Tuesday, November 12, 2013 9:34 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Eclipse Question
> 
> "mouseClicked(MouseEvent) must override MouseClicked(MouseEvent)"
> 
> I meant for the second mouseClicked in that line to start with a lower-case
> 'm' by the way.
> 
> 
> On Tue, Nov 12, 2013 at 10:33 AM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
> 
> > Hi,
> >
> > I fetched and rebased on top of master yesterday and noticed in
> > Eclipse that I now have 61 error messages.
> >
> > Many of them are related to @Override, but I am not sure why Eclipse
> > is complaining.
> >
> > For example:
> >
> > package common;
> >
> > import java.awt.event.MouseEvent;
> > import java.awt.event.MouseListener;
> > import java.awt.event.MouseMotionListener;
> >
> > import streamer.BaseElement;
> > import streamer.ByteBuffer;
> >
> > public class AwtMouseEventSource extends BaseElement implements
> > MouseListener, MouseMotionListener {
> >
> >   public AwtMouseEventSource(String id) {
> > super(id);
> >   }
> >
> >   @Override
> >   public void mouseClicked(MouseEvent e) {
> > // Nothing to do
> >   }
> >
> > Eclipse says mouseClicked(MouseEvent) must override
> > MouseClicked(MouseEvent) for us to be able to use the @Override
> annotation.
> >
> > That makes sense, of course.
> >
> > The problem is that we do override that method properly: It comes from
> > the MouseListener interface.
> >
> > Any thoughts on why Eclipse does not recognize this? It seems to only
> > be looking in the class we extend and the class that that class extends.
> >
> > Thanks!
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkow...@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud
> > *(tm)*
> >
> 
> 
> 
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *(tm)*

[ANNOUNCE] New committer: Sangeetha Hariharan

[Alex Huang]

The Project Management Committee (PMC) for Apache CloudStack has asked 
Sangeetha Hariharan  to become a committer and we are pleased to announce that 
they have accepted.

Being a committer allows many contributors to contribute more autonomously. For 
developers, it makes it easier to submit changes and eliminates the need to 
have contributions reviewed via the patch submission process. Whether 
contributions are development-related or otherwise, it is a recognition of a 
contributor's participation in the project and commitment to the project and 
the Apache Way.

Please join me in congratulating Sangeetha!

--Alex
on behalf of the CloudStack PMC

RE: Convention on UUID column

[Alex Huang]

+1 on adding the constraints.  Just make sure you add them after upgrading the 
data.

Populating with id in the upgrade prevents exactly the problem you've 
described.  That's why we're doing it.


--Alex

> -Original Message-
> From: Nitin Mehta [mailto:nitin.me...@citrix.com]
> Sent: Thursday, May 23, 2013 11:09 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Convention on UUID column
> 
> Agree with Koushik. Lets enforce the not null constraint on all the UUID 
> fields
> now that we are populating it.
> 
> Also wanted to ask why we are populating it with the ID and not a generated
> UUID ? Doesn't break the backward compatibilty going either ways but the
> later is more consistent right and can be used in future as they are always
> unique across the system - say for storing templates directly under UUID of
> the template/vol/snapshot on sec. storage than using the folder structure of
> templates/account_id/template_id/real_template ?
> Clients who have hardcoded against CS for system vm ids, service offering ids
> etc. will break definitely though - is that ok ?
> 
> Thanks,
> -Nitin
> 
> On 24/05/13 9:26 AM, "Koushik Das"  wrote:
> 
> >
> >It is better to add constraints in the db for all uuid fields. That way
> >uuid field will never get missed out. I see that for some tables there
> >is a NOT NULL constraint.
> >
> >-Koushik
> >
> >> -Original Message-
> >> From: Min Chen [mailto:min.c...@citrix.com]
> >> Sent: Friday, May 24, 2013 4:09 AM
> >> To: dev@cloudstack.apache.org
> >> Subject: Convention on UUID column
> >>
> >> Hi there,
> >>
> >> During API refactoring efforts, Rohit and I run into several issues
> >>due to  empty UUID column for existing DB entries. Since UUID was
> >>introduced later  in 3.0.x, we have to always conditionally handle
> >>existing customers with  empty UUID columns for different entities,
> >>causing much headache for  various upgrade cases. To make sure that we
> >>have a consistent upgrade base  for all 4.1 customers later, in
> >>schema-410to420.sql, we have added sql scripts  to populate UUID
> >>column of all pre-4.1 schema tables with values from their  ID column
> >>if UUID column is NULL. To make this work properly, we require  that
> >>all UUID columns should be populated with values when you add new
> >>data into both pre-4.1 schema or post-4.1 schema. I just noticed that
> >>this  assumption may not be well known to community, since I am seeing
> >>this sql  in schema-410to420.sql:
> >>
> >>
> >> INSERT INTO `cloud`.`vm_template` (id, unique_name, name, public,
> >>created, type, hvm, bits, account_id, url, checksum, enable_password,
> >>display_text, format, guest_os_id, featured, cross_zones,
> >>hypervisor_type)
> >>
> >>  VALUES (10, 'routing-10', 'SystemVM Template (LXC)', 0, now(),
> >>'SYSTEM',  0, 64, 1,
> >>'http://download.cloud.com/templates/acton/acton-systemvm-
> >> 02062012.qcow2.bz2', '2755de1f9ef2ce4d6f2bee2efbb4da92', 0,
> 'SystemVM
> >>Template (LXC)', 'QCOW2', 15, 0, 1, 'LXC');
> >>
> >>
> >> Instead, this should be modified as:
> >>
> >>
> >> INSERT INTO `cloud`.`vm_template` (id, uuid, unique_name, name,
> >>public,  created, type, hvm, bits, account_id, url, checksum,
> >>enable_password,  display_text, format, guest_os_id, featured,
> >>cross_zones,
> >>hypervisor_type)
> >>
> >>  VALUES (10, UUID(), 'routing-10', 'SystemVM Template (LXC)', 0,
> >>now(),  'SYSTEM', 0, 64, 1,
> >>'http://download.cloud.com/templates/acton/acton-
> >> systemvm-02062012.qcow2.bz2', '2755de1f9ef2ce4d6f2bee2efbb4da92', 0,
> >>'SystemVM Template (LXC)', 'QCOW2', 15, 0, 1, 'LXC');
> >>
> >> I have made this fix in master, but want to raise this topic to get
> >>community's  attention on this.
> >>
> >> Thanks
> >> -min
> >>
> >

RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest

[Alex Huang]

Daan,

The other possibility is that the test is affected by another test so that it 
breaks when the tests are in a certain order.  Have you noticed what's the 
order of the tests ran when it breaks?

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Monday, May 27, 2013 2:37 AM
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Being in the habit of replying on my own mails as I seems to be on this list;
> 
> This seems to be related to eclipse refresh/build code. Does anybody have a
> solution to it? The problem is not consistently present but all the more
> irritating.
> 
> Regards,
> 
> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: maandag 27 mei 2013 7:49
> To: dev@cloudstack.apache.org
> Subject: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> LS,
> 
> Lately I've been getting,
> "com.cloud.exception.InvalidParameterValueException: ACL item with
> number 1 already exists in ACL: null", while building cloudstack. I first 
> thought
> it was my own hack so I don't know when it was introduced (it is in a clean
> master that it happens). The full surfire report follows:
> 
> ---
> Test set: com.cloud.vpc.NetworkACLServiceTest
> ---
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.219 sec <<<
> FAILURE!
> testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest)  Time elapsed:
> 0.016 sec  <<< ERROR!
> com.cloud.exception.InvalidParameterValueException: ACL item with
> number 1 already exists in ACL: null
> at
> com.cloud.network.vpc.NetworkACLServiceImpl.createNetworkACLItem(Ne
> tworkACLServiceImpl.java:270)
> at
> com.cloud.vpc.NetworkACLServiceTest.testCreateACLItem(NetworkACLServ
> iceTest.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
> Method.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.jav
> a:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(Framework
> Method.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMeth
> od.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java
> :28)
> at
> org.springframework.test.context.junit4.statements.RunBeforeTestMethod
> Callbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
> at
> org.springframework.test.context.junit4.statements.RunAfterTestMethodC
> allbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
> at
> org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(
> SpringRepeat.java:72)
> at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(S
> pringJUnit4ClassRunner.java:231)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> ava:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.springframework.test.context.junit4.statements.RunBeforeTestClassCall
> backs.evaluate(RunBeforeTestClassCallbacks.java:61)
> at
> org.springframework.test.context.junit4.statements.RunAfterTestClassCallb
> acks.evaluate(RunAfterTestClassCallbacks.java:71)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(Spring
> JUnit4ClassRunner.java:174)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.jav
> a:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Prov
> ider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java
> :113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
>

RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest

[Alex Huang]

Unit tests are ran in random order because each unit test should be 
self-sufficient.  I'm saying I've seen cases where unit tests aren't written to 
be self-sufficient and that's why an unit test fails sometimes and passes 
sometimes.  

In those cases, we should fix the unit test and not the order.  A common 
problem is sharing the mocked classes.  For example, unit test A tests the set 
of a mock object and unit test B tests if the field is null but they shared the 
mock instance.  In this case, unit test B passes if it is ran before unit test 
A but fails if it was ran after.

But often times the randomness of how it's ran is actually pseudo-random so it 
works consistently on someone's laptop but as soon as it gets ran on another 
machine, the order changes and it fails.  So when you see the problem such as 
this, you should always let the unit test writer know the order in which the 
unit tests were executed so that the unit test writer can try it.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Monday, May 27, 2013 9:23 AM
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> And if not how do I manipulate the order of tests?
> 
> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: maandag 27 mei 2013 17:02
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Alex,
> 
> Can this change out of the box? I had the problem on a clean checkout of
> master.
> 
> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: maandag 27 mei 2013 15:55
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Daan,
> 
> The other possibility is that the test is affected by another test so that it
> breaks when the tests are in a certain order.  Have you noticed what's the
> order of the tests ran when it breaks?
> 
> --Alex
> 
> > -Original Message-
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: Monday, May 27, 2013 2:37 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > Being in the habit of replying on my own mails as I seems to be on
> > this list;
> >
> > This seems to be related to eclipse refresh/build code. Does anybody
> > have a solution to it? The problem is not consistently present but all
> > the more irritating.
> >
> > Regards,
> >
> > -Original Message-
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: maandag 27 mei 2013 7:49
> > To: dev@cloudstack.apache.org
> > Subject: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> >
> > LS,
> >
> > Lately I've been getting,
> > "com.cloud.exception.InvalidParameterValueException: ACL item with
> > number 1 already exists in ACL: null", while building cloudstack. I
> > first thought it was my own hack so I don't know when it was
> > introduced (it is in a clean master that it happens). The full surfire 
> > report
> follows:
> >
> > --
> > - Test set: com.cloud.vpc.NetworkACLServiceTest
> > --
> > - Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time
> > elapsed: 0.219 sec <<< FAILURE!
> > testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest)  Time elapsed:
> > 0.016 sec  <<< ERROR!
> > com.cloud.exception.InvalidParameterValueException: ACL item with
> > number 1 already exists in ACL: null
> > at
> >
> com.cloud.network.vpc.NetworkACLServiceImpl.createNetworkACLItem(Ne
> > tworkACLServiceImpl.java:270)
> > at
> >
> com.cloud.vpc.NetworkACLServiceTest.testCreateACLItem(NetworkACLServ
> > iceTest.java:141)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > Source)
> > at java.lang.reflect.Method.invoke(Unknown Source)
> > at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
> > Method.java:45)
> > at
> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCall
> > ab

RE: PLEASE: master build breaks at com.cloud.vpc.NetworkACLServiceTest

[Alex Huang]

Kishan,

Git blame shows the file is checked in by you.  Can you take a look at why it's 
failing?

Thanks.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Monday, May 27, 2013 9:39 AM
> To: dev@cloudstack.apache.org
> Subject: PLEASE: master build breaks at
> com.cloud.vpc.NetworkACLServiceTest
> 
> I ran the test in eclipse, it passes, next I do a maven build in eclipse 
> which fails
> on this test. How do I force it to pass, please. This is costing me serious 
> time. I
> am not working on ACLs
> 
> Regards,
> Daan Hoogland
> 
> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: maandag 27 mei 2013 18:23
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> And if not how do I manipulate the order of tests?
> 
> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: maandag 27 mei 2013 17:02
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Alex,
> 
> Can this change out of the box? I had the problem on a clean checkout of
> master.
> 
> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: maandag 27 mei 2013 15:55
> To: dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Daan,
> 
> The other possibility is that the test is affected by another test so that it
> breaks when the tests are in a certain order.  Have you noticed what's the
> order of the tests ran when it breaks?
> 
> --Alex
> 
> > -Original Message-
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: Monday, May 27, 2013 2:37 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: master build breaks at
> > com.cloud.vpc.NetworkACLServiceTest
> >
> > Being in the habit of replying on my own mails as I seems to be on
> > this list;
> >
> > This seems to be related to eclipse refresh/build code. Does anybody
> > have a solution to it? The problem is not consistently present but all
> > the more irritating.
> >
> > Regards,
> >
> > -Original Message-
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: maandag 27 mei 2013 7:49
> > To: dev@cloudstack.apache.org
> > Subject: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> >
> > LS,
> >
> > Lately I've been getting,
> > "com.cloud.exception.InvalidParameterValueException: ACL item with
> > number 1 already exists in ACL: null", while building cloudstack. I
> > first thought it was my own hack so I don't know when it was
> > introduced (it is in a clean master that it happens). The full surfire 
> > report
> follows:
> >
> > --
> > - Test set: com.cloud.vpc.NetworkACLServiceTest
> > --
> > - Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time
> > elapsed: 0.219 sec <<< FAILURE!
> > testCreateACLItem(com.cloud.vpc.NetworkACLServiceTest)  Time elapsed:
> > 0.016 sec  <<< ERROR!
> > com.cloud.exception.InvalidParameterValueException: ACL item with
> > number 1 already exists in ACL: null
> > at
> >
> com.cloud.network.vpc.NetworkACLServiceImpl.createNetworkACLItem(Ne
> > tworkACLServiceImpl.java:270)
> > at
> >
> com.cloud.vpc.NetworkACLServiceTest.testCreateACLItem(NetworkACLServ
> > iceTest.java:141)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > Source)
> > at java.lang.reflect.Method.invoke(Unknown Source)
> > at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
> > Method.java:45)
> > at
> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCall
> > able.jav
> > a:15)
> > at
> > org.junit.runners.model.FrameworkMethod.invokeExplosively(Framework
> > Method.java:42)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMeth
> > od.java:20)
> > at
> > org.junit.internal.runne

RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest

[Alex Huang]

Kishan,

The problem is caused by these two methods.  I hope everyone reads this because 
I know some of us are new to writing unit tests.  The problem is basically as I 
described in a previous email.  In here, both tests makes use of and shares of 
the _networkACLItemDao and the mock is shared because the DAO is a singleton 
and is injected.  Now, note that testCreateACLItem() invokes the method 
findByAclAndNumber() via the _networkAclMgr but it didn't specifically mock the 
return value of the method in the test.  If this test is ran before 
testCreateACLItemDuplicateNumber(), then it will be successful because by 
default mocks return null.  However, testCreateACLItemDuplicateNumber() mocks 
findByAclAndNumber() to return a value, so if testCreateACLItem() is ran after 
testCreateACLItemDuplicateNumber(), the test will fail because a value is 
returned and the condition inside _networkAclMgr will fire just as Daan found 
it.

I was hoping maven prints out the order of the tests, but from Daan's log, it 
only printed out the order of the bigger test suite so it's not helpful.  
However, it's not too hard to find.  Just go to the code with the error, check 
the method that fails and go back to the test and see if your test mocked that 
method and if it didn't, was it mocked by any other tests.

@Test
public void testCreateACLItem() throws Exception {
Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new VpcVO());

Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenReturn(acl);
Mockito.when(_networkAclMgr.createNetworkACLItem(Mockito.anyInt(), 
Mockito.anyInt(), Mockito.anyString(), Mockito.anyList(), Mockito.anyInt(), 
Mockito.anyInt(),
Mockito.any(NetworkACLItem.TrafficType.class), 
Mockito.anyLong(),  Mockito.anyString(), Mockito.anyInt())).thenReturn(new 
NetworkACLItemVO());
assertNotNull(_aclService.createNetworkACLItem(createACLItemCmd));
}

@Test(expected = InvalidParameterValueException.class)
public void testCreateACLItemDuplicateNumber() throws Exception {
Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new VpcVO());

Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenReturn(acl);
Mockito.when(_networkACLItemDao.findByAclAndNumber(Mockito.anyLong(), 
Mockito.anyInt())).thenReturn(new NetworkACLItemVO());
_aclService.createNetworkACLItem(createACLItemCmd);
}

> -Original Message-
> From: Kishan Kavala
> Sent: Wednesday, May 29, 2013 4:48 AM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Daan,
>  I mocked the method causing the failure (I couldn't replicate the error you
> are seeing though). Please check with latest master and let us know if you
> still see issues.
> 
> ~kishan
> 
> From: Daan Hoogland [dhoogl...@schubergphilis.com]
> Sent: Wednesday, May 29, 2013 1:49 PM
> To: dev@cloudstack.apache.org
> Cc: Alex Huang
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> H,
> 
> I hope this is what Alex means the test programmer needs.
> 
> The order that is reported is as follows (I included the server test run 
> only):
> ---
>  T E S T S
> ---
> Running com.cloud.alert.AlertControlsUnitTest
> log4j:WARN No appenders could be found for logger
> (com.cloud.alert.AlertControlsUnitTest).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.546 sec 
> Running
> com.cloud.capacity.CapacityManagerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.11 sec 
> Running
> com.cloud.configuration.ConfigurationManagerTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.499 sec 
> Running
> com.cloud.configuration.ValidateIpRangeTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec 
> Running
> com.cloud.event.EventControlsUnitTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec Running
> com.cloud.keystore.KeystoreTest
> org.apache.cloudstack.api.response.UserVmResponse/null/{"id":"3","securit
> ygroup":[],"nic":[],"tags":[],"affinitygroup":[]}
> org.apache.cloudstack.api.response.AlertResponse/null/{"id":"100","descrip
> tion":"Hello"}
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109 sec 
> Running
> com.cloud.metadata.Resource

RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest

[Alex Huang]

Daan,

I'm new to mockito as well and I haven't had time to really look at what the 
right patterns are.  In general, in my dealings with other mocking frameworks, 
I have never liked having the mocking framework returning a default value.   
I've always like the mocking framework to throw an exception if the method 
being access was not explicitly set.  Not sure if that can be a setting in 
mockito.

Thanks.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Wednesday, May 29, 2013 8:58 AM
> To: 'dev@cloudstack.apache.org'
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Alex,
> 
> Thanks for the mockito lesson. I think a lot of us are now going to improve
> their tests.
> Can one make mockito forget the when-conditions? I would intuitively say
> MockitoAnnotations.initMocks(this); would do this.
> 
> Alternative to the answer to that last question would be to follow the
> javadoc for when:
> //you can set different behavior for consecutive method calls.
>  //Last stubbing (e.g: thenReturn("foo")) determines the behavior of further
> consecutive calls.
>  when(mock.someMethod("some arg"))
>   .thenThrow(new RuntimeException())
>   .thenReturn("foo");
> 
>  //Alternative, shorter version for consecutive stubbing:
>  when(mock.someMethod("some arg"))
>   .thenReturn("one", "two");
>  //is the same as:
>  when(mock.someMethod("some arg"))
>   .thenReturn("one")
>   .thenReturn("two");
> 
> Regards,
> Daan (not new to unit-tests sec but to mockito though) Hoogland
> 
> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: woensdag 29 mei 2013 16:34
> To: Kishan Kavala; dev@cloudstack.apache.org
> Subject: RE: master build breaks at com.cloud.vpc.NetworkACLServiceTest
> 
> Kishan,
> 
> The problem is caused by these two methods.  I hope everyone reads this
> because I know some of us are new to writing unit tests.  The problem is
> basically as I described in a previous email.  In here, both tests makes use 
> of
> and shares of the _networkACLItemDao and the mock is shared because the
> DAO is a singleton and is injected.  Now, note that testCreateACLItem()
> invokes the method findByAclAndNumber() via the _networkAclMgr but it
> didn't specifically mock the return value of the method in the test.  If this 
> test
> is ran before testCreateACLItemDuplicateNumber(), then it will be successful
> because by default mocks return null.  However,
> testCreateACLItemDuplicateNumber() mocks findByAclAndNumber() to
> return a value, so if testCreateACLItem() is ran after
> testCreateACLItemDuplicateNumber(), the test will fail because a value is
> returned and the condition inside _networkAclMgr will fire just as Daan
> found it.
> 
> I was hoping maven prints out the order of the tests, but from Daan's log, it
> only printed out the order of the bigger test suite so it's not helpful.
> However, it's not too hard to find.  Just go to the code with the error, check
> the method that fails and go back to the test and see if your test mocked that
> method and if it didn't, was it mocked by any other tests.
> 
> @Test
> public void testCreateACLItem() throws Exception {
> Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new
> VpcVO());
> 
> Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenR
> eturn(acl);
> 
> Mockito.when(_networkAclMgr.createNetworkACLItem(Mockito.anyInt(),
> Mockito.anyInt(), Mockito.anyString(), Mockito.anyList(), Mockito.anyInt(),
> Mockito.anyInt(),
> Mockito.any(NetworkACLItem.TrafficType.class), 
> Mockito.anyLong(),
> Mockito.anyString(), Mockito.anyInt())).thenReturn(new
> NetworkACLItemVO());
> assertNotNull(_aclService.createNetworkACLItem(createACLItemCmd));
> }
> 
> @Test(expected = InvalidParameterValueException.class)
> public void testCreateACLItemDuplicateNumber() throws Exception {
> Mockito.when(_vpcMgr.getVpc(Mockito.anyLong())).thenReturn(new
> VpcVO());
> 
> Mockito.when(_networkAclMgr.getNetworkACL(Mockito.anyLong())).thenR
> eturn(acl);
> 
> Mockito.when(_networkACLItemDao.findByAclAndNumber(Mockito.anyLon
> g(), Mockito.anyInt())).thenReturn(new NetworkACLItemVO());
> _aclService.createNetworkACLItem(createACLItemCmd);
> }
> 
> > -Original Message-
> > From: Kishan Kavala
> > Sent: Wednesday, May 29, 2013 4:48 AM
> > To: dev@cloudstack.apache.org
> > Cc: Alex Huang
> > Subject: RE: master build breaks at
&

RE: maintenance mode for secondary storage?

[Alex Huang]

Daan,

We have a procedure for doing that.  The problem in general has to do with the 
large capacity of the secondary storage so it's actually better to do migration 
of data outside of cloudstack with some manual process in stages and then just 
fix up the secondary storage in the database.  

I do agree having an API to do these fixups are better than direct database 
manipulations.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Thursday, May 30, 2013 11:39 AM
> To: dev
> Subject: Re: maintenance mode for secondary storage?
> 
> Yes Chiradeep,
> 
> That or maintenance on the secondary storage machine itself.
> 
> 
> On Thu, May 30, 2013 at 7:46 PM, Chiradeep Vittal <
> chiradeep.vit...@citrix.com> wrote:
> 
> > I can see a use for that: that the secondary storage becomes read-only
> > during the maintenance (so no template creation/snapshots/etc). This
> > allows for stuff like migration to new (file server) hardware.
> >
> > On 5/30/13 10:20 AM, "Nitin Mehta"  wrote:
> >
> > >Not that I have heard of. You can at best replace the sec. storage
> > >(there is a manual procedure for that), but since generally there is
> > >a single sec. storage how can you implement maintenance mode ?
> > >Can you please explain is your use case ?
> > >
> > >On 30/05/13 7:32 PM, "Daan Hoogland" 
> > wrote:
> > >
> > >>LS,
> > >>
> > >>Has there ever been discussion about implementing maintenance mode
> > >>for secondary storage? And are people thinking about this?
> > >>
> > >>thanks
> > >>Daan Hoogland
> > >
> >
> >

RE: [VOTE] Pushback 4.2.0 Feature Freeze

[Alex Huang]

+1 [binding]

--Alex

> -Original Message-
> From: Will Chan [mailto:will.c...@citrix.com]
> Sent: Monday, June 3, 2013 11:08 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [VOTE] Pushback 4.2.0 Feature Freeze
> 
> +1 [Binding]
> 
> It looks like there are a couple of last minute features that would make 4.1 a
> superb release.   I would say that we should not allow any new features that
> haven't already been proposed and that the extension does not go beyond 4
> weeks.  If beyond that, I'm a  -1.
> 
> Will
> 
> > -Original Message-
> > From: Chip Childers [mailto:chip.child...@sungard.com]
> > Sent: Friday, May 31, 2013 8:00 AM
> > To: dev@cloudstack.apache.org
> > Subject: [VOTE] Pushback 4.2.0 Feature Freeze
> >
> > Following our discussion on the proposal to push back the feature
> > freeze date for 4.2.0 [1], we have not yet achieved a clear consensus.
> Well...
> > we have already defined the "project rules" for figuring out what to do.
> > In out project by-laws [2], we have defined a "release plan" decision
> > as
> > follows:
> >
> > > 3.4.2. Release Plan
> > >
> > > Defines the timetable and work items for a release. The plan also
> > > nominates a Release Manager.
> > >
> > > A lazy majority of active committers is required for approval.
> > >
> > > Any active committer or PMC member may call a vote. The vote must
> > > occur on a project development mailing list.
> >
> > And our lazy majority is defined as:
> >
> > > 3.2.2. Lazy Majority - A lazy majority vote requires 3 binding +1
> > > votes and more binding +1 votes than binding -1 votes.
> >
> > Our current plan is the starting point, so this VOTE is a vote to
> > change the current plan.  We require a 72 hour window for this vote,
> > so IMO we are in an odd position where the feature freeze date is at
> > least extended until Tuesday of next week.
> >
> > Our current plan of record for 4.2.0 is at [3].
> >
> > [1] http://markmail.org/message/vi3nsd2yo763kzua
> > [2] http://s.apache.org/csbylaws
> > [3]
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Cloudstack+4.2
> > +Release
> >
> > 
> >
> > I'd like to call a VOTE on the following:
> >
> > Proposal: Extend the feature freeze date for our 4.2.0 feature release
> > from today (2013-05-31) to 2013-06-28.  All other dates following the
> > feature freeze date in the plan would be pushed out 4 weeks as well.
> >
> > Please respond with one of the following:
> >
> > +1 : change the plan as listed above
> > +/-0 : no strong opinion, but leaning + or -
> > -1 : do not change the plan
> >
> > This vote will remain open until Tuesday morning US eastern time.
> >
> > -chip

RE: [ANNOUNCE] New CloudStack PMC Member: John Burwell

[Alex Huang]

Congrats!  Well deserved!

--Alex

> -Original Message-
> From: Chip Childers [mailto:chipchild...@apache.org]
> Sent: Tuesday, June 4, 2013 6:35 AM
> To: dev@cloudstack.apache.org
> Subject: [ANNOUNCE] New CloudStack PMC Member: John Burwell
> 
> The Apache CloudStack PMC has invited John Burwell to join the PMC, and
> he has accepted.  Congrats John!
> 
> -chip

RE: [DISCUSS] code-freeze and integration tests

[Alex Huang]

It should not affect tests.  We want to move to mostly if not all automated 
tests.  If we freeze the tests that means no more testing for the release.  :P

--Alex

> -Original Message-
> From: Prasanna Santhanam [mailto:t...@apache.org]
> Sent: Wednesday, June 5, 2013 10:20 PM
> To: CloudStack Dev
> Subject: [DISCUSS] code-freeze and integration tests
> 
> Hi,
> 
> I would like to get everyone's opinions on the timeline and policies for
> bringing in automated tests into the repo. Integration tests are written in
> marvin by various people today within and without Citrix.
> Similar to docs I'd like to propose that tests can be committed to the
> repository beyond the freeze date.
> 
> Right now all tests are being committed to master since that's the branch that
> we cut our releases out of. But after the branch for a release has been cut
> tests will be committed to both release branch and master if everyone
> agrees this is a good thing.
> 
> Thoughts?
> 
> --
> Prasanna.,
> 
> 
> Powered by BigRock.com

RE: Quick DB Question

[Alex Huang]

Mike,

The Data Access Layer page[1] have the answer to this.   Specifically this 
particular part in the example.

// getters and setters must follow the Java
// convention of putting get/set in front of
// the field name.
public String getText() {
return text;
}

get_iscsiName is not a Java coding convention.  We should not use it.  The 
coding convention is "get" followed by the variable name with the first 
character capitalized.

--Alex
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Data+Access+Layer


> -Original Message-
> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> Sent: Friday, June 7, 2013 3:12 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Quick DB Question
> 
> Hi,
> 
> Yeah, I should have been more explicit in what problem I was seeing. :)
> 
> What I'm seeing is that I set the field with a string (IQN) that is not null, 
> but it
> doesn't make it to the DB.
> 
> I have other, similar "sets" and all is well with them (their data make it to 
> the
> DB just fine).
> 
> Perhaps it's getting overwritten later in the processing with a null. I'll 
> have to
> look into it more.
> 
> I'm actually on master, as well. I do see an "iscsi_name" field.
> varchar(255) in the cloud.volumes table. I haven't updated in a week or so,
> but I doubt the column's been removed. Strange.
> 
> I'm developing a storage plug-in which creates an iSCSI volume on a SAN and
> I just wanted to use (what I thought was) an existing column to store the IQN.
> It looked like the "iscsi_name" column would be a good place to store this
> info.
> 
> Thanks!
> 
> 
> On Fri, Jun 7, 2013 at 4:05 PM, Vijayendra Bhamidipati <
> vijayendra.bhamidip...@citrix.com> wrote:
> 
> > Hi Mike,
> >
> > You're probably calling those setter methods in the constructor and I
> > don't see any problem having an '_' in the function name. What is the
> > problem you're seeing?
> >
> > Also I don't see this iscsi_name in VolumeVO.java on master - I'm
> > guessing you're working off a private branch.
> >
> > I'm yet to go through your earlier mails on the alias - so sorry for
> > the following question if it has already been discussed - why do you
> > want to put an iscsi iqn here in volumeVO? Isn't it better to put in a
> > class of its own that derives VolumeVO?
> >
> > Regards,
> > Vijay
> >
> > -Original Message-
> > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > Sent: Friday, June 07, 2013 2:52 PM
> > To: dev@cloudstack.apache.org
> > Subject: Quick DB Question
> >
> > Hi,
> >
> > I'd like to place an IQN in the "iscsi_name" field available in the
> > cloud.volumes table after I create an appropriate iSCSI target on a SAN.
> >
> > For some reason, we don't seem to be using this column in the VolumeVO
> > class, so I went ahead and added access to it.
> >
> > I've successfully added columns to tables before in CloudStack and
> > created read/write access to them, but I am - for some reason - having
> > trouble with this "iscsi_name" column.
> >
> > In the VolumeVO class, I've added the following (below). Can anyone
> > see any flaws in what I've done? I could have made the member variable
> > private (I just copied, pasted, and modified an existing field), but
> > that shouldn't matter for this purpose. I don't usually use the "_" in
> > a method name, but it just looked better to me in this case.
> >
> > Thanks!
> >
> > @Column(name = "iscsi_name")
> >
> > String iScsiName;
> >
> >
> > @Override
> >
> > public String get_iScsiName() {
> >
> > return this.iScsiName;
> >
> > }
> >
> >
> >
> > public void set_iScsiName(String iScsiName) {
> >
> > this.iScsiName = iScsiName;
> >
> > }
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkow...@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud
> > *(tm)*
> >
> 
> 
> 
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *(tm)*

Re: Review Request: The equals method checks type against Pair

[Alex Huang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11752/#review21676
---

Ship it!


Commit ID on master: a1a68ed0c4b722f6f83733d0b110debde2a1041e

- Alex Huang


On June 8, 2013, 9:12 p.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11752/
> ---
> 
> (Updated June 8, 2013, 9:12 p.m.)
> 
> 
> Review request for cloudstack.
> 
> 
> Description
> ---
> 
> The equals method checks type against Pair - may be a copy-paste error
> 
> 
> Diffs
> -
> 
>   utils/src/com/cloud/utils/Ternary.java fee69ee 
>   utils/test/com/cloud/utils/TernaryTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/11752/diff/
> 
> 
> Testing
> ---
> 
> Test included
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

Re: Review Request: use commons-lang StringUtils

[Alex Huang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11767/#review21678
---


The patch did not apply cleanly.  Please resubmit.  Since you have to resubmit 
anyways, I think why not just change all of the code to use the commons.lang 
version.  I see no point in keeping the method in StringUtils if it's 
replaceable by one in a standard library.


- Alex Huang


On June 9, 2013, 7:47 p.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11767/
> ---
> 
> (Updated June 9, 2013, 7:47 p.m.)
> 
> 
> Review request for cloudstack.
> 
> 
> Description
> ---
> 
> commons-lang is already a transitive dependency of the utils project, which 
> allows removing some duplicated functionality.
> This patch replaces StringUtils.join(String, Object...) with it's 
> commons-lang counterpart.
> It also replaces calls to String join(Iterable, String) in 
> cases where an array is already exist and it is only wrapped into a List.
> 
> 
> Diffs
> -
> 
>   server/src/com/cloud/storage/s3/S3ManagerImpl.java 61e5573 
>   
> services/secondary-storage/src/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java
>  e7fa5b2 
>   utils/src/com/cloud/utils/S3Utils.java b7273a1 
>   utils/src/com/cloud/utils/StringUtils.java 14ff4b1 
>   utils/test/com/cloud/utils/StringUtilsTest.java 3c162c7 
> 
> Diff: https://reviews.apache.org/r/11767/diff/
> 
> 
> Testing
> ---
> 
> - Unit test added
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

RE: Contributing as a non-committer

[Alex Huang]

Paul,

That line is old.  You should follow this page.

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+up+CloudStack+Development+Environment

--Alex

> -Original Message-
> From: Paul Angus [mailto:paul.an...@shapeblue.com]
> Sent: Monday, June 10, 2013 6:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: Contributing as a non-committer
> 
> Thanks Seb,
> I've bought pro git as extra reading too.
> 
> Now I'm trying to setup my dev environment, the 'Setting Up an Apache
> CloudStack Development Environment' says "Most Apache CloudStack
> developers use Eclipse as their primary IDE. CloudStack source code already
> includes Eclipse .project file in each project folder"
> 
> But I can't find the .project file in the CloudStack directory...?
> 
> 
> Regards,
> 
> Paul Angus
> S: +44 20 3603 0540 | M: +447711418784
> paul.an...@shapeblue.com
> 
> -Original Message-
> From: Sebastien Goasguen [mailto:run...@gmail.com]
> Sent: 09 June 2013 08:46
> To: dev@cloudstack.apache.org
> Subject: Re: Contributing as a non-committer
> 
> Paul, I fixed the wiki link.
> 
> Thanks for reporting it, note that if you have a wiki account you can edit the
> page yourself and edit it.
> 
> Let us know if the instructions are not clear. I also have a screencast on git
> basics for non-committers:
> http://www.youtube.com/watch?v=3c5JIW4onGk&list=PLb899uhkHRoZCRE
> 00h_9CRgUSiHEgFDbC&index=5
> 
> 
> 
> -sebastien
> 
> On Jun 9, 2013, at 2:28 AM, Paul Angus  wrote:
> 
> > Hi Joe,
> >
> > It's on
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Home
> > under 'Developer Docs and Resources'
> >
> >
> >
> > Regards,
> >
> > Paul Angus
> > S: +44 20 3603 0540 | M: +447711418784 paul.an...@shapeblue.com
> >
> > -Original Message-
> > From: Joe Brockmeier [mailto:j...@zonker.net]
> > Sent: 08 June 2013 23:04
> > To: dev@cloudstack.apache.org
> > Subject: Re: Contributing as a non-committer
> >
> > Paul, where are you seeing that link?
> >
> > On Fri, Jun 7, 2013, at 03:55 PM, Paul Angus wrote:
> >> Guys,
> >>
> >> I'm just trying to get up to speed with how I can contribute more
> >> (starting with a minor doc fix) but the link
> >> http://cloudstack.apache.org/develop/non-contributors.html is broken.
> >>
> >> Can it be fixed pls (does it count as a bug that I need to report).
> >> Is there an alternate link for the time being?
> >>
> >>
> >> Regards
> >>
> >> Paul Angus
> >> Senior Consultant / Cloud Architect
> >>
> >> [cid:image002.png@01CE1071.C6CC9C10]
> >>
> >> S: +44 20 3603 0540 | M:
> >> +447711418784
> >> paul.an...@shapeblue.com
> |
> >> www.shapeblue.com | Twitter:@shapeblue
> >> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS
> >>
> >> ShapeBlue are proud to be sponsoring CloudStack Collaboration
> >> Conference NA
> >>
> [https://cwiki.apache.org/confluence/download/attachments/30760149/Cl
> >> o
> >>
> udStack+Collaboration+Conference+Banner+v2+Blue+Background+Only.jpg
> ?v
> >> udStack+Collaboration+Conference+Banner+v2+Blue+Background+e
> >>
> rsion=3&modificationDate=1367282397297] >> e
> >> t/>
> >>
> >> Apache CloudStack Bootcamp training courses
> >> 19/20 June,
> >> London course/>
> >> 22/23 June, Santa Clara
> >> CA
> >> 10/11 July, Bangalore,
> >> India
> >> 21/22 August,
> >> London course/>
> >>
> >> This email and any attachments to it may be confidential and are
> >> intended solely for the use of the individual to whom it is addressed.
> >> Any views or opinions expressed are solely those of the author and do
> >> not necessarily represent those of Shape Blue Ltd or related
> >> companies. If you are not the intended recipient of this email, you
> >> must neither take any action based upon its contents, nor copy or
> >> show it to anyone. Please contact the sender if you believe you have
> received this email in error.
> >> Shape Blue Ltd is a company incorporated in England & Wales.
> >> ShapeBlue Services India LLP is operated under license from Shape Blue
> Ltd.
> >> ShapeBlue is a registered trademark.
> >
> >
> > Best,
> >
> > jzb
> > --
> > Joe Brockmeier
> > j...@zonker.net
> > Twitter: @jzb
> > http://www.dissociatedpress.net/
> >
> > This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender if
> you believe y

RE: Contributing as a non-committer

[Alex Huang]

> Forget about eclipse for now :) just use vi :)

Why don't we just go back to ed?  

--Alex

RE: Build failed in Jenkins: cloudstack-rat-master #1468

[Alex Huang]

I don't know much about Jenkins.  Can this be fix?  It's not good to keep 
getting these build errors.  It hides actual errors.

Anyone know how to fix it?

--Alex

> -Original Message-
> From: Hugo Trippaers [mailto:htrippa...@schubergphilis.com]
> Sent: Tuesday, June 11, 2013 1:43 AM
> To: dev@cloudstack.apache.org
> Subject: RE: Build failed in Jenkins: cloudstack-rat-master #1468
> 
> This is local to jenkins.
> 
> It uses tag to keep track of the changes between the different runs of the
> build. It is not pushing these tags anywhere, they stay in the git environment
> in the workspace.
> 
> This looks like a new slave was not yet configured with the global identity in
> the Jenkins configuration.
> 
> Cheers,
> 
> Hugo
> 
> > -Original Message-
> > From: Animesh Chaturvedi [mailto:animesh.chaturv...@citrix.com]
> > Sent: dinsdag 11 juni 2013 7:49
> > To: dev@cloudstack.apache.org
> > Subject: RE: Build failed in Jenkins: cloudstack-rat-master #1468
> >
> > Seeing this again, shouldn't jenkins just pull and compile?
> > > -Original Message-
> > > From: David Nalley [mailto:da...@gnsa.us]
> > > Sent: Thursday, June 06, 2013 6:06 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: Build failed in Jenkins: cloudstack-rat-master #1468
> > >
> > > Why is jenkins trying to create a tag in our repo?
> > >
> > > --David
> > >
> > > On Thu, Jun 6, 2013 at 9:00 AM, Apache Jenkins Server
> > >  wrote:
> > > > See 
> > > >
> > > > --
> > > > Started by an SCM change
> > > > Building remotely on ubuntu2 in workspace
> > > > 
> > > > Checkout:cloudstack-rat-master /
> > > >  -
> > > > hudson.remoting.Channel@9907404:ubuntu2
> > > > Using strategy: Default
> > > > Last Built Revision: Revision
> > > > d98289baca7fbc8a793adadfa386e6ab234952f7
> > > > (origin/master) Fetching changes from 1 remote Git repository
> > > > Fetching upstream changes from
> > > > https://git-wip-us.apache.org/repos/asf/cloudstack.git
> > > > Commencing build of Revision
> > > > c0d894346a57e61626f332a9ef25efa9b5e77646
> > > > (origin/master) Checking out Revision
> > > > c0d894346a57e61626f332a9ef25efa9b5e77646 (origin/master)
> > > > FATAL: Could not apply tag jenkins-cloudstack-rat-master-1468
> > > > hudson.plugins.git.GitException: Could not apply tag jenkins-
> > > cloudstack-rat-master-1468
> > > > at hudson.plugins.git.GitAPI.tag(GitAPI.java:829)
> > > > at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1270)
> > > > at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1231)
> > > > at
> > > hudson.FilePath$FileCallableWrapper.call(FilePath.java:2348)
> > > > at hudson.remoting.UserRequest.perform(UserRequest.java:118)
> > > > at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> > > > at hudson.remoting.Request$2.run(Request.java:326)
> > > > at
> > > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecu
> > > to
> > > rS
> > > ervice.java:72)
> > > > at
> > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
> > > .j
> > > av
> > > a:1146)
> > > > at
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > > ja
> > > va:615)
> > > > at java.lang.Thread.run(Thread.java:679)
> > > > Caused by: hudson.plugins.git.GitException: Command "git tag -a -f
> > > > -m
> > > Jenkins Build #1468 jenkins-cloudstack-rat-master-1468" returned
> > > status code 128:
> > > > stdout:
> > > > stderr:
> > > > *** Please tell me who you are.
> > > >
> > > > Run
> > > >
> > > >   git config --global user.email "y...@example.com"
> > > >   git config --global user.name "Your Name"
> > > >
> > > > to set your account's default identity.
> > > > Omit --global to set the identity only in this repository.
> > > >
> > > > fatal: empty ident   not allowed
> > > >
> > > > at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:897)
> > > > at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:858)
> > > > at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:868)
> > > > at hudson.plugins.git.GitAPI.tag(GitAPI.java:827)
> > > > ... 12 more

RE: Review Request: use commons-lang StringUtils

[Alex Huang]

Ok...please resubmit.

--Alex

> -Original Message-
> From: Laszlo Hornyak [mailto:nore...@reviews.apache.org] On Behalf Of
> Laszlo Hornyak
> Sent: Tuesday, June 11, 2013 11:48 AM
> To: cloudstack; Laszlo Hornyak; Alex Huang
> Subject: Re: Review Request: use commons-lang StringUtils
> 
> 
> 
> > On June 10, 2013, 11:56 p.m., Alex Huang wrote:
> > > The patch did not apply cleanly.  Please resubmit.  Since you have to
> resubmit anyways, I think why not just change all of the code to use the
> commons.lang version.  I see no point in keeping the method in StringUtils if
> it's replaceable by one in a standard library.
> > >
> 
> Hi Alex,
> 
> I think it makes sense to keep the StringUtils.join(String, Object...) method
> because it allows to use varargs, unlike the one in commons-lang, and the
> code somewhat builds on it.
> 
> 
> - Laszlo
> 
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11767/#review21678
> ---
> 
> 
> On June 9, 2013, 7:47 p.m., Laszlo Hornyak wrote:
> >
> > ---
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/11767/
> > ---
> >
> > (Updated June 9, 2013, 7:47 p.m.)
> >
> >
> > Review request for cloudstack.
> >
> >
> > Description
> > ---
> >
> > commons-lang is already a transitive dependency of the utils project, which
> allows removing some duplicated functionality.
> > This patch replaces StringUtils.join(String, Object...) with it's 
> > commons-lang
> counterpart.
> > It also replaces calls to String join(Iterable, String) in
> cases where an array is already exist and it is only wrapped into a List.
> >
> >
> > Diffs
> > -
> >
> >   server/src/com/cloud/storage/s3/S3ManagerImpl.java 61e5573
> >   services/secondary-
> storage/src/org/apache/cloudstack/storage/resource/NfsSecondaryStorage
> Resource.java e7fa5b2
> >   utils/src/com/cloud/utils/S3Utils.java b7273a1
> >   utils/src/com/cloud/utils/StringUtils.java 14ff4b1
> >   utils/test/com/cloud/utils/StringUtilsTest.java 3c162c7
> >
> > Diff: https://reviews.apache.org/r/11767/diff/
> >
> >
> > Testing
> > ---
> >
> > - Unit test added
> >
> >
> > Thanks,
> >
> > Laszlo Hornyak
> >
> >

Please run with assert on when you're developing...

[Alex Huang]

Hi All,

CloudStack code have many asserts to guarantee code is written correctly for 
the developers.  I recently realized that since we've converted to maven, we no 
longer run with assert on as developers.  It is very important that we do 
because it will find problems for you during load time and run time.

To run with assert on, you can add "-ea" to MAVEN_DEBUG_OPTS.  

I tried this recently and there are many places that are asserting.  Please do 
a run and fix what you can.

Thanks.

--Alex

RE: Hadoop cluster running in cloudstack

[Alex Huang]

I thought about this a bit yesterday after Chiradeep talked to me.

The first fix is definitely allow multiple local storage per host.  That 
requires some work on cloudstack but I don't see it as a big problem.

Then a storage-pool allocator can be written such that it always allocates 
separate local storage pools to vms on the same host.  That should be minimal 
work and can be taken as a side project.

--Alex

> -Original Message-
> From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com]
> Sent: Tuesday, June 11, 2013 1:07 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Hadoop cluster running in cloudstack
> 
> Taking it to dev@ to see if there is any interest.
> 
> 
> It is a good and interesting requirement. I can see hacking 'pre-setup'
> storage with tags to achieve this, but it is going to be a fragile hack.
> I believe GCE also has the concept of some instance types having dedicated
> spindles.
> 
> 
> On 6/6/13 11:14 AM, "David Ortiz"  wrote:
> 
> >Chiradeep,
> > Currently I am working with KVM hypervisor nodes.  The use case of
> >having 4 spindles and assigning one to each node is exactly what I
> >would like to do.  For the moment I have all four spindles configured
> >in a RAID with the cloudstack local storage pointed at it.
> >Shanker,
> >  I had not seen that slideshow yet, so thank you for pointing me
> >to it.  As of now, the hadoop resources I am using are statically
> >allocated between 4 hosts.  As it stands now, I am constrained to those
> >resources without the ability to add any additional storage cluster (or
> >additional storage to my current shared storage appliance), or additional
> nodes.
> >Fortunately, my use cases don't require any kind of reallocation of the
> >hadoop nodes.  It's more clients for the cluster as well as web service
> >nodes that run clients that are being dynamically spun up and down.  I
> >have found that I can get through my jobs alright, they just take a lot
> >of extra time to run since I have the storage acting as a bottleneck
> >right now.
> >Thanks, David Ortiz
> >
> >> From: run...@gmail.com
> >> Subject: Re: Hadoop cluster running in cloudstack
> >> Date: Thu, 6 Jun 2013 10:23:50 -0400
> >> To: us...@cloudstack.apache.org
> >>
> >>
> >> On Jun 6, 2013, at 4:05 AM, Shanker Balan
> >>
> >>wrote:
> >>
> >> > On 05-Jun-2013, at 12:13 AM, David Ortiz  wrote:
> >> >
> >> >> Hello,
> >> >>Has anyone tried running a hadoop cluster in a cloudstack
> >>environment?  I have set one up, but I am finding that I am having
> >>some IO contention between slave nodes on each host since they all
> >>share one local storage pool.  As I understand it, there is not
> >>currently a method for using multiple local storage pools with VMs
> >>through cloudstack.  Has anyone found a workaround for this by any
> chance?
> >> >
> >> >
> >> > Hi David,
> >> >
> >> > Have you seen Seb's
> >>http://www.slideshare.net/sebastiengoasguen/cloudstack-and-bigdata
> >>slides yet?
> >>
> >> As a quick disclaimer, the various configurations I highlight in this
> >>deck are a bit hand wavy and I did not test them. I just made a guess
> >>about how one might want to use the baremetal functionality in
> >>cloudstack. The main distinction being between using a "big data"
> >>store as storage backends of cloudstack and using cloudstack to
> >>provision a bigdata store on-demand.
> >>
> >> -sebastien
> >>
> >> >
> >> > In my experience running Hadoop (100+ nodes) on traditional
> >> > servers,
> >>its going to be really hard to scale up Hadoop workloads using local
> >>storage and HDFS on a cloud.
> >> >
> >> > I ran out of IOPS very quickly. There was enough CPU headroom but
> >>could not add more slots as disk became the bottleneck. Every time
> >>there was a node/disk failure, rebalancing was a nightmare with a 3x
> >>HDFS replication factor.
> >> >
> >> > If I were to run Hadoop on an IaaS cloud, I would do it very
> >> > similar
> >>to Amazon AWS EMR - instances backed by a "Storage As A Service" layer
> >>(S3) for big data instead of HDFS.
> >> >
> >> > The system would work as below:
> >> >
> >> > - Create a dedicated big data storage tier using a distributed
> >>filesystem like Gluster/Ceph/Isilon. Most of the vendors now provide
> >>S3 compat connectors for Hadoop.
> >> >
> >> > http://ceph.com/docs/master/cephfs/hadoop/
> >> > http://gluster.org/community/documentation/index.php/Hadoop
> >> > http://www.emc.com/big-data/scale-out-storage-hadoop.htm
> >> >
> >> > - Hadoop instances are spun up on bare metal or on hypervisors. The
> >>service offerings for "big data" instances could will run on dedicated
> >>hypervisors (via tags) with high bandwidth network connectivity to the
> >>storage service.
> >> >
> >> > - Hadoop instances use Local storage for run time data.
> >> >
> >> > - Hadoop VMs connect to the storage tier via connectors for
> >> > permanent
> >>storage
> >> >
> >> > Benefits:
> >> >
> >> > - Spinning up/down VMs don't cause HDFS rebalanci

RE: Chip and David elected ASF Members

[Alex Huang]

Congratulations Chip and Dave!

--Alex

> -Original Message-
> From: Sebastien Goasguen [mailto:run...@gmail.com]
> Sent: Wednesday, June 12, 2013 1:31 AM
> To: dev@cloudstack.apache.org
> Subject: Chip and David elected ASF Members
> 
> Congratulations to Chip and David for being elected members of the ASF:
> 
> https://blogs.apache.org/foundation/entry/the_apache_software_foundati
> on_welcomes2
> 
> -Sebastien

RE: UI Development

[Alex Huang]

Soheil,

Yes there is.   Look under the plugins folder for all the current plugins.  

CloudStack plugins allow not only code to be inserted to add functionality but 
APIs can be added to facilitate new functionalities.

I do ask everyone who are developing plugins to consider and discuss end user 
facing API changes on the list before making them.  Admin API changes can be 
added fairly quickly.

--Alex

> -Original Message-
> From: Brian Federle [mailto:brian.fede...@citrix.com]
> Sent: Tuesday, June 11, 2013 12:44 PM
> To: 'dev@cloudstack.apache.org'
> Subject: RE: UI Development
> 
> I'm pretty sure there is a modular system in place for the backend, though I
> only do front-end development so I'm not familiar with it. Maybe a server-
> side dev can answer that?
> 
> -Brian
> 
> -Original Message-
> From: Soheil Eizadi [mailto:seiz...@infoblox.com]
> Sent: Tuesday, June 11, 2013 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: RE: UI Development
> 
> Hi Brian,
> Thanks for the detail.
> 
> I have looked at the UI-Plugin page, is there a matching server side plugin
> component to go along with the UI-Plugin to extend the CloudStack
> functionality?
> 
> -Soheil
> 
> 
> From: Brian Federle [brian.fede...@citrix.com]
> Sent: Tuesday, June 11, 2013 11:44 AM
> To: 'dev@cloudstack.apache.org'
> Subject: RE: UI Development
> 
> Hi Soheil,
> 
> For the most part pretty much any IDE/text editor will work for UI
> development, as the code base is pure JS+HTML+CSS. I usually use Firebug
> for debugging purposes.
> 
> For IDEs, I would recommend IntelliJ IDEA, which has the best support for JS,
> including an integrated JS debugger. Not too familiar with Eclipse for web 
> dev,
> though.
> 
> Unfortunately UI documentation is a bit sparse right now; I'm in the process
> of adding more documentation when I can find time. For now, I would
> recommend going through the UI plugin development tutorial and see if it
> will address your needs for the feature:
> https://cwiki.apache.org/CLOUDSTACK/ui-plugin-tutorial.html. It goes
> through setting up a new list view with a set of actions.
> 
> -Brian
> 
> -Original Message-
> From: Soheil Eizadi [mailto:seiz...@infoblox.com]
> Sent: Tuesday, June 11, 2013 11:35 AM
> To: dev@cloudstack.apache.org
> Subject: UI Development
> 
> I did not find any UI development resources on the Wiki. I need to update
> the UI to support a new Device. I wanted to know if there is there any
> recommended tooling for the CloudStack UI. I was planning to use Eclipse
> JavaScript Development Tools (JSDT), but wanted to see if there is any
> recommended setup also what the recommended debugging environment
> would look like?
> Thanks,
> -Soheil

RE: Please run with assert on when you're developing...

[Alex Huang]

Some (ok one) have asked me what is MAVEN_DEBUG_OPTS.  That's my bad for not 
being clear.

Adding "-ea" to MAVEN_OPTS will cause assert to be on for maven and maven's 
plugins as well.  That means during clean, install stages, you might hit some 
weird asserts (according to others experience on the web).

So what I did is added it to the script mvnDebug which utilizes 
MAVEN_DEBUG_OPTS.  When I run jetty:run, I always use mvnDebug which sets up 
the debug port for the jvm and with this change also run in assert mode.  You 
always want to use mvnDebug for your own testing of the server anyways because 
you can easily attach for debugging purposes.

The drawback to doing it this way is that the unit tests that are ran during 
the install stage do not hit asserts.  (I think but haven't confirmed yet.)

--Alex

> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: Tuesday, June 11, 2013 5:18 PM
> To: dev@cloudstack.apache.org
> Subject: Please run with assert on when you're developing...
> 
> Hi All,
> 
> CloudStack code have many asserts to guarantee code is written correctly for
> the developers.  I recently realized that since we've converted to maven, we
> no longer run with assert on as developers.  It is very important that we do
> because it will find problems for you during load time and run time.
> 
> To run with assert on, you can add "-ea" to MAVEN_DEBUG_OPTS.
> 
> I tried this recently and there are many places that are asserting.  Please 
> do a
> run and fix what you can.
> 
> Thanks.
> 
> --Alex

RE: tests for projects

[Alex Huang]

Dan,

It's tough to write unit tests for engine-schema because the requirement is for 
all unit tests to work without a DB so testing DB access without a DB is kinda 
useless.

The tests themselves probably won't be very interesting anyways.  All of the 
classes are actually based off one class, GenericDaoBase, which does most of 
the SQL generation.  

There may have been some tests and I may have left them in the server directory 
when I pulled together the engine-schema directory a couple of months ago. 


--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Wednesday, June 12, 2013 12:52 PM
> To: dev@cloudstack.apache.org
> Subject: tests for projects
> 
> LS,
> 
> When I look at engine-schema I don't see a test source folder. Is this because
> tests for db objects are not done or because this code should go somewhere
> else, or maybe because there are no tests yet and should go and make some?
> 
> Thanks,
> Daan Hoogland

RE: help on db code wanted

[Alex Huang]

Daan,

You can enable trace logging on the category 
"com.cloud.utils.db.Transaction.Statement" in log4j-cloud.xml.  It will log 
every SQL statement sent to the DB.  Look for update to the specific table, 
that will give you an indication what the db layer thinks is being set.  The 
logs grow very quickly with this on though.

BTW,  what does LS mean?  Are you referring to us as some sort of animal farm 
characters (Live Stock)?

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Wednesday, June 12, 2013 3:13 PM
> To: dev
> Subject: help on db code wanted
> 
> LS,
> 
> 
> I am trying to remove a double slash from the field install_path in the table
> template_host_ref. It would seem to me that this can be done by removing
> it in the VMTemplateHostVO, that gets used by the VMTemplateHostDao to
> write to the database. I added a fixPath function that I stole from some other
> class and had it log to see if the expected fixup happened. The log line comes
> by but the database gets filled with the double slashes anyhow. Is the
> database filled from another class then the VO/Dao/DaoImpl combination?
> 
> 
> Many thanks,
> 
> Daan

RE: help on db code wanted

[Alex Huang]

Nice...learn something new every day.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Wednesday, June 12, 2013 3:36 PM
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> Thanks Alex,
> 
> I didn't think of live stock yet, but that is a good backronym as well. The
> original is latin : lectori salutem (I hope I have the inclinations right) 
> meaning
> readers be greeted
> 
> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: donderdag 13 juni 2013 0:31
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> Daan,
> 
> You can enable trace logging on the category
> "com.cloud.utils.db.Transaction.Statement" in log4j-cloud.xml.  It will log
> every SQL statement sent to the DB.  Look for update to the specific table,
> that will give you an indication what the db layer thinks is being set.  The 
> logs
> grow very quickly with this on though.
> 
> BTW,  what does LS mean?  Are you referring to us as some sort of animal
> farm characters (Live Stock)?
> 
> --Alex
> 
> > -Original Message-
> > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > Sent: Wednesday, June 12, 2013 3:13 PM
> > To: dev
> > Subject: help on db code wanted
> >
> > LS,
> >
> >
> > I am trying to remove a double slash from the field install_path in
> > the table template_host_ref. It would seem to me that this can be done
> > by removing it in the VMTemplateHostVO, that gets used by the
> > VMTemplateHostDao to write to the database. I added a fixPath function
> > that I stole from some other class and had it log to see if the
> > expected fixup happened. The log line comes by but the database gets
> > filled with the double slashes anyhow. Is the database filled from another
> class then the VO/Dao/DaoImpl combination?
> >
> >
> > Many thanks,
> >
> > Daan

RE: [MERGE] disk_io_throttling to MASTER

[Alex Huang]

Hi All,

I didn't follow this thread as much so maybe it's already been discussed.  If 
so, let me know.

I don't see why these limits should be used as a way to do allocation.  CS 
already supports storage tiering through tags.  Why not reuse that concept?

A use case would be.
An admin wants to use SolidFire storage for example and enable the rate 
limiting feature.

SolidFire's plugin add tags to the storage pools it added to cloudstack.  
When the admin creates disk offering, they can either add the tag to the disk 
offering themselves or call an api command on the solid fire plugin to let 
solid fire plugin add that tag to the disk offering and set the values for the 
rate limits.
CloudStack's allocator will use tags to match the storage pools.  No changes 
there so big plus.
Rate limits are carried to the storage provider and hypervisor by CS.  You can 
add integration in the resources to implement the rate limits.

There are several benefits to doing this.

- Utilizes something that already works.
- Less code to maintain (don't need to modify all the allocators and planners)

I think even the values themselves should just be name-value pairs rather than 
fields in the offering.  CS should be responsible for transporting these fields 
to the interested parties (plugins, and resources) but it itself doesn't 
understand them.

--Alex

> -Original Message-
> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> Sent: Wednesday, June 12, 2013 3:31 PM
> To: John Burwell
> Cc: dev@cloudstack.apache.org; Wei Zhou
> Subject: Re: [MERGE] disk_io_throttling to MASTER
> 
> We'd also have to recognize that if the Min value is filled in and Storage QoS
> is not available that Hypervisor QoS (rate limiting here) cannot satisfy that
> constraint.
> 
> 
> On Wed, Jun 12, 2013 at 4:26 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
> 
> > Yeah, this is interesting.
> >
> > We'll have to wait and see what Wei's thoughts are on this.
> >
> >
> > On Wed, Jun 12, 2013 at 4:17 PM, John Burwell 
> wrote:
> >
> >> Mike,
> >>
> >> Yes.  To your point, the appropriate logic would be to check the
> >> Volume allocated by the StorageAllocator.  If it doesn't support a
> >> QoS, then the VM allocator would attempt to fulfill the QoS through the
> hypervisor.
> >>  Another question would be -- what would be in the behavior in the
> >> event that no resources are available to fulfill the QoS?  We could
> >> outright fail or proceed with some kind of warning -- sounds like a
> >> good place for a global setting to configure the policy.
> >>
> >> The other question we haven't answered are usage records.  By pushing
> >> the QoS concept out to allocation and making it a more general
> >> concept, it could make usage record capture more straightforward.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Jun 12, 2013, at 6:11 PM, Mike Tutkowski
> >> 
> >> wrote:
> >>
> >> I see, John.
> >>
> >> That is an interesting idea.
> >>
> >> We'd also have to change the storage allocator(s) to favor
> >> QoS-supported storage if those fields are filled in.
> >>
> >>
> >> On Wed, Jun 12, 2013 at 4:09 PM, John Burwell 
> wrote:
> >>
> >>> Mike,
> >>>
> >>> My thought is that we present the min/max IOPS fields for read/write
> >>> operations for all compute and disk offerings.  When the VM is
> >>> allocated, we determine the best way to fulfill that QoS.  It sounds
> >>> like storage level guarantees would always be preferred.  If no
> >>> storage is available to guarantee the QoS, the allocator falls back
> >>> to using hypervisor.  This approach only works if summing the
> >>> discrete read/write min/max values to get to min/max total IOPS would
> be acceptable.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>>
> >>> On Jun 12, 2013, at 6:03 PM, Mike Tutkowski <
> >>> mike.tutkow...@solidfire.com> wrote:
> >>>
> >>> I hate to say it, but I believe Storage QoS with a Min and Max will
> >>> always be optimal over hypervisor rate limiting.
> >>>
> >>> The only time you'd want to use hypervisor rate limiting is if
> >>> storage QoS is not available.
> >>>
> >>> We currently have no way to know what storage the root or data disk
> >>> will be deployed to, so we have to present the fields in all situation.
> >>>
> >>> This is because of - in my opinion - the somewhat flawed way
> >>> CloudStack handles storage tags. You are free to enter in any text
> >>> there and we don't know what it maps to when the Compute or Disk
> Offering dialog is up.
> >>>
> >>>
> >>> On Wed, Jun 12, 2013 at 3:58 PM, John Burwell
> wrote:
> >>>
>  Mike,
> 
>  Please see my comments/questions in-line below.
> 
>  Thanks,
>  -John
> 
>  On Jun 12, 2013, at 5:37 PM, Mike Tutkowski <
>  mike.tutkow...@solidfire.com> wrote:
> 
>  Hi Wei,
> 
>  So, not entirely sure I follow.
> 
>  Will what I wrote earlier work? Here is a copy of what I wrote:
> 
>  Let's just called these radio buttons 1) Hypervisor

RE: help on db code wanted

[Alex Huang]

When you're debugging, you should modify the log4j-cloud.xml that jetty is 
reading.  I can't remember where it is by default.  I have to find this file 
everytime but it's not any of the files you find in the source as it is copied 
to some jetty deployment directory.  Maybe someone knows where that file is.  
If not, if you're on linux, you can always find out by doing a lsof on the java 
process and see which log4j-cloud.xml it is using.  I assume you can do 
something similar on the mac.  On windows, there's a utility called handle from 
system internals package that will tell you.

Doing this means you'll lose your setting everytime you do a  mvn clean 
install.  So editing one of the files in the source tree will give you a more 
permanent solution.  However, you have to be careful not to check it in.  

BTW, CloudStack gets updated live on logging configs so there's no need to 
restart cloudstack after you change the configuration.  Just edit the file and 
save and within a minute or so, cloudstack will use the new logging 
configuration.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Wednesday, June 12, 2013 3:48 PM
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> Glad to be teaching for a change;) quite common greeting in dutch if you
> don't know how to start.
> 
> About the log4j-cloud file. There is a lot of them. I suppose I chose the one
> from secondary-storage and they all get picked up?  I also see some .in files.
> No need to worry about those?
> 
> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: donderdag 13 juni 2013 0:44
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> Nice...learn something new every day.
> 
> --Alex
> 
> > -Original Message-
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: Wednesday, June 12, 2013 3:36 PM
> > To: dev@cloudstack.apache.org
> > Subject: RE: help on db code wanted
> >
> > Thanks Alex,
> >
> > I didn't think of live stock yet, but that is a good backronym as
> > well. The original is latin : lectori salutem (I hope I have the
> > inclinations right) meaning readers be greeted
> >
> > -Original Message-
> > From: Alex Huang [mailto:alex.hu...@citrix.com]
> > Sent: donderdag 13 juni 2013 0:31
> > To: dev@cloudstack.apache.org
> > Subject: RE: help on db code wanted
> >
> > Daan,
> >
> > You can enable trace logging on the category
> > "com.cloud.utils.db.Transaction.Statement" in log4j-cloud.xml.  It
> > will log every SQL statement sent to the DB.  Look for update to the
> > specific table, that will give you an indication what the db layer
> > thinks is being set.  The logs grow very quickly with this on though.
> >
> > BTW,  what does LS mean?  Are you referring to us as some sort of
> > animal farm characters (Live Stock)?
> >
> > --Alex
> >
> > > -Original Message-
> > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > > Sent: Wednesday, June 12, 2013 3:13 PM
> > > To: dev
> > > Subject: help on db code wanted
> > >
> > > LS,
> > >
> > >
> > > I am trying to remove a double slash from the field install_path in
> > > the table template_host_ref. It would seem to me that this can be
> > > done by removing it in the VMTemplateHostVO, that gets used by the
> > > VMTemplateHostDao to write to the database. I added a fixPath
> > > function that I stole from some other class and had it log to see if
> > > the expected fixup happened. The log line comes by but the database
> > > gets filled with the double slashes anyhow. Is the database filled
> > > from another
> > class then the VO/Dao/DaoImpl combination?
> > >
> > >
> > > Many thanks,
> > >
> > > Daan

RE: help on db code wanted

[Alex Huang]

Dan,

Looks like the problem is with the DB layer.  It doesn't actually retrieve the 
value stored in the vo object but instead records the value passed to the vo 
object when the set method is called.  So it assumes the value being passed is 
the value is stored.  For now, best to make the change outside of the VO object.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: Wednesday, June 12, 2013 4:36 PM
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> Dear readers,
> 
> I have some logging stating that I change the value in the VO, and then a db
> update using the double quotes anyway, as if the VO is not the actual object
> used for the update. Note the two 2F2F (double 2F's) in the
> template_host_ref.install_path in the last log line.
> 
> 2013-06-13 01:19:54,411 INFO  [cloud.storage.VMTemplateHostVO]
> (AgentManager-Handler-7:null) new install_path for VMTemplateHost:
> template/tmpl//2/208//59a1a94e-d69e-37e2-adb4-2e14472c3890.vhd
> changed into template/tmpl/2/208/59a1a94e-d69e-37e2-adb4-
> 2e14472c3890.vhd
> 
> 2013-06-13 01:19:54,427 TRACE [db.Transaction.Statement] (AgentManager-
> Handler-7:null) Preparing: UPDATE template_host_ref SET
> template_host_ref.local_path=?, template_host_ref.job_id=?,
> template_host_ref.physical_size=?, template_host_ref.download_pct=?,
> template_host_ref.install_path=?, template_host_ref.last_updated=?,
> template_host_ref.download_state=?, template_host_ref.error_str=?,
> template_host_ref.size=? WHERE template_host_ref.id = ?
> 
> 2013-06-13 01:19:54,427 TRACE [db.Transaction.Statement] (AgentManager-
> Handler-7:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@142e688:
> UPDATE template_host_ref SET
> template_host_ref.local_path=x'2F6D6E742F53656353746F726167652F356139
> 30333963662D363337612D333737302D623634342D616264376163353565663237
> 2F74656D706C6174652F746D706C2F322F3230382F646E6C643835313733323938
> 373030343437323537746D705F',
> template_host_ref.job_id=x'36303563303331342D666532332D343231662D61
> 3663322D313662396532626535303664', template_host_ref.physical_size=0,
> template_host_ref.download_pct=100,
> template_host_ref.install_path=x'74656D706C6174652F746D706C2F2F322F32
> 30382F2F35396131613934652D643639652D333765322D616462342D3265313434
> 37326839302E766864', template_host_ref.last_updated='2013-06-12
> 23:19:54', template_host_ref.download_state='DOWNLOAD_IN_PROGRESS',
> template_host_ref.error_str=x'446F776E6C6F616420737563636573732C20737
> 4617274696E6720696E7374616C6C20', template_host_ref.size=0 WHERE
> template_host_ref.id = 16
> 
> Who has got the magic word for me?
> 
> Regards,
> 
> -Original Message-
> From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> Sent: donderdag 13 juni 2013 1:08
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> Ok,
> 
> 2013-06-13 00:54:21,841 INFO  [cloud.utils.LogUtils] (main:null) log4j
> configuration found at
> C:\Users\dhoogland\cloudstack\cloudstack\client\target\cloud-client-ui-
> 4.2.0-SNAPSHOT\WEB-INF\classes\log4j-cloud.xml
> 
> This is from the vmops.log. guess this must be the one.
> 
> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: donderdag 13 juni 2013 1:04
> To: dev@cloudstack.apache.org
> Subject: RE: help on db code wanted
> 
> When you're debugging, you should modify the log4j-cloud.xml that jetty is
> reading.  I can't remember where it is by default.  I have to find this file
> everytime but it's not any of the files you find in the source as it is 
> copied to
> some jetty deployment directory.  Maybe someone knows where that file is.
> If not, if you're on linux, you can always find out by doing a lsof on the 
> java
> process and see which log4j-cloud.xml it is using.  I assume you can do
> something similar on the mac.  On windows, there's a utility called handle
> from system internals package that will tell you.
> 
> Doing this means you'll lose your setting everytime you do a  mvn clean 
> install.
> So editing one of the files in the source tree will give you a more permanent
> solution.  However, you have to be careful not to check it in.
> 
> BTW, CloudStack gets updated live on logging configs so there's no need to
> restart cloudstack after you change the configuration.  Just edit the file and
> save and within a minute or so, cloudstack will use the new logging
> configuration.
> 
> --Alex
> 
> > -Original Message-
> > From: Daan Hoogland [mailto:dhoogl...@schubergphilis.com]
> > Sent: Wednesday, June 12

FW: [ANNOUNCE] New committer: Mike Tutkowski

[Alex Huang]

The Project Management Committee (PMC) for Apache CloudStack has asked Mike 
Tutkowski to become a committer and we are pleased to announce that they have 
accepted.

Being a committer allows many contributors to contribute more autonomously. For 
developers, it makes it easier to submit changes and eliminates the need to 
have contributions reviewed via the patch submission process. Whether 
contributions are development-related or otherwise, it is a recognition of a 
contributor's participation in the project and commitment to the project and 
the Apache Way.

Please join me in congratulating Mike!

--Alex
on behalf of the CloudStack PMC

RE: IRC meeting today

[Alex Huang]

Sorry.  I had to drop out due to day job as well.

--Alex

> -Original Message-
> From: John Kinsella [mailto:j...@stratosec.co]
> Sent: Wednesday, June 19, 2013 10:19 AM
> To: 
> Subject: Re: IRC meeting today
> 
> We had about 4 of us in the meeting, so we just dropped it until next week.
> 
> On Jun 19, 2013, at 9:15 AM, John Kinsella 
>  wrote:
> 
> > I'm happy to run it
> >
> > On Jun 19, 2013, at 8:45 AM, Chip Childers 
> > wrote:
> >
> >> Hi all,
> >>
> >> Joe asked me to facilitate the meeting today, however I have a
> >> $dayjob responsibility that just came up.  I'm not going to be able
> >> to do anything other than lurk in the channel.
> >>
> >> I'm going to suggest that we cancel the meeting for today, unless
> >> someone else wants to run it.
> >>
> >> -chip
> >

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

Hi All,

Kelven had an emergency so I'm submitting the changes on vmsync for him.  The 
patch are on 
https://reviews.apache.org/r/12126.

Hugo took a look and already had some questions on why so many files were 
changed and added/deleted,  So I like to explain a bit in this email.

As part of the vmsync change, we try to move the files that were relevant to vm 
orchestration into the right places so that others can properly view the 
different jars and understand the relationship between the jars.  This process 
is ongoing and will continue in other changes.

- Work related to jobs management have been put under cloud-framework-jobs as 
handling jobs is not really part of our server but is a library.  By doing it 
this way, changes in it can be properly reviewed and there's a good separation 
from things that utilizes jobs and jobs itself.
- VirtualMachine orchestration has been moved from cloud-server to 
cloud-engine-orchestration.  Cloud-engine-orchestration then only depends on 
cloud-engine-schema and cloud-api.  This creates a clear compilation boundary 
between the self-service server implementation and orchestration 
implementation. 

As part of these changes, then we surfaced many problems because we setup the 
proper compilation boundaries and fixed all of these problems.  For example, 
HostAllocator interface is something people can write plugins for but it was 
buried inside cloud-server package.  We've moved a majority of these changes to 
cloud-api.  There's quite a bit of file movements based on this change.  For 
vmsync changes itself, the changes are centered around 
VirtualMachineManagerImpl.java so you can review that file instead.

Thanks.

--Alex


> -Original Message-
> From: Kelven Yang [mailto:kelven.y...@citrix.com]
> Sent: Tuesday, June 18, 2013 3:38 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [MERGE] Merge VMSync improvement branch into master
> 
> 
> 
> On 6/17/13 7:43 PM, "Mice Xia"  wrote:
> 
> >Kelven,
> >
> >After the refactoring, will CS still restart HA enabled VM when it is
> >power off externally (e.g. using xencenter) or internally (user
> >initiated shutdown or crash)?
> 
> 
> If HA functionality is provided by external manager (i.e., vCenter), CS won't
> try to restart it, but if HA is provided by CloudStack, we will restore the 
> legacy
> logic. The hook up with old HA manager (between
> HighAvailabilityManager/VirtualMachineManager) is in the wrap-up process
> 
> 
> >
> >Is seems the method HighAvailabilityManagerImpl .scheduleRestart() is
> >not called when VM's actual state is stopped while expected state is
> running.
> >
> >
> >Regards
> >Mice
> >
> >-Original Message-
> >From: Kelven Yang [mailto:kelven.y...@citrix.com]
> >Sent: Tuesday, June 18, 2013 5:21 AM
> >To: dev@cloudstack.apache.org
> >Subject: Re: [MERGE] Merge VMSync improvement branch into master
> >
> >Haven't created a patch yet, will do it soon after some last wrap-ups.
> >
> >Kelven
> >
> >On 6/17/13 12:03 PM, "John Burwell"  wrote:
> >
> >>Kelven,
> >>
> >>Did this patch get pushed to Review Board?  If so, what is the URL?
> >>
> >>Thanks.
> >>-John
> >>
> >>On Jun 17, 2013, at 1:40 PM, Kelven Yang  wrote:
> >>
> >>> Low level classes were tested in unit tests(MessageBus, Job
> >>>framework, Job  dispatchers etc), interface layer changes are guarded
> >>>through matching the  old semantics, but changes are tested manually,
> >>>we are planning to get  this part of testing through BVT system after
> >>>we have re-based the latest  master.
> >>>
> >>> Kelven
> >>>
> >>> On 6/17/13 10:01 AM, "Chip Childers" 
> wrote:
> >>>
>  On Mon, Jun 17, 2013 at 04:59:00PM +, Kelven Yang wrote:
> > I'd like to kick off the official merge process. We will start the
> >merge  process after the branch has passed necessary tests
> >
> > Kelven
> 
>  Can you share what testing is being run against the branch?
> >>>
> >>
> >

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

Dave,

Chip has asked this before and we also stated specifically that we won't merge 
it in unless we see equivalent pass rate on the BVT as master.  We're doing 
that right now.

--Alex

> -Original Message-
> From: David Nalley [mailto:da...@gnsa.us]
> Sent: Thursday, June 27, 2013 4:34 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [MERGE] Merge VMSync improvement branch into master
> 
> On Thu, Jun 27, 2013 at 5:51 PM, Hugo Trippaers  wrote:
> >
> > I think Ilya offers is great, my current stance is also to see how we can 
> > bring
> this forward.
> >
> > I've had the opportunity to meet with several people at the Citrix office in
> Santa Clara, i'm actually working from their office at this moment. I think 
> it's
> also the responsibility of someone who put in a -1 to work with the original
> committer to get the situation resolved. So i'll invest the time to help with
> the review as well.
> >
> > It would be great if Alex or Kelven could take the time to explain how this
> feature has been tested. That would give the community some insight as
> well.
> >
> > My main technical problem with this merge is that stuff is moving all over
> the place without having even the slightest idea why. Now having discussed
> this with Alex in person i get the general idea of this merge, so can actually
> try to review it.
> >
> > I think that John have nicely explained what we could do to prevent
> situations like this in advance. I fully understand that big features or 
> rewrites
> don't happen overnight and might show up near the end of the release cycle.
> With the time based release cycle it's always a risk that some feature might
> not make it in on time. Getting more people involved and chunking the
> commits into master will greatly speed up the reviewing process.
> >
> > I'll get back to this after spending some time on reviewing the actual 
> > patch.
> In fact i would like to ask more people to have a look at this patch and reply
> to this thread with comments or remarks.
> >
> > Cheers,
> >
> > Hugo
> >
> 
> So the problem in my mind, is that we don't have a way of verifying that
> master isn't broken, and won't be broken by any given merge. I look at even
> the minimal level of automated testing that I see today, and ~20% of
> integration tests are failing[1]. The regression set of tests (which isn't 
> running
> as often) is seeing 75% of tests failing[2]. Heaping on more change when we
> are demonstrably already failing in many places is not behaving responsibly
> IMO.
> The question I'd pose is this - running the various automated tests is pretty
> cheap - whats the output of that compared to the current test output on
> master? Better or worse? If it hasn't been done, why not?
> I desperately want these features, but not necessarily at the cost of further
> destabilizing what we have now in master - we can't continue accruing
> technical debt.
> 
> --David
> 
> [1] http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-smoke-
> matrix/lastCompletedBuild/testReport/
> [2] http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-regression-
> matrix/28/testReport/

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

gt;>>>> develop context late.  To be clear, identifying reviewers early
> >>>>> does not preclude a new reviewer from becoming involved later, but
> >>>>> the reviews will greatly reduce the probability of a late review
> >>>>> finding significant issues.  Additionally, the early reviewers can
> >>>>> help the late
> >> reviewers come up to speed -- reducing the burden on the contributor.
> >>>>>
> >>>>> Generally, my recommendation is to submit patches early and often.
> >>>>> Ultimately, contributors determine how they wish to develop and
> >>>>> deliver their contributions.  These are only my recommendations as
> >>>>> a reviewer to increase the probability that a feature will make a
> >>>>> particular
> >> release train.
> >>>>> Finally, it appears that reviewers are growing less and less
> >>>>> tolerant of large patches that appear just before freeze dates,
> >>>>> and those these patches face a much higher risk of being
> >>>>> immediately receiving a -1 for the reasons stated above.
> >>>>>
> >>>>> Thanks,
> >>>>> -John
> >>>>>
> >>>>> On Jun 26, 2013, at 8:45 PM, Hugo Trippaers 
> wrote:
> >>>>>
> >>>>>> Hey Alex, Kelven,
> >>>>>>
> >>>>>> I've been looking though the code and thanks for the very
> >>>>>> insightfull
> >>>>> conversation and follow-up email. With merges of this magnitude
> >>>>> it's essential to help people understand what is going on.
> >>>>>>
> >>>>>> Purely technical this is a merge i'm really pleased with. It will
> >>>>>> for sure
> >>>>> improve our handling of virtual machines.
> >>>>>>
> >>>>>> However the timing of the merge request is not ideal. We are very
> >>>>>> close to
> >>>>> the end of the already extended feature freeze. We have a
> >>>>> consensus within the dev community to do large architectural
> >>>>> changes in the beginning of the cycle and not at the very end.
> >>>>> This not only means a lot of extra testing effort and other
> >>>>> developers will have to get used to the changes introduced right
> >>>>> at the moment when everybody should be focussed on bug fixing,
> documentation and stability.
> >>>>> Without wanting to rip open old wounds, i can imagine we all want
> >>>>> to
> >> avoid a javelin incident.
> >>>>>>
> >>>>>> I respect the work going into this and the effort it will take to
> >>>>>> keep this up
> >>>>> to date with the current speed of master, but still i'm going to
> >>>>> veto this with a
> >>>>> -1 for inclusion before we cut of the 4.2 branch.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Hugo
> >>>>>>
> >>>>>> On Jun 26, 2013, at 5:32 PM, Alex Huang 
> >> wrote:
> >>>>>>
> >>>>>>> Hi All,
> >>>>>>>
> >>>>>>> Kelven had an emergency so I'm submitting the changes on vmsync
> >>>>>>> for him.  The patch are on https://reviews.apache.org/r/12126.
> >>>>>>>
> >>>>>>> Hugo took a look and already had some questions on why so many
> >>>>>>> files
> >>>>> were changed and added/deleted,  So I like to explain a bit in this
> email.
> >>>>>>>
> >>>>>>> As part of the vmsync change, we try to move the files that were
> >>>>>>> relevant
> >>>>> to vm orchestration into the right places so that others can
> >>>>> properly view the different jars and understand the relationship
> >>>>> between the jars.  This process is ongoing and will continue in
> >>>>> other
> >> changes.
> >>>>>>>
> >>>>>>> - Work related to jobs management have been put under cloud-
> >>>>> framework-jobs as handling jobs is not really part of our server
> >>&g

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

+1 to frequent patches and reviews.

I wish we started with chunk features and narrow patch scope.   There are some 
circumstances (some unique to this situation) that led to this but I won't use 
that as an excuse.  Thanks to all who are offering to lend a hand in reviewing 
and testing.  It is much appreciated.

--Alex

> -Original Message-
> From: John Burwell [mailto:jburw...@basho.com]
> Sent: Thursday, June 27, 2013 11:33 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [MERGE] Merge VMSync improvement branch into master
> 
> Hugo,
> 
> I completely agree with this stance, and will add a -1 as well.  It has been a
> significant challenge this cycle to complete high quality reviews due to the
> large patch size and short time turnaround time.  Going forward, I am going
> to be more likely to follow Hugo's example, and -1 patches for which there is
> not adequate review time.  I think the following techniques will help
> streamline the review process:
> 
> 1.  Narrow Patch Scope: Ensure that the patch is confined to a single
> enhancement or defect fix.  If you doing refactoring as part of a feature,
> submit the refactoring work a separate commit.  As a further suggestion, if
> you are performing multiple refactorings (e.g. package reorganization and
> utility method extraction), split each refactoring into separate patches.
> 2. Chunk Features: For each large feature being developed, logically chunk
> the functionality and submit each chunk as a separate patch.  I recommend
> erring on the side of chunks being too small.  Let the reviewer determine if
> the patch is not large enough to be representative  of the feature.  The worst
> that will happen is that a reviewer will provide feedback on the work
> completed and simply ask for more work to be done before it can be merged
> to master.  For features while chunks can't be merged all the way to master,
> intermediate patches can be submitted to Review Board for review
> throughout the cycle -- allowing large merges to actually be the accumulation
> of several smaller reviews.
> 3. Identify Reviewers Early: For big features, solicit the reviewers whiling 
> to
> review a feature from FS through merge from the mailing list.  Reviews will
> go much more quickly if reviewers have been involved from the beginning
> because they will not help identify issues before coding starts, but also will
> not have to develop context late.  To be clear, identifying reviewers early
> does not preclude a new reviewer from becoming involved later, but the
> reviews will greatly reduce the probability of a late review finding 
> significant
> issues.  Additionally, the early reviewers can help the late reviewers come up
> to speed -- reducing the burden on the contributor.
> 
> Generally, my recommendation is to submit patches early and often.
> Ultimately, contributors determine how they wish to develop and deliver
> their contributions.  These are only my recommendations as a reviewer to
> increase the probability that a feature will make a particular release train.
> Finally, it appears that reviewers are growing less and less tolerant of large
> patches that appear just before freeze dates, and those these patches face a
> much higher risk of being immediately receiving a -1 for the reasons stated
> above.
> 
> Thanks,
> -John
> 
> On Jun 26, 2013, at 8:45 PM, Hugo Trippaers  wrote:
> 
> > Hey Alex, Kelven,
> >
> > I've been looking though the code and thanks for the very insightfull
> conversation and follow-up email. With merges of this magnitude it's
> essential to help people understand what is going on.
> >
> > Purely technical this is a merge i'm really pleased with. It will for sure
> improve our handling of virtual machines.
> >
> > However the timing of the merge request is not ideal. We are very close to
> the end of the already extended feature freeze. We have a consensus within
> the dev community to do large architectural changes in the beginning of the
> cycle and not at the very end. This not only means a lot of extra testing 
> effort
> and other developers will have to get used to the changes introduced right at
> the moment when everybody should be focussed on bug fixing,
> documentation and stability. Without wanting to rip open old wounds, i can
> imagine we all want to avoid a javelin incident.
> >
> > I respect the work going into this and the effort it will take to keep this 
> > up
> to date with the current speed of master, but still i'm going to veto this 
> with a
> -1 for inclusion before we cut of the 4.2 branch.
> >
> > Cheers,
> >
> > Hugo
> >
> > On Jun 26, 2013, at 5:32 PM, Alex Huang  w

[DISCUSS] If BVT breaks, revert the commits...

[Alex Huang]

After Dave's complain in the vmsync [MERGE] thread about BVT in horrible shape 
on master, I went around to figure out what exactly happened.  The best I can 
figure is that after a certain merge (I will leave out which merge as that's 
not important), BVT no longer runs automatically.  It was promised to be fixed 
and there are people who are actively fixing it but it's been in this way for 
about two weeks.  People running BVTs are working around the problem but it's 
not automated anymore and so it's no longer running on master.  I understand 
people are nice and tried to be accommodating to other people by working around 
the problem but sometimes we just have to be an arse.  So let me be that arse...

New Rule
If BVT or automated regression tests break on master or any release branch, we 
revert all commits that broke it.  It doesn't matter if they promise to fix it 
within the next hour.  If it's broken, the release manager will revert the 
commits and developers must resubmit.  It sounds mean but it's the only way 
this problem can be fixed.  

To avoid having a bunch of reverts and resubmits, the developers should be able 
to request that BVT run on their branch and don't merge until BVT on their 
branch is at 100%.  We will work on figuring out how to do that.

Comments?

--Alex

RE: [DISCUSS] If BVT breaks, revert the commits...

[Alex Huang]

> This only thing i'm a bit afraid of is the technical side of reverting a 
> merge. It's
> one of those this that you need to be careful with in git.

I think this means all merges must come in as one big squashed patch.  If they 
come in as multiple commits or rebase, then it gets reverted automatically.

--Alex

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

Given the current state of BVT, I don't think we can reliably merge this into 
master.  It will have to wait for 4.3.  I apologize to those who really want to 
see this feature in.  I myself have slaved over this for some weeks, including 
missing the collab conference, but I just cannot conscientiously push it in 
under the current circumstances.

--Alex

> -Original Message-
> From: Sudha Ponnaganti [mailto:sudha.ponnaga...@citrix.com]
> Sent: Friday, June 28, 2013 7:11 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [MERGE] Merge VMSync improvement branch into master
> 
> Ideally I would like to see all validation to be done on feature branch - BVTs
> and also manual validation of the P1 test cases from VM Sync test plan just
> like object store validation - sadhu has signed up for it along with Ilya. 
> This
> would help us to identify feature specific issues.
> 
> We are running BVTs right now which have high failure rate. Focusing on this
> part right now to reach parity with Master branch.
> 
> 
> -Original Message-
> From: Trippie [mailto:trip...@gmail.com] On Behalf Of Hugo Trippaers
> Sent: Thursday, June 27, 2013 5:32 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [MERGE] Merge VMSync improvement branch into master
> 
> I agree with John that a change like this is very hard to test in an automated
> fashion. Still i have been looking at the numbers for the code coverage with
> cobertura. I was a bit disappointed to find that we have not made any
> progress with this merge with regards to unit tests and total code coverage.
> We do not seem to be in a worse shape than before.
> 
> @Sudha, it would be nice if you could add your view on this patch from the
> QA perspective? How would this patch affect your planning for example?
> 
> Cheers,
> 
> Hugo
> 
> On Jun 27, 2013, at 5:12 PM, John Burwell  wrote:
> 
> > @David The types of concurrency changes introduced in this patch are
> > extremely difficult to completely test in an automated fashion.
> > Therefore, code review for correctness is critical to ensure quality.
> > To be clear, I am not questioning the value of automated testing.  I
> > am just noting that it's next to impossible to achieve full coverage,
> > and code review is an critical supplement.
> >
> > @Ilya I plan to review this patch, but I will be able to start until
> > next week.  I am also still reviewing object_store (a separate
> > procedural issue for another thread), and need to complete solidfire.
> > This backlog is precisely why need to be reviewing iteratively
> > throughout the dev cycle.
> >
> > Thanks,
> > -John
> >
> > On Jun 27, 2013, at 7:35 PM, David Nalley  wrote:
> >
> >> On Thu, Jun 27, 2013 at 5:51 PM, Hugo Trippaers 
> wrote:
> >>>
> >>> I think Ilya offers is great, my current stance is also to see how we can
> bring this forward.
> >>>
> >>> I've had the opportunity to meet with several people at the Citrix office
> in Santa Clara, i'm actually working from their office at this moment. I think
> it's also the responsibility of someone who put in a -1 to work with the
> original committer to get the situation resolved. So i'll invest the time to 
> help
> with the review as well.
> >>>
> >>> It would be great if Alex or Kelven could take the time to explain how
> this feature has been tested. That would give the community some insight as
> well.
> >>>
> >>> My main technical problem with this merge is that stuff is moving all over
> the place without having even the slightest idea why. Now having discussed
> this with Alex in person i get the general idea of this merge, so can actually
> try to review it.
> >>>
> >>> I think that John have nicely explained what we could do to prevent
> situations like this in advance. I fully understand that big features or 
> rewrites
> don't happen overnight and might show up near the end of the release cycle.
> With the time based release cycle it's always a risk that some feature might
> not make it in on time. Getting more people involved and chunking the
> commits into master will greatly speed up the reviewing process.
> >>>
> >>> I'll get back to this after spending some time on reviewing the actual
> patch. In fact i would like to ask more people to have a look at this patch 
> and
> reply to this thread with comments or remarks.
> >>>
> >>> Cheers,
> >>>
> >>> Hugo
> >>
> >> So the problem in my mind, is that we don't have a way of verifying
> >> that master isn't broken, and won't be broken by any given merge. I
> >> look at even the minimal level of automated testing that I see today,
> >> and ~20% of integration tests are failing[1]. The regression set of
> >> tests (which isn't running as often) is seeing 75% of tests
> >> failing[2]. Heaping on more change when we are demonstrably already
> >> failing in many places is not behaving responsibly IMO.
> >> The question I'd pose is this - running the various automated tests
> >> is pretty cheap - whats the output of that compared to the curren

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

Ilya,

Thanks!  The current problem on this branch is we're trying to distinguish 
what's actually broken on master and what's broken by introducing this change.  
At this point, the branch is basically frozen and only applying bug fixes once 
BVT found problems.

--Alex

> -Original Message-
> From: Musayev, Ilya [mailto:imusa...@webmd.net]
> Sent: Monday, July 1, 2013 10:16 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [MERGE] Merge VMSync improvement branch into master
> 
> Alex,
> 
> I completely understand. Please keep us in the loop on the progress. If its
> functional to some extent - i.e. at least build succeeds without errors and
> major functionality is there, I can merge the code into CloudSand CS distro -
> to test it out and share it with whoever wants to try it.
> 
> Thanks
> ilya
> 
> > -Original Message-
> > From: Alex Huang [mailto:alex.hu...@citrix.com]
> > Sent: Friday, June 28, 2013 9:16 PM
> > To: dev@cloudstack.apache.org
> > Subject: RE: [MERGE] Merge VMSync improvement branch into master
> >
> > Given the current state of BVT, I don't think we can reliably merge
> > this into master.  It will have to wait for 4.3.  I apologize to those
> > who really want to see this feature in.  I myself have slaved over
> > this for some weeks, including missing the collab conference, but I
> > just cannot conscientiously push it in under the current circumstances.
> >
> > --Alex
> >
> > > -Original Message-
> > > From: Sudha Ponnaganti [mailto:sudha.ponnaga...@citrix.com]
> > > Sent: Friday, June 28, 2013 7:11 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: [MERGE] Merge VMSync improvement branch into master
> > >
> > > Ideally I would like to see all validation to be done on feature
> > > branch - BVTs and also manual validation of the P1 test cases from
> > > VM Sync test plan just like object store validation - sadhu has
> > > signed up for it along with Ilya. This would help us to identify feature
> specific issues.
> > >
> > > We are running BVTs right now which have high failure rate. Focusing
> > > on this part right now to reach parity with Master branch.
> > >
> > >
> > > -Original Message-
> > > From: Trippie [mailto:trip...@gmail.com] On Behalf Of Hugo Trippaers
> > > Sent: Thursday, June 27, 2013 5:32 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [MERGE] Merge VMSync improvement branch into master
> > >
> > > I agree with John that a change like this is very hard to test in an
> > > automated fashion. Still i have been looking at the numbers for the
> > > code coverage with cobertura. I was a bit disappointed to find that
> > > we have not made any progress with this merge with regards to unit
> > > tests and
> > total code coverage.
> > > We do not seem to be in a worse shape than before.
> > >
> > > @Sudha, it would be nice if you could add your view on this patch
> > > from the QA perspective? How would this patch affect your planning
> > > for
> > example?
> > >
> > > Cheers,
> > >
> > > Hugo
> > >
> > > On Jun 27, 2013, at 5:12 PM, John Burwell  wrote:
> > >
> > > > @David The types of concurrency changes introduced in this patch
> > > > are extremely difficult to completely test in an automated fashion.
> > > > Therefore, code review for correctness is critical to ensure quality.
> > > > To be clear, I am not questioning the value of automated testing.
> > > > I am just noting that it's next to impossible to achieve full
> > > > coverage, and code review is an critical supplement.
> > > >
> > > > @Ilya I plan to review this patch, but I will be able to start
> > > > until next week.  I am also still reviewing object_store (a
> > > > separate procedural issue for another thread), and need to complete
> solidfire.
> > > > This backlog is precisely why need to be reviewing iteratively
> > > > throughout the dev cycle.
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > > > On Jun 27, 2013, at 7:35 PM, David Nalley  wrote:
> > > >
> > > >> On Thu, Jun 27, 2013 at 5:51 PM, Hugo Trippaers
> > > >> 
> > > wrote:
> > > >>>
> > > >>> I think Ilya offers is great, my current stance is also to see
> > > >>> how we can
>

RE: Async Job / Event Bus issue

[Alex Huang]

Ryan,

There's explicit code to remove those two columns.  IIRC, the reason is because 
asyncjob actually uses those two fields to sequence jobs on the same id and 
type.  So when the job is done, then it removes those two fields so it doesn't 
block the next job.

--Alex

> -Original Message-
> From: Ryan Dietrich [mailto:r...@betterservers.com]
> Sent: Tuesday, July 2, 2013 1:18 PM
> To: dev@cloudstack.apache.org
> Subject: Async Job / Event Bus issue
> 
> I've found a bug that I am having trouble determining the cause.  Basically,
> my instance_type and instance_id columns are showing up when the job is
> submitted, but they do not show up when the job is completed or updated.
> When I look in the database, those columns are null, however, they are
> published correctly to the event bus on submit.
> 
> I went a step further and verified that SqlGenerator.java is actually using
> these columns on insertion and for querying the fields back out.
> 
> INSERT INTO async_job (async_job.user_id, async_job.account_id,
> async_job.session_key, async_job.job_cmd, async_job.job_cmd_originator,
> async_job.job_cmd_ver, async_job.job_cmd_info, async_job.callback_type,
> async_job.callback_address, async_job.job_status,
> async_job.job_process_status, async_job.job_re sult_code,
> async_job.job_result, async_job.instance_type, async_job.instance_id,
> async_job.job_init_msid, async_job.job_complete_msid, async_job.created,
> async_job.last_updated, async_job.last_polled, async_job.uuid) VALUES
> (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
> 
> I also verified that the column has a database @Column annotation.
> SqlGenerator appears to be using reflection to determine the "attributes"
> that get mapped into columns.
> 
> (from AsyncJobVO.java)
> @Enumerated(value=EnumType.STRING)
> @Column(name="instance_type", length=64)
> private Type instanceType;
> 
> @Column(name="instance_id", length=64)
> private Long instanceId;
> 
> What could be causing the instance_id and instance_type to drop on the
> floor?  I am literally logging the job right after it says it has been 
> written to the
> database, I log it again right after it is retrieved from the dao via 
> findById.
> Either the dao isn't populating the volumes when it queries the AsyncJobVO
> back out (and subsequently updates the columns with null values), or it's
> something that I'm simply not seeing.
> 
> Help?
> 
> On Jul 2, 2013, at 8:15 AM, Ryan Dietrich  wrote:
> 
> > Patch submitted.
> >
> > https://reviews.apache.org/r/12223/
> >
> > Things that still need to be thought through:
> >
> > 1. Should the async command event type be a column on async job (and be
> an attribute of AsyncJobVO)?
> > 2. Should getInstanceId and getInstanceType of AsyncCmd be abstract
> > instead of implemented with null values? (would require updating every
> > async command in the system, and would break everyones async plugin
> > commands)
> >
> > Oh, and after figuring out how the parameter annotation works (and
> ApiDispatcher uses reflection to iterate over parameters, ugh), I saw
> BaseCmd.java has a HashMap of the arguments passed in
> from the user.
> > I went with the APIDBUtils query function for UUID's because I saw no
> precedent for accessing that HashMap after initial argument handling.
> >
> > -Ryan Dietrich

Coding Convention Reminder

[Alex Huang]

I like to remind everyone to review our coding conventions.  Our coding 
conventions have been going all over the place recently.  Please take a look.

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions

I also like to propose that we extend the 120 column limit to 180 columns. 

I recently was reading the following code.  If it followed even our current 
coding conventions, this would have been 11 lines but it ends up to be 23 
lines, more than doubled.  The whole file was like this.  Just thinking about 
all the extra scrolling I have to do makes my cts act up.  We are in the 21st 
century and using wide screen lcd monitors.  Let's not format our code to fit 
80 column amber text screens please!

What's worse is I've found that some people are actively breaking existing 
source code to 80 columns, causing a bunch of unnecessary merge activities.  On 
Eclipse, you can actually set all types of formatting rules.  I can send out my 
epf (eclipse preferences file) if you don't want to deal with the trouble.  
DiskOfferingVO diskOffering = _diskOfferingDao
.findById(vol.getDiskOfferingId());
if (diskOffering.getUseLocalStorage()) {
if (s_logger.isDebugEnabled()) {
s_logger.debug("Local volume "
+ vol
+ " will be recreated on storage 
pool "
+ assignedPool
+ " assigned by deploymentPlanner");
}
VolumeTask task = new 
VolumeTask(VolumeTaskType.RECREATE, vol, null);
tasks.add(task);
} else {
if (s_logger.isDebugEnabled()) {
s_logger.debug("Shared volume "
+ vol
+ " will be migrated on storage 
pool "
+ assignedPool
+ " assigned by deploymentPlanner");
}
VolumeTask task = new 
VolumeTask(VolumeTaskType.MIGRATE, vol, assignedPool);
tasks.add(task);
}

Please be mindful of my cts.  Thanks for caring about my health!  :)

--Alex

RE: Coding Convention Reminder

[Alex Huang]

I checked it into tools/eclipse/eclipse.epf.

On Saving a file, it performs the following actions:
  - reorganizes imports to the following order: java, javax, org. com, 
org.apache.cloudstack. com.cloud.  This places all of the CloudStack imports at 
the bottom of the import list.
  - Removes all trailing white space
  - Removes unused imports
  - Reformats the EDITED portions of the file using the formatting rules.

It also enforces space for indent.  

--Alex

> -Original Message-
> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com]
> Sent: Tuesday, July 2, 2013 8:05 PM
> To: dev@cloudstack.apache.org
> Subject: RE: Coding Convention Reminder
> 
> +1. Please check-in the preferences file.
> 
> > -Original Message-
> > From: Vijayendra Bhamidipati
> > [mailto:vijayendra.bhamidip...@citrix.com]
> > Sent: Wednesday, July 3, 2013 5:58 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: Coding Convention Reminder
> >
> > +1 to checking in the epf file to source, will be good to have it as a
> > +reference
> > for those using other IDEs as well.
> >
> > Regards,
> > Vijay
> >
> > -Original Message-
> > From: Min Chen [mailto:min.c...@citrix.com]
> > Sent: Tuesday, July 02, 2013 5:21 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: Coding Convention Reminder
> >
> > +1. It is better to check in your eclipse.eps into git repo, so
> > +anybody
> > who checked out the CloudStack can access the preference file.
> >
> > -min
> >
> > On 7/2/13 5:17 PM, "Prachi Damle"  wrote:
> >
> > >+ 1
> > >Please upload the code template for reference.
> > >
> > >-Original Message-
> > >From: Edison Su [mailto:edison...@citrix.com]
> > >Sent: Tuesday, July 02, 2013 5:08 PM
> > >To: dev@cloudstack.apache.org
> > >Subject: RE: Coding Convention Reminder
> > >
> > >We'd better export the code template from Eclipse, then everybody can
> > >import the template into their IDE, then possible, everybody will
> > >have the same configuration.
> > >
> > >> -Original Message-
> > >> From: Alex Huang [mailto:alex.hu...@citrix.com]
> > >> Sent: Tuesday, July 02, 2013 3:11 PM
> > >> To: dev@cloudstack.apache.org
> > >> Subject: Coding Convention Reminder
> > >>
> > >> I like to remind everyone to review our coding conventions.  Our
> > >>coding conventions have been going all over the place recently.
> > >>Please take a look.
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+convent
> > >> i
> > >> ons
> > >>
> > >> I also like to propose that we extend the 120 column limit to 180
> > >>columns.
> > >>
> > >> I recently was reading the following code.  If it followed even our
> > >> current coding conventions, this would have been 11 lines but it
> > >> ends up to be 23 lines, more than doubled.  The whole file was like this.
> > >> Just thinking about all the extra scrolling I have to do makes my
> > >> cts act up.  We are in the 21st century and using wide screen lcd
> > >> monitors.  Let's not format our code to fit
> > >> 80 column amber text screens please!
> > >>
> > >> What's worse is I've found that some people are actively breaking
> > >>existing source code to 80 columns, causing a bunch of unnecessary
> > >>merge activities.
> > >> On Eclipse, you can actually set all types of formatting rules.  I
> > >>can  send out my epf (eclipse preferences file) if you don't want to
> > >>deal with the trouble.
> > >> DiskOfferingVO diskOffering =
> > >>_diskOfferingDao
> > >> .findById(vol.getDiskOfferingId());
> > >> if (diskOffering.getUseLocalStorage()) {
> > >> if (s_logger.isDebugEnabled()) {
> > >> s_logger.debug("Local volume "
> > >> + vol
> > >> + " will be recreated
> > >>on storage pool "
> > >> + assignedPool
> > >> + " assigned by
> > >>dep

[ECLIPSE] Atlassian plugin...

[Alex Huang]

Hi All,

Hugo showed me this really cool eclipse plugin from Atlassian which makes Jira. 
 It allows you to list all of the bugs assigned to you and then changes the 
workspace and the files opened based on which bug you're working on.  You can 
also make changes to the bug itself.  Very cool!  Thanks Hugo!

You can find this plugin by going to 

Help->Install New Software
Select All Available Sites
The plugin is under Atlassian Connector.

After installation, you can configure the plugin to point to our jira[1] and 
then configure it to search for unresolved bugs assigned to myself.  Issues 
will appear in the Task View.  To work on an issue, right click on the issue, 
and select activate.  When you're done, then select deactivate.  

--Alex

[1] https://issues.apache.org/jira

RE: [MERGE] Merge VMSync improvement branch into master

[Alex Huang]

While I'm working on the BVT issues to be resolved.  I will attempt to pull 
some of the less architectural changes over to master in pieces.  That way when 
people do have to review the VMSync code, the code difference will be much 
smaller.  Hopefully, the merges will be smaller too.

For these changes, I will not be asking for review requests.  They're 
refactoring done by Eclipse..

--Alex

> -Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: Monday, July 1, 2013 10:20 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [MERGE] Merge VMSync improvement branch into master
> 
> Ilya,
> 
> Thanks!  The current problem on this branch is we're trying to distinguish
> what's actually broken on master and what's broken by introducing this
> change.  At this point, the branch is basically frozen and only applying bug
> fixes once BVT found problems.
> 
> --Alex
> 
> > -Original Message-
> > From: Musayev, Ilya [mailto:imusa...@webmd.net]
> > Sent: Monday, July 1, 2013 10:16 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: [MERGE] Merge VMSync improvement branch into master
> >
> > Alex,
> >
> > I completely understand. Please keep us in the loop on the progress.
> > If its functional to some extent - i.e. at least build succeeds
> > without errors and major functionality is there, I can merge the code
> > into CloudSand CS distro - to test it out and share it with whoever wants to
> try it.
> >
> > Thanks
> > ilya
> >
> > > -Original Message-
> > > From: Alex Huang [mailto:alex.hu...@citrix.com]
> > > Sent: Friday, June 28, 2013 9:16 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: [MERGE] Merge VMSync improvement branch into master
> > >
> > > Given the current state of BVT, I don't think we can reliably merge
> > > this into master.  It will have to wait for 4.3.  I apologize to
> > > those who really want to see this feature in.  I myself have slaved
> > > over this for some weeks, including missing the collab conference,
> > > but I just cannot conscientiously push it in under the current
> circumstances.
> > >
> > > --Alex
> > >
> > > > -Original Message-
> > > > From: Sudha Ponnaganti [mailto:sudha.ponnaga...@citrix.com]
> > > > Sent: Friday, June 28, 2013 7:11 AM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: RE: [MERGE] Merge VMSync improvement branch into master
> > > >
> > > > Ideally I would like to see all validation to be done on feature
> > > > branch - BVTs and also manual validation of the P1 test cases from
> > > > VM Sync test plan just like object store validation - sadhu has
> > > > signed up for it along with Ilya. This would help us to identify
> > > > feature
> > specific issues.
> > > >
> > > > We are running BVTs right now which have high failure rate.
> > > > Focusing on this part right now to reach parity with Master branch.
> > > >
> > > >
> > > > -Original Message-
> > > > From: Trippie [mailto:trip...@gmail.com] On Behalf Of Hugo
> > > > Trippaers
> > > > Sent: Thursday, June 27, 2013 5:32 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [MERGE] Merge VMSync improvement branch into master
> > > >
> > > > I agree with John that a change like this is very hard to test in
> > > > an automated fashion. Still i have been looking at the numbers for
> > > > the code coverage with cobertura. I was a bit disappointed to find
> > > > that we have not made any progress with this merge with regards to
> > > > unit tests and
> > > total code coverage.
> > > > We do not seem to be in a worse shape than before.
> > > >
> > > > @Sudha, it would be nice if you could add your view on this patch
> > > > from the QA perspective? How would this patch affect your planning
> > > > for
> > > example?
> > > >
> > > > Cheers,
> > > >
> > > > Hugo
> > > >
> > > > On Jun 27, 2013, at 5:12 PM, John Burwell 
> wrote:
> > > >
> > > > > @David The types of concurrency changes introduced in this patch
> > > > > are extremely difficult to completely test in an automated fashion.
> > > > > Therefore, code review for correctness is critical to ensure quality.
> > > &g

RE: Coding Convention Reminder

[Alex Huang]

I think the coding convention did not cover these.  It's a good time to update 
the wiki for python and javascript. 

--Alex

> -Original Message-
> From: Ian Duffy [mailto:i...@ianduffy.ie]
> Sent: Wednesday, July 3, 2013 1:49 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Coding Convention Reminder
> 
> Any suggestions for formatting the Javascript files? The few I've looked at
> seem to be riddled with indentation errors and trailing whitespace. When I
> right click on them in eclipse and hit clean up I get the following error: 
> "The
> resource is not on the include path of a javascript project"
> 
> Thanks!
> Ian
> 
> On 3 July 2013 07:36, Wido den Hollander  wrote:
> > On 07/03/2013 12:11 AM, Alex Huang wrote:
> >>
> >> I like to remind everyone to review our coding conventions.  Our
> >> coding conventions have been going all over the place recently.
> >> Please take a look.
> >>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+convent
> >> ions
> >>
> >> I also like to propose that we extend the 120 column limit to 180 columns.
> >>
> >
> > +1 on that.
> >
> > It's not only line length, but I also still see a lot of trailing
> > spaces, tabs and empty lines with spaces.
> >
> > I'm not using Eclipse (hate it), so I'm using Kate on my Ubuntu
> > desktop which I find great to work with, but that's also completely
> configurable.
> >
> > Wido
> >
> >
> >> I recently was reading the following code.  If it followed even our
> >> current coding conventions, this would have been 11 lines but it ends
> >> up to be 23 lines, more than doubled.  The whole file was like this.
> >> Just thinking about all the extra scrolling I have to do makes my cts
> >> act up.  We are in the 21st century and using wide screen lcd
> >> monitors.  Let's not format our code to fit 80 column amber text screens
> please!
> >>
> >> What's worse is I've found that some people are actively breaking
> >> existing source code to 80 columns, causing a bunch of unnecessary
> merge activities.
> >> On Eclipse, you can actually set all types of formatting rules.  I
> >> can send out my epf (eclipse preferences file) if you don't want to
> >> deal with the trouble.
> >>  DiskOfferingVO diskOffering =
> >> _diskOfferingDao
> >>  .findById(vol.getDiskOfferingId());
> >>  if (diskOffering.getUseLocalStorage()) {
> >>  if (s_logger.isDebugEnabled()) {
> >>  s_logger.debug("Local volume "
> >>  + vol
> >>  + " will be recreated on
> >> storage pool "
> >>  + assignedPool
> >>  + " assigned by
> >> deploymentPlanner");
> >>  }
> >>  VolumeTask task = new
> >> VolumeTask(VolumeTaskType.RECREATE, vol, null);
> >>  tasks.add(task);
> >>  } else {
> >>  if (s_logger.isDebugEnabled()) {
> >>  s_logger.debug("Shared volume "
> >>  + vol
> >>  + " will be migrated on
> >> storage pool "
> >>  + assignedPool
> >>  + " assigned by
> >> deploymentPlanner");
> >>  }
> >>  VolumeTask task = new
> >> VolumeTask(VolumeTaskType.MIGRATE, vol, assignedPool);
> >>  tasks.add(task);
> >>  }
> >>
> >> Please be mindful of my cts.  Thanks for caring about my health!  :)
> >>
> >> --Alex
> >>
> >

RE: How to evaluate performance of a resource allocation policy?

[Alex Huang]

> 
> Consider using the simulator plugin, which will simulate hosts.  You can 
> follow
> the testing instructions on the Marvin testing page [1], but modify your setup
> to implement whatever number of hosts you want to test against.  From
> there, you would need to modify the allocator that's enabled.
> 
+1  That's exactly how it should be done.  It's how stress tests are performed.

You can modify the simulator plugin to vary its response to play devil's 
advocate to test out different theories.

Also, the simulator today does not have this but you should be able to make it 
implement the PluggableService interface to expose commands to allow your test 
client to adjust responses on the fly.

--Alex

RE: [DISCUSS] If BVT breaks, revert the commits...

[Alex Huang]

Sorry this took a little longer than expected with the holiday in US.  Here's 
the first proposal [1] on the automated test system.  Comments welcome.  

Specifically, I have one question on if we should have a staging branch for all 
release branches and master where all checkins go and the build system 
automatically cherry-pick over commits that passes the tests.

I believe we have a lot of pieces in place.  Prassana, Sudha, Rayees, Ram 
Ganesh, myself, and a few others will be working to get this system in place.  

--Alex
[1] 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Automated+Tests+Rules+and+Guidelines

> -Original Message-
> From: David Nalley [mailto:da...@gnsa.us]
> Sent: Friday, June 28, 2013 8:44 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] If BVT breaks, revert the commits...
> 
> On Fri, Jun 28, 2013 at 8:18 PM, Alex Huang  wrote:
> > After Dave's complain in the vmsync [MERGE] thread about BVT in horrible
> shape on master, I went around to figure out what exactly happened.  The
> best I can figure is that after a certain merge (I will leave out which merge 
> as
> that's not important), BVT no longer runs automatically.  It was promised to
> be fixed and there are people who are actively fixing it but it's been in this
> way for about two weeks.  People running BVTs are working around the
> problem but it's not automated anymore and so it's no longer running on
> master.  I understand people are nice and tried to be accommodating to
> other people by working around the problem but sometimes we just have to
> be an arse.  So let me be that arse...
> >
> > New Rule
> > If BVT or automated regression tests break on master or any release
> branch, we revert all commits that broke it.  It doesn't matter if they 
> promise
> to fix it within the next hour.  If it's broken, the release manager will 
> revert
> the commits and developers must resubmit.  It sounds mean but it's the only
> way this problem can be fixed.
> >
> > To avoid having a bunch of reverts and resubmits, the developers should
> be able to request that BVT run on their branch and don't merge until BVT on
> their branch is at 100%.  We will work on figuring out how to do that.
> >
> > Comments?
> >
> > --Alex
> 
> +100 - not only +100 but I will increment ASFBots $beverage counter a
> few in your favor for suggesting this.
> 
> --David

RE: [DISCUSS] What other "features" or code is sitting around that might be suffering from bit rot?

[Alex Huang]

Thanks Chip for starting this thread.

I can at least think of the netapp plugin integration as something that was 
tried before ASF but no longer tested and used.

I'm all for coming up with this list but I don't see how this list can be 
conclusive.  The problem that Edison dealt with in swift support is very real.  
I'm not trying to make an excuse for making a unilateral decision to drop 
support without getting community feedback first.  That is wrong and I'm glad 
it's being resolved.  However, without automated testing backing functionality, 
we can have different set of these problems pop up every release.  Should we 
start with only functionalities tested in automated testing to be the supported 
feature set and add features back in as more testing is added back in? 

To that end, how much of 4.0/4.1/4.2 features are actually added to automated 
testing?  Or else we can face the same problem with those features too.

I still support gathering this list, even if it might be inconclusive. 

--Alex

> -Original Message-
> From: Chip Childers [mailto:chip.child...@sungard.com]
> Sent: Wednesday, July 10, 2013 6:23 AM
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS] What other "features" or code is sitting around that might
> be suffering from bit rot?
> 
> Hi all,
> 
> So we've run into a couple of features that have turned out to have never
> really been "production grade", perhaps due to their creation as prototypes
> during the cloud.com startup period.  Swift, Bare metal provisioning and
> OVM are examples.  Bare metal is obviously resolved now, but Swift is an
> open discussion and OVM remains open for someone to decide to fix it.
> 
> I'd like to propose that those devs that have been around this code-base
> from before it's entry into Apache take some time to review things from the
> past.  What else is lurking in the repo that some of us might
> *think* are functional, but that haven't been tested in years?  What code
> was a prototype that never got fully implemented / supported?
> 
> If we can get all this out in the open, perhaps we can have a solid foundation
> to move forward.
> 
> On the other hand, if nobody has any examples beyond the ones listed
> below, then I think that those of us that are relatively new to the code will
> have to work under the assumption that everything is expected to be
> functional.
> 
> After we establish our foundation, we will need to be very consistent about
> our support of the features.  We'll need to be clear about intentions to
> deprecate something, and perhaps even provide a full feature release cycle
> worth of warning about a pending deprecation.
> 
> As a user, I not been stung by a feature that was yanked... but I was 
> certainly
> surprised when I found out that OVM and Bare Metal weren't being kept up
> to date (again, bare metal is resolved now).  Those features were part of our
> evaluation of the software, and me $dayjob has plans to at least use bare
> metal.
> 
> So why did I share that little story?  Well, it's because features coming and
> going are actually significant events to users of the software.  Just because
> we don't know of someone using a feature doesn't mean that it isn't in use.
> I'd like us to have that solid foundation as a start, and then be very clear
> when we need/want to make a decision that removes a feature from the
> software.
> 
> -chip

RE: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?

[Alex Huang]

John,

I'm concerned that object store implementations that's going into 4.2 will 
repeat this fate if we don't add them into the automated test environment.  
Perhaps, you, me, Edison, Prassana, and perhaps Thomas can work together about 
how to add the current implementations into the regression test suite?

--Alex

> -Original Message-
> From: John Burwell [mailto:jburw...@basho.com]
> Sent: Wednesday, July 10, 2013 2:15 PM
> To: dev@cloudstack.apache.org
> Cc: 'Caleb Call'
> Subject: Re: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?
> 
> All,
> 
> For me, there are significant issues with the object_store patch.  First, it 
> was
> merged to master with a unresolved -1 against it.  Second, it merged a
> feature depreciation without community consensus.  On their own, each of
> these actions violate core community values.  Cumulatively, I am concerned
> that these actions will erode our self governance, collaboration, technical
> quality, and community growth.  So, as Matt suggested, let's focus on re-
> implementing and testing Swift integration, and ensuring that these process
> anomalies remain isolated rather than the beginning of a destructive trend.
> In that vein, how can I help fill this gap?
> 
> Thanks,
> -John
> 
> P.S. I highly suggest the devstack (http://devstack.org) project to get a 
> Swift
> instance up and running.  With it, you can build a full OpenStack (including
> Swift) environment locally in an hour or two (dependent on Internet
> connection speeds).
> 
> On Jul 10, 2013, at 2:35 PM, Chip Childers 
> wrote:
> 
> > On Wed, Jul 10, 2013 at 06:13:07PM +, Edison Su wrote:
> >> 1. Add swift back is just one or two days work, plus maybe one or two
> days, to setup a swift environment.
> >
> > Great!
> >
> >> 3. If we add this feature back, will we test it for each release? Such as
> adding it into automate test? Right now, I break this feature, I am pretty 
> sure,
> it will be broken by other developers, if we continue adding feature without
> test.
> >
> > Then let's test it until such time that we actually agree to deprecate
> > it (if that ever happens).
> >
> >> 4.  Claim a feature is supported for each release without test, is worse
> than saying not supported a feature. If we want to support a feature, we
> should test it for each release. If so, who will want to test this feature?
> >
> > As stated earlier, we have a user that's volunteered to test it out
> > for us already.

RE: Expanding a volume on a SAN

[Alex Huang]

You might have to force a reconnection to the xenserver in order for CS to see 
it.  On every connection, CS checks these items and updates its database.

--Alex

> -Original Message-
> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> Sent: Wednesday, July 10, 2013 2:14 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Expanding a volume on a SAN
> 
> OK, great - thanks!
> 
> So, it sounds like on XenServer, you have to perform some manual activity
> for it to see the new size of the iSCSI target (and there is probably a 
> similar
> requirement with ESX), but CloudStack will notice the size difference
> automatically (once the storage repository or datastore has been manually
> re-configured).
> 
> 
> On Wed, Jul 10, 2013 at 2:48 PM, Anthony Xu  wrote:
> 
> > CS will see the new size,
> >
> >
> > Anthony
> >
> > -Original Message-
> > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > Sent: Tuesday, July 09, 2013 5:25 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: Expanding a volume on a SAN
> >
> > Hey Anthony,
> >
> > I assume this would be a candidate situation where you'd put the
> > primary storage in maintenance mode and then perform the steps you
> referred me to?
> >
> > When the storage is brought out of maintenance mode, will it see the
> > new size or is there something more that has to be done on the CS side?
> >
> > Thanks!
> >
> >
> > On Tue, Jul 9, 2013 at 6:09 PM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com
> > > wrote:
> >
> > > Thanks!
> > >
> > > Anyone know if CloudStack will recognize the new size of a storage
> > > repository or datastore on its own?
> > >
> > >
> > > On Tue, Jul 9, 2013 at 4:01 PM, Anthony Xu 
> wrote:
> > >
> > >> http://support.citrix.com/article/CTX120865
> > >>
> > >> for XenServer, VMs need to be shut down or migrated away before
> > >> expanding a volume.
> > >>
> > >>
> > >> Anthony
> > >>
> > >> -Original Message-
> > >> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > >> Sent: Tuesday, July 09, 2013 2:25 PM
> > >> To: dev@cloudstack.apache.org
> > >> Cc: Edison Su; John Burwell
> > >> Subject: Expanding a volume on a SAN
> > >>
> > >> Hi everyone,
> > >>
> > >> I had a question posed to me today regarding how CloudStack and the
> > >> underlying hypervisor deal with an iSCSI volume that is expanded.
> > >>
> > >> For example, let's say I'm using XenServer or ESX and I create a
> > >> storage repository or datastore, respectively, for each hypervisor
> > >> based on an iSCSI target. I then tie this into CloudStack as
> > >> Primary
> > Storage.
> > >>
> > >> If I increase the size of the iSCSI target (the SAN volume/LUN),
> > >> does this increased size feed into the hypervisor and CloudStack?
> > >>
> > >> Thanks!
> > >>
> > >> --
> > >> *Mike Tutkowski*
> > >> *Senior CloudStack Developer, SolidFire Inc.*
> > >> e: mike.tutkow...@solidfire.com
> > >> o: 303.746.7302
> > >> Advancing the way the world uses the
> > >> cloud
> > >> *(tm)*
> > >>
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkow...@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the
> > > cloud
> > > *(tm)*
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkow...@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud
> > *(tm)*
> >
> 
> 
> 
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *(tm)*

RE: [DISCUSS] What other "features" or code is sitting around that might be suffering from bit rot?

[Alex Huang]

> -Original Message-
> From: Sebastien Goasguen [mailto:run...@gmail.com]
> Sent: Wednesday, July 10, 2013 3:38 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] What other "features" or code is sitting around that
> might be suffering from bit rot?
> 
> 
> On Jul 10, 2013, at 5:06 PM, Chip Childers 
> wrote:
> 
> > On Wed, Jul 10, 2013 at 08:44:24PM +, Alex Huang wrote:
> >> Thanks Chip for starting this thread.
> >>
> >> I can at least think of the netapp plugin integration as something that was
> tried before ASF but no longer tested and used.
> >>
> >
> > Awesome, that's one.  Any others?
> 
> The S3 interface in awsapi ?
> 


AFAIK, awsapi was EC2 only.  Prachi?

--Alex

RE: Expanding a volume on a SAN

[Alex Huang]

ah sorry.  Yes.  From the CloudStack management UI, you can force cloudstack to 
reconnect to the host.  This forces cloudstack to flush it's connection and 
restablish the connection.  On restablishing the connection, there's a series 
of checks and information gathered.  Size of the storage pool is one such 
information.

--Alex

> -Original Message-
> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> Sent: Wednesday, July 10, 2013 3:13 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Expanding a volume on a SAN
> 
> Hey Alex,
> 
> When you say, "force a reconnection," I'm not sure to what part of the
> system you're referring. Is this an action performed on the CloudStack side?
> 
> Thanks!
> 
> 
> On Wed, Jul 10, 2013 at 4:10 PM, Alex Huang  wrote:
> 
> > You might have to force a reconnection to the xenserver in order for
> > CS to see it.  On every connection, CS checks these items and updates
> > its database.
> >
> > --Alex
> >
> > > -Original Message-
> > > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > > Sent: Wednesday, July 10, 2013 2:14 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: Expanding a volume on a SAN
> > >
> > > OK, great - thanks!
> > >
> > > So, it sounds like on XenServer, you have to perform some manual
> > > activity for it to see the new size of the iSCSI target (and there
> > > is probably a
> > similar
> > > requirement with ESX), but CloudStack will notice the size
> > > difference automatically (once the storage repository or datastore
> > > has been manually re-configured).
> > >
> > >
> > > On Wed, Jul 10, 2013 at 2:48 PM, Anthony Xu 
> > wrote:
> > >
> > > > CS will see the new size,
> > > >
> > > >
> > > > Anthony
> > > >
> > > > -Original Message-
> > > > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > > > Sent: Tuesday, July 09, 2013 5:25 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: Expanding a volume on a SAN
> > > >
> > > > Hey Anthony,
> > > >
> > > > I assume this would be a candidate situation where you'd put the
> > > > primary storage in maintenance mode and then perform the steps you
> > > referred me to?
> > > >
> > > > When the storage is brought out of maintenance mode, will it see
> > > > the new size or is there something more that has to be done on the CS
> side?
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > On Tue, Jul 9, 2013 at 6:09 PM, Mike Tutkowski <
> > > > mike.tutkow...@solidfire.com
> > > > > wrote:
> > > >
> > > > > Thanks!
> > > > >
> > > > > Anyone know if CloudStack will recognize the new size of a
> > > > > storage repository or datastore on its own?
> > > > >
> > > > >
> > > > > On Tue, Jul 9, 2013 at 4:01 PM, Anthony Xu
> > > > > 
> > > wrote:
> > > > >
> > > > >> http://support.citrix.com/article/CTX120865
> > > > >>
> > > > >> for XenServer, VMs need to be shut down or migrated away before
> > > > >> expanding a volume.
> > > > >>
> > > > >>
> > > > >> Anthony
> > > > >>
> > > > >> -Original Message-
> > > > >> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > > > >> Sent: Tuesday, July 09, 2013 2:25 PM
> > > > >> To: dev@cloudstack.apache.org
> > > > >> Cc: Edison Su; John Burwell
> > > > >> Subject: Expanding a volume on a SAN
> > > > >>
> > > > >> Hi everyone,
> > > > >>
> > > > >> I had a question posed to me today regarding how CloudStack and
> > > > >> the underlying hypervisor deal with an iSCSI volume that is
> expanded.
> > > > >>
> > > > >> For example, let's say I'm using XenServer or ESX and I create
> > > > >> a storage repository or datastore, respectively, for each
> > > > >> hypervisor based on an iSCSI target. I then tie this into
> > > > >> CloudStack as Primary
> > > > Storage.
> > > > 

RE: [DISCUSS] What other "features" or code is sitting around that might be suffering from bit rot?

[Alex Huang]

> 
> I agree with your points above, but think that we need to take this step by
> step.  Let's figure out what code isn't actually in shape, based on historical
> understanding first.  We then at least have a target to ask the next question:
> what's covered by testing (automated or manual) for each feature release?
> Then we have a list of areas to focus on for automated testing.
> 

Agreed.  I guess I'm just thinking ahead given that this thread is already 
started.

Here's some more I can think of:

CloudZones implementation which allows someone to deploy a zone in their own 
data center but uses a hosted management server to manage it.
Local Storage based secondary storage which uses available local storage in the 
hypervisor host for secondary storage.
HyperV prototype (Not the new one that Donal is working on)
Cifs based secondary storage as part of the HyperV work

--Alex

RE: Expanding a volume on a SAN

[Alex Huang]

Cooli like that even better.  :)

--Alex

> -Original Message-
> From: Anthony Xu [mailto:xuefei...@citrix.com]
> Sent: Wednesday, July 10, 2013 4:17 PM
> To: dev@cloudstack.apache.org
> Subject: RE: Expanding a volume on a SAN
> 
> If you put the primary storage in maintenance mode to evacuate the VM,
> when you cancel maintenance mode for the primary storage, CS will get the
> new size of the primary storage.
> 
> 
> Anthony
> 
> -----Original Message-
> From: Alex Huang [mailto:alex.hu...@citrix.com]
> Sent: Wednesday, July 10, 2013 4:07 PM
> To: dev@cloudstack.apache.org
> Subject: RE: Expanding a volume on a SAN
> 
> ah sorry.  Yes.  From the CloudStack management UI, you can force
> cloudstack to reconnect to the host.  This forces cloudstack to flush it's
> connection and restablish the connection.  On restablishing the connection,
> there's a series of checks and information gathered.  Size of the storage pool
> is one such information.
> 
> --Alex
> 
> > -Original Message-
> > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > Sent: Wednesday, July 10, 2013 3:13 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: Expanding a volume on a SAN
> >
> > Hey Alex,
> >
> > When you say, "force a reconnection," I'm not sure to what part of the
> > system you're referring. Is this an action performed on the CloudStack side?
> >
> > Thanks!
> >
> >
> > On Wed, Jul 10, 2013 at 4:10 PM, Alex Huang 
> wrote:
> >
> > > You might have to force a reconnection to the xenserver in order for
> > > CS to see it.  On every connection, CS checks these items and
> > > updates its database.
> > >
> > > --Alex
> > >
> > > > -Original Message-
> > > > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > > > Sent: Wednesday, July 10, 2013 2:14 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: Expanding a volume on a SAN
> > > >
> > > > OK, great - thanks!
> > > >
> > > > So, it sounds like on XenServer, you have to perform some manual
> > > > activity for it to see the new size of the iSCSI target (and there
> > > > is probably a
> > > similar
> > > > requirement with ESX), but CloudStack will notice the size
> > > > difference automatically (once the storage repository or datastore
> > > > has been manually re-configured).
> > > >
> > > >
> > > > On Wed, Jul 10, 2013 at 2:48 PM, Anthony Xu 
> > > wrote:
> > > >
> > > > > CS will see the new size,
> > > > >
> > > > >
> > > > > Anthony
> > > > >
> > > > > -Original Message-
> > > > > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com]
> > > > > Sent: Tuesday, July 09, 2013 5:25 PM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: Expanding a volume on a SAN
> > > > >
> > > > > Hey Anthony,
> > > > >
> > > > > I assume this would be a candidate situation where you'd put the
> > > > > primary storage in maintenance mode and then perform the steps
> > > > > you
> > > > referred me to?
> > > > >
> > > > > When the storage is brought out of maintenance mode, will it see
> > > > > the new size or is there something more that has to be done on
> > > > > the CS
> > side?
> > > > >
> > > > > Thanks!
> > > > >
> > > > >
> > > > > On Tue, Jul 9, 2013 at 6:09 PM, Mike Tutkowski <
> > > > > mike.tutkow...@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Anyone know if CloudStack will recognize the new size of a
> > > > > > storage repository or datastore on its own?
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 9, 2013 at 4:01 PM, Anthony Xu
> > > > > > 
> > > > wrote:
> > > > > >
> > > > > >> http://support.citrix.com/article/CTX120865
> > > > > >>
> > > > > >> for XenServer, VMs need to be shut down or migrated away
> > > > > >> before expanding a volume.
&

RE: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?

[Alex Huang]

Dave,

It also has to be with native swift API.  It can't be exposing S3 API or their 
own API but using swift in the back end.

--Alex

> -Original Message-
> From: David Nalley [mailto:da...@gnsa.us]
> Sent: Thursday, July 11, 2013 7:45 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?
> 
> On Thu, Jul 11, 2013 at 10:42 AM, Chip Childers 
> wrote:
> > On Thu, Jul 11, 2013 at 10:39 AM, David Nalley  wrote:
> >> On Thu, Jul 11, 2013 at 10:37 AM, Joe Brockmeier 
> wrote:
> >>> On Wed, Jul 10, 2013, at 05:53 PM, Edison Su wrote:
>  Oh man, my two hours are wasted on devstack already. After
>  installed devstack, there is no swift service at all.
> >>>
> >>> For Edison, and anybody else who wants to use Swift with CloudStack,
> >>> what about SwiftStack?
> >>>
> >>> http://swiftstack.com/docs/install/index.html
> >>>
> >>> I think you have to request a trial account, but it looks not overly
> >>> difficult.
> >>>
> >>> Best,
> >>>
> >>> jzb
> >>> --
> >>> Joe Brockmeier
> >>> j...@zonker.net
> >>> Twitter: @jzb
> >>> http://www.dissociatedpress.net/
> >>
> >> I am curious why we'd set this up at all?
> >> There are several public cloud providers offering swift as a service.
> >> If all we are doing is testing, why wouldn't we just use one of them?
> >>
> >> --David
> >>
> >
> > Any *free* ones?  ;-)
> 
> So I know of at least one project that has free access - let me confirm with
> the provider that they'd be amenable to us using it.
> 
> --David

Adding git commit ids to our packaging...

[Alex Huang]

Hi All,

I added the branch and git commit id to the war file.  You can find it by 
opening the war file and retrieving the META-INF/MANIFEST.MF.  Inside, there is 
a Implementation-Revision which contains the git commit id and a 
Implementation-Branch which contains the branch. 

We should add these to all of our jira issues so developers can easily figure 
out the revision being tested and work appropriately.

I will also cherry-pick this to 4.2.  I assume there's no objections.
 
--Alex

RE: Adding git commit ids to our packaging...

[Alex Huang]

Will work on making it nicer but do we want to put this information into 
release?  Is there any harm to have this in the release build?  I can't think 
of any off-hand.  If we don't want them, we can change build_asf.sh to  set 
those two variables to something nicer.  If we want them, I might be able to 
get it to read this information from a file instead.

--Alex

> -Original Message-
> From: Chip Childers [mailto:chip.child...@sungard.com]
> Sent: Thursday, July 11, 2013 6:35 PM
> To: 
> Subject: Re: Adding git commit ids to our packaging...
> 
> Cool. +1 then
> 
> On Jul 11, 2013, at 9:28 PM, David Nalley  wrote:
> 
> > I was worried about the same thing, but ran build_asf.sh to create a
> > tarball and then built it.
> >
> > Here's the resulting output, which isn't perfectly nice, but also isn't 
> > broken.
> >
> > ke4qqq@mba
> > ~/apache-cloudstack-4.2.0-RC2-src/client/target/META-INF[master*]$
> > cat MANIFEST.MF
> > Manifest-Version: 1.0
> > Implementation-Vendor: The Apache Software Foundation
> > Implementation-Title: Apache CloudStack Client UI
> > Implementation-Version: 4.2.0-RC2
> > Implementation-Revision: UNKNOWN_REVISION
> > Implementation-Vendor-Id: org.apache.cloudstack
> > Built-By: ke4qqq
> > Build-Jdk: 1.7.0_09-icedtea
> > Created-By: Apache Maven 3.0.4
> > Implementation-Branch: UNKNOWN_BRANCH
> > Archiver-Version: Plexus Archiver
> >
> > --David
> >
> > On Thu, Jul 11, 2013 at 9:01 PM, Chip Childers
> >  wrote:
> >> Is it off by default?
> >>
> >> Our releases aren't from git.
> >>
> >> On Jul 11, 2013, at 8:38 PM, Alex Huang  wrote:
> >>
> >>> Hi All,
> >>>
> >>> I added the branch and git commit id to the war file.  You can find it by
> opening the war file and retrieving the META-INF/MANIFEST.MF.  Inside,
> there is a Implementation-Revision which contains the git commit id and a
> Implementation-Branch which contains the branch.
> >>>
> >>> We should add these to all of our jira issues so developers can easily
> figure out the revision being tested and work appropriately.
> >>>
> >>> I will also cherry-pick this to 4.2.  I assume there's no objections.
> >>>
> >>> --Alex
> >>>
> >

RE: Adding git commit ids to our packaging...

[Alex Huang]

I think it should be in both.  In fact, I am hoping to push them further down.

Before our switch to git, the build number, which with svn included the 
revision as the last octet, was pushed to every script, javascript, jar, and 
war.  I like to see that back again.  It's very useful when trying to figure 
out exactly what was deployed.  I will work toward that if there's no 
objections.  

I pushed in the changes to war file for now so QAs at least have some way to 
find the revision number to communicate in Jira.

--Alex

> -Original Message-
> From: Chip Childers [mailto:chip.child...@sungard.com]
> Sent: Friday, July 12, 2013 6:53 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Adding git commit ids to our packaging...
> 
> On Fri, Jul 12, 2013 at 09:48:12AM -0400, David Nalley wrote:
> > On Thu, Jul 11, 2013 at 11:05 PM, Alex Huang 
> wrote:
> > > Will work on making it nicer but do we want to put this information into
> release?  Is there any harm to have this in the release build?  I can't think 
> of
> any off-hand.  If we don't want them, we can change build_asf.sh to  set
> those two variables to something nicer.  If we want them, I might be able to
> get it to read this information from a file instead.
> > >
> > > --Alex
> > >
> >
> > Thanks for putting the work in to get this done.
> > I agree - no harm in having this in the release build, and could be
> > very useful to boot.
> > I personally would like to see this information, present - setting it
> > at tarball creation makes sense to me.
> >
> > --David
> >
> 
> +1
> 
> The only reason it was removed previously was that it was breaking the build
> process when someone tried to build from the release tarball.  If we want to
> drop the details into the release archive before tar'ing, great!
> 
> The only other minor concern would be the release verification process will
> have to be updated to account for the fact that we'll have one file (or some
> lines) that are different between the actual tagged commit in the repo and
> what's in the tarball.  As far as I know, that's just a process change for us.
> Easy...
> 
> -chip

RE: RE: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?

[Alex Huang]

> -Original Message-
> From: Chip Childers [mailto:chip.child...@sungard.com]
> Sent: Monday, July 15, 2013 7:25 AM
> To: dev@cloudstack.apache.org
> Cc: Edison Su
> Subject: Re: RE: Swift in 4.2 is broken, anybody wants it to be supported in
> 4.2?
> 
> On Fri, Jul 12, 2013 at 07:08:41PM -0600, Mike Tutkowski wrote:
> > So, I haven't been following this thread in detail, but was curious:
> > If it's too much work to fix this by the end of the month (code
> > freeze), what are we planning on doing (moving 4.2 back or allowing
> > this feature to not exist in 4.2)?
> 
> I'm personally not happy either way, but I'd much rather not break existing
> environments in a new release on purpose.

+1 Same thoughts here.

--Alex

RE: Invoking an API from an API plugin

[Alex Huang]

Chris,

You should be able to just create the command instances yourself and feed it to 
the service class.  It does not require the handler.

--Alex

> -Original Message-
> From: Chiradeep Vittal
> Sent: Monday, July 15, 2013 7:36 PM
> To: dev@cloudstack.apache.org
> Cc: Min Chen; Alex Huang
> Subject: Re: Invoking an API from an API plugin
> 
> Yes, that is unfortunate design. The service interface was not supposed to do
> that.
> CC Min and Alex to see if they have a quick workaround.
> It would also help to know what exactly you are trying to achieve.
> 
> On 7/15/13 11:12 PM, "SuichII, Christopher" 
> wrote:
> 
> >It looks like the service interfaces all expect to be invoked directly
> >from an API handler (they only have Cmds for parameters). For example,
> >QueryService/QueryManagerImpl.searchForServers() takes a ListHostsCmd.
> >This ListsHostsCmd can only be created by invoking the listHosts API.
> >If it had public setters, then the command could be instantiated and
> >execute() could be called directly, but that wouldn't be working at the
> >service interface level.
> >
> >Am I missing something here or is this not the service interface you
> >meant?
> >
> >-Chris
> >
> >On Jul 15, 2013, at 12:29 AM, Chiradeep Vittal
> > wrote:
> >
> >> APIs should call the service interface directly and not call other APIs.
> >>
> >> On 7/12/13 1:40 AM, "SuichII, Christopher" 
> >>wrote:
> >>
> >>> Er, I should have mentioned that it is not as easy as simply
> >>>invoking one  API command from another since all the CS API commands
> >>>have private  @paramenter members. Because of this, I cannot simply
> >>>instantiate a  command, populate it with the parameters and call
> >>>execute() on it.
> >>>
> >>> -Chris
> >>>
> >>> On Jul 11, 2013, at 4:09 PM, Chris Suich 
> >>> wrote:
> >>>
> >>>> I'm writing an API plugin that needs to consume an API that already
> >>>>exists in CS. The only way I can find to do this would be to send
> >>>>the  REST/HTTP request from my plugin, but I'm hoping there is an
> >>>>easier way.
> >>>> Since both plugins are on the same Java classpath they have the
> >>>>ability  to invoke each other.
> >>>>
> >>>> Is there a simple way to do this that I'm missing?
> >>>>
> >>>> -Chris
> >>>
> >>
> >

RE: [discuss] 80 char max length leads to illegiable source

[Alex Huang]

Donal,

Our coding convention says 180.  It was discussed about a week or two back.

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions

--Alex

> -Original Message-
> From: Donal Lafferty [mailto:donal.laffe...@citrix.com]
> Sent: Tuesday, July 16, 2013 6:02 AM
> To: dev@cloudstack.apache.org
> Subject: [discuss] 80 char max length leads to illegiable source
> 
> This issue appeared when I used StyleCheck and Eclipse to find and fix my
> coding convention violations automatically.  Details at
> http://dlafferty.blogspot.co.uk/2013/07/apache-cloudstack-java-coding.html
> 
> We inherit the 80 char line limit inherited from the Java Programming Style,
> and it makes my tests unreadable.  JSON serialised objects in long strings are
> confusing.  Readability problems also arise in non-test Java source.
> Can we loosen/ignore this rule when readability becomes an issue?
> 
> 
> DL

RE: Invoking an API from an API plugin

[Alex Huang]

Chris,

If it is then it was written wrong.  You can change it to public and send in a 
patch.

--Alex

> -Original Message-
> From: SuichII, Christopher [mailto:chris.su...@netapp.com]
> Sent: Monday, July 15, 2013 9:23 PM
> To: 
> Cc: Alex Huang; Chiradeep Vittal; Min Chen
> Subject: Re: Invoking an API from an API plugin
> 
> I thought about this approach, but all of the commands have private
> members/parameters and do not have setters.
> 
> Chris
> 
> On Jul 15, 2013, at 11:50 PM, Alex Huang 
>  wrote:
> 
> > Chris,
> >
> > You should be able to just create the command instances yourself and feed
> it to the service class.  It does not require the handler.
> >
> > --Alex
> >
> >> -Original Message-
> >> From: Chiradeep Vittal
> >> Sent: Monday, July 15, 2013 7:36 PM
> >> To: dev@cloudstack.apache.org
> >> Cc: Min Chen; Alex Huang
> >> Subject: Re: Invoking an API from an API plugin
> >>
> >> Yes, that is unfortunate design. The service interface was not
> >> supposed to do that.
> >> CC Min and Alex to see if they have a quick workaround.
> >> It would also help to know what exactly you are trying to achieve.
> >>
> >> On 7/15/13 11:12 PM, "SuichII, Christopher" 
> >> wrote:
> >>
> >>> It looks like the service interfaces all expect to be invoked
> >>> directly from an API handler (they only have Cmds for parameters).
> >>> For example,
> >>> QueryService/QueryManagerImpl.searchForServers() takes a
> ListHostsCmd.
> >>> This ListsHostsCmd can only be created by invoking the listHosts API.
> >>> If it had public setters, then the command could be instantiated and
> >>> execute() could be called directly, but that wouldn't be working at
> >>> the service interface level.
> >>>
> >>> Am I missing something here or is this not the service interface you
> >>> meant?
> >>>
> >>> -Chris
> >>>
> >>> On Jul 15, 2013, at 12:29 AM, Chiradeep Vittal
> >>>  wrote:
> >>>
> >>>> APIs should call the service interface directly and not call other APIs.
> >>>>
> >>>> On 7/12/13 1:40 AM, "SuichII, Christopher" 
> >>>> wrote:
> >>>>
> >>>>> Er, I should have mentioned that it is not as easy as simply
> >>>>> invoking one  API command from another since all the CS API
> >>>>> commands have private  @paramenter members. Because of this, I
> >>>>> cannot simply instantiate a  command, populate it with the
> >>>>> parameters and call
> >>>>> execute() on it.
> >>>>>
> >>>>> -Chris
> >>>>>
> >>>>> On Jul 11, 2013, at 4:09 PM, Chris Suich 
> >>>>> wrote:
> >>>>>
> >>>>>> I'm writing an API plugin that needs to consume an API that
> >>>>>> already exists in CS. The only way I can find to do this would be
> >>>>>> to send the  REST/HTTP request from my plugin, but I'm hoping
> >>>>>> there is an easier way.
> >>>>>> Since both plugins are on the same Java classpath they have the
> >>>>>> ability  to invoke each other.
> >>>>>>
> >>>>>> Is there a simple way to do this that I'm missing?
> >>>>>>
> >>>>>> -Chris
> >>>>>
> >>>>
> >>>
> >

RE: [discuss] 80 char max length leads to illegiable source

[Alex Huang]

I updated the wiki because several people asked me to post my eclipse 
preference file which includes 180 column widths.  There's also a follow up 
mail thread after I checked it in.  The column width was not questioned.  I can 
send out a separate proposal if needed but supposedly developers are already 
using it.

--Alex

> -Original Message-
> From: Joe Brockmeier [mailto:j...@zonker.net]
> Sent: Tuesday, July 16, 2013 7:26 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [discuss] 80 char max length leads to illegiable source
> 
> On Tue, Jul 16, 2013, at 08:53 AM, Alex Huang wrote:
> > Donal,
> >
> > Our coding convention says 180.  It was discussed about a week or two
> > back.
> 
> So, I am surprised that it's already been changed. I noticed the comment
> when I was doing the weekly news - but it was buried in an email with the
> subject "coding convention reminder" - which tells most readers "this email is
> a reminder, there are no changes involved in this." There was no discussion
> of the change, though there was discussion about the Eclipse templates, etc.
> 
> In the future, might be good to break out proposals for changes in a separate
> email. I suspect a few folks are going to be surprised by the change.
> 
> Best,
> 
> jzb
> --
> Joe Brockmeier
> j...@zonker.net
> Twitter: @jzb
> http://www.dissociatedpress.net/

RE: code formatting for enums

[Alex Huang]

Windows->Preferences
Java->Formatter
Click on Edit in Active Profiles
Line Wrapping tab
Look for 'enum' declaration->Constants
Select Wrap all elements, every element on a new line in the "Line Wrapping 
policy:" drop down

--Alex


> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Wednesday, July 17, 2013 6:22 AM
> To: dev
> Subject: code formatting for enums
> 
> H,
> 
> I am working on Networks with the eclipse.epf file loaded. Now the enum
> BroadcastDomainType gets saved as
> Native(null, null), Vlan("vlan", Integer.class), Vswitch("vs",
> String.class), LinkLocal(null, null), Vnet("vnet", 
> Long.class), Storage(
> "storage", Integer.class), Lswitch("lswitch", String.class), 
> Mido(
> "mido", String.class), Pvlan("pvlan", String.class), 
> UnDecided(
> null, null);
> instead of
> Native(null, null),
> Vlan("vlan", Integer.class),
> Vswitch("vs", String.class),
> LinkLocal(null, null),
> Vnet("vnet", Long.class),
> Storage("storage", Integer.class),
> Lswitch("lswitch", String.class),
> Mido("mido", String.class),
> Pvlan("pvlan", String.class),
> UnDecided(null, null);
> anybody know how to fix this?
> 
> thanks,
> Daan

RE: code formatting for enums

[Alex Huang]

That's because the first rule of auto-formatting is do no harm.

The formatter is set not to screw with lines that are already wrapped assuming 
the previous developer intended it that way.

--Alex

> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Wednesday, July 17, 2013 8:23 AM
> To: dev
> Subject: Re: code formatting for enums
> 
> thanks,
> it doesn't correct back to the one per line format, but at least it doesn't
> garble the enum when right anymore.
> 
> 
> 
> On Wed, Jul 17, 2013 at 4:24 PM, Alex Huang  wrote:
> 
> > Windows->Preferences
> > Java->Formatter
> > Click on Edit in Active Profiles
> > Line Wrapping tab
> > Look for 'enum' declaration->Constants Select Wrap all elements, every
> > element on a new line in the "Line Wrapping policy:" drop down
> >
> > --Alex
> >
> >
> > > -Original Message-
> > > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> > > Sent: Wednesday, July 17, 2013 6:22 AM
> > > To: dev
> > > Subject: code formatting for enums
> > >
> > > H,
> > >
> > > I am working on Networks with the eclipse.epf file loaded. Now the
> > > enum BroadcastDomainType gets saved as
> > > Native(null, null), Vlan("vlan", Integer.class), Vswitch("vs",
> > > String.class), LinkLocal(null, null), Vnet("vnet",
> > Long.class), Storage(
> > > "storage", Integer.class), Lswitch("lswitch",
> > String.class), Mido(
> > > "mido", String.class), Pvlan("pvlan", String.class),
> > UnDecided(
> > > null, null);
> > > instead of
> > > Native(null, null),
> > > Vlan("vlan", Integer.class),
> > > Vswitch("vs", String.class),
> > > LinkLocal(null, null),
> > > Vnet("vnet", Long.class),
> > > Storage("storage", Integer.class),
> > > Lswitch("lswitch", String.class),
> > > Mido("mido", String.class),
> > > Pvlan("pvlan", String.class),
> > > UnDecided(null, null);
> > > anybody know how to fix this?
> > >
> > > thanks,
> > > Daan
> >