Cloustack packaging
Hi all,
I'm new to the cloudstack community, but i'm a seasoned developer and
systems engineer and i'm sure i'll be able to help out a lot :)
I've just started on getting cloudstack running on Centos 7.1 and noticed
many problems, which make it NOT work out of the box.
I'll be doing a series of fixes and submitting the code for review, since
i'm not a commiter yet, if no one else is working on this.
I've seen that there's a branch called centos7-rpm which is already merged
with master, but there hasn't been any activity on that on the last six
months, so i guess it's currently stale.
Perhaps some of these things that i spotted were working on 7.0, but i
haven't tested it out on that version yet.. ideally we can make specs that
work on ANY 7.x version and not have a spec file for every minor release.
One of the things i immediately noticed (that is present in all
centos/fedora specs, not just centos7) is the following:
%define __os_install_post %{nil}
%global debug_package %{nil}
# DISABLE the post-percentinstall java repacking and line number stripping
# we need to find a way to just disable the java repacking and line number
stripping, but not the autodeps
It's easy enough to get the __os_install_post to only do that, but it
shouldn't be disabling autodeps.. rather the rpmbuild's dependency routines
can't find any required external libs in the java or python files :)
Replace:
%define __os_install_post %{nil}
with:
%define __jar_repack %{nil}
%define __debug_package %{nil}
That will do it, as checked in "rpm --showrc", although there won't be much
to be gained with it.
-14: __os_install_post
/usr/lib/rpm/redhat/brp-compress
%{!?__debug_package:
/usr/lib/rpm/redhat/brp-strip %{__strip}
/usr/lib/rpm/redhat/brp-strip-comment-note %{__strip} %{__objdump}
}
/usr/lib/rpm/redhat/brp-strip-static-archive %{__strip}
/usr/lib/rpm/brp-python-bytecompile %{__python}
%{?_python_bytecompile_errors_terminate_build}
/usr/lib/rpm/redhat/brp-python-hardlink
%{!?__jar_repack:/usr/lib/rpm/redhat/brp-java-repack-jars}
%{nil}
Another bad packaging issue
I've started taking a look at the size of the RPMs and the contents.. and it's really nasty :) The cloudstack-common package for instance.. has been over 100MB for at least a few versions... looking into that.. there's a duplication of files regarding the systemvm iso... rpm -qlp cloudstack-common* shows: /usr/share/cloudstack-common/vms/systemvm.iso /usr/share/cloudstack-common/vms/systemvm.zip Zipping this is redundant, since the iso is already compressed. This gets BOTH files into the RPM and subsequently into the server's filesystem: [root@testcloud cloudstack]# ls -lah /usr/share/cloudstack-common/vms/ total 114M drwxr-xr-x. 2 root root 44 Apr 25 11:27 . drwxr-xr-x. 5 root root 40 Apr 25 11:27 .. -rw-r--r--. 1 root root 57M Apr 25 11:18 systemvm.iso -rw-r--r--. 1 root root 57M Apr 25 11:18 systemvm.zip waste of space in both the rpm and the filesystem... it should take half the size. I see no particular benefit on using a zipped version of that iso.. altough an extra set of files need to be edited to get rid of that (like patchsystemvm.sh). This also happens for other packages like cloudstack-management* A lot of .css and .js files are getting compressed and then including both versions into the RPM and finally both versions into the filesystem.. you can check all those duplicate files, for example, in /usr/share/cloudstack-management/webapps/client/scripts/ This directory alone should save us a few megabytes in the RPM and Filesystem.
Re: Cloustack packaging
Hi Rafael,
Welcome to CloudStack and thanks for your mail.
What RPM packages did you use for your testing?
From my personal experience I know that the CentOS7 (el7) packages for
CloudStack 4.4 work just fine as a KVM hypervisor (on CentOS 7.1). In 4.5 work
is done to also make it work as a Management server. So, what exact problems do
you run into when installing the RPM’s on CentOS 7.1?
As far as I know, the centos7-rpm branch is old and no longer maintained. I’d
recommend using the 4.5 and/or master branches instead, as these are most up to
date and actively maintained.
Feel free to send a PR to fix what you think is broken.
Regards,
Remi
> On 26 Apr 2015, at 15:30 , Rafael Fonseca wrote:
>
> Hi all,
>
> I'm new to the cloudstack community, but i'm a seasoned developer and
> systems engineer and i'm sure i'll be able to help out a lot :)
>
> I've just started on getting cloudstack running on Centos 7.1 and noticed
> many problems, which make it NOT work out of the box.
>
> I'll be doing a series of fixes and submitting the code for review, since
> i'm not a commiter yet, if no one else is working on this.
>
> I've seen that there's a branch called centos7-rpm which is already merged
> with master, but there hasn't been any activity on that on the last six
> months, so i guess it's currently stale.
>
> Perhaps some of these things that i spotted were working on 7.0, but i
> haven't tested it out on that version yet.. ideally we can make specs that
> work on ANY 7.x version and not have a spec file for every minor release.
>
> One of the things i immediately noticed (that is present in all
> centos/fedora specs, not just centos7) is the following:
>
> %define __os_install_post %{nil}
> %global debug_package %{nil}
>
> # DISABLE the post-percentinstall java repacking and line number stripping
> # we need to find a way to just disable the java repacking and line number
> stripping, but not the autodeps
>
>
> It's easy enough to get the __os_install_post to only do that, but it
> shouldn't be disabling autodeps.. rather the rpmbuild's dependency routines
> can't find any required external libs in the java or python files :)
>
> Replace:
>
> %define __os_install_post %{nil}
>
> with:
>
> %define __jar_repack %{nil}
> %define __debug_package %{nil}
>
> That will do it, as checked in "rpm --showrc", although there won't be much
> to be gained with it.
>
> -14: __os_install_post
>/usr/lib/rpm/redhat/brp-compress
>%{!?__debug_package:
>/usr/lib/rpm/redhat/brp-strip %{__strip}
>/usr/lib/rpm/redhat/brp-strip-comment-note %{__strip} %{__objdump}
>}
>/usr/lib/rpm/redhat/brp-strip-static-archive %{__strip}
>/usr/lib/rpm/brp-python-bytecompile %{__python}
> %{?_python_bytecompile_errors_terminate_build}
>/usr/lib/rpm/redhat/brp-python-hardlink
>%{!?__jar_repack:/usr/lib/rpm/redhat/brp-java-repack-jars}
> %{nil}
[GitHub] cloudstack pull request: fix CLOUDSTACK-8402
GitHub user giraffeforestg opened a pull request: https://github.com/apache/cloudstack/pull/192 fix CLOUDSTACK-8402 You can merge this pull request into a Git repository by running: $ git pull https://github.com/giraffeforestg/cloudstack bugfix_CLOUDSTACK-8402 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/192.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #192 commit a5732be186a72db2927634a75815eae242299d2e Author: Satoru Nakaya Date: 2015-04-26T14:10:06Z fix CLOUDSTACK-8402 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Cloustack packaging
Hi Remi,
I'm building the RPMs from the latest master branch, i just referred to
that old branch because it was the only one i found where someone was
working on this.
Some of the issues i've encountered were:
Thsese files did not get created by the install, so
the cloudstack-setup-management script failed.
/etc/cloudstack/management/server-nonssl.xml
/etc/cloudstack/management/tomcat6-nonssl.conf
Also, when this script failed it refered to a wrong log file...
Configure CloudStack Management Server ...[Failed]
Failed to configure CloudStack Management Server, please see the
/var/log/cloudstack/setupManagement.log for detail
wrong file.. correct is /var/log/cloudstack/management/setupManagement.log
Besides that, the systemd init script tries to use /usr/sbin/tomcat-sysd to
start the service.. this file doesn't exist, only /usr/sbin/tomcat (perhaps
it got changed in a more recent version of the tomcat rpm that is in the
CentOS 7 repos...
Anyway.. the current tomcat systemd script is
calling /usr/libexec/tomcat/server and not any of those.
If i create a symlink from /usr/sbin/tomcat-sysd to /usr/sbin/tomcat it's
still missing the $NAME to be able to start the service.. though i can
start it manually if I do "export NAME=cloudstack-management"
Another thing i spotted in the logs is this:
DEBUG:root:execute:chkconfig --del tomcat6
DEBUG:root:Failed to execute:error reading information on service tomcat6:
No such file or directory
Which causes the cloudstack-setup-management to fail on the first run,
after i put in the missing files.. on the second run it's fine, but the
tomcat service is still registered in systemd ;)
Some more issues are occurring if you uninstall some packages (nothing too
nasty, but still worth looking into) and specifically i had problems with
reinstalling the cloudstack-management package... i didn't look much into
that yet, but it may be related to the server not getting stopped
automatically when i removed the packages... i needed do "rm -rf
/usr/share/cloudstack-management/*" before i could reinstall.
I ran into a couple more issues, but these are the ones i remember off the
top of my head or that i made some notes about.
I'll patch it myself and submit the code for review as soon as i have some
more time, perhaps still later today :)
Rafael
On Sun, Apr 26, 2015 at 4:11 PM, Remi Bergsma wrote:
> Hi Rafael,
>
> Welcome to CloudStack and thanks for your mail.
>
> What RPM packages did you use for your testing?
> From my personal experience I know that the CentOS7 (el7) packages for
> CloudStack 4.4 work just fine as a KVM hypervisor (on CentOS 7.1). In 4.5
> work is done to also make it work as a Management server. So, what exact
> problems do you run into when installing the RPM’s on CentOS 7.1?
>
> As far as I know, the centos7-rpm branch is old and no longer maintained.
> I’d recommend using the 4.5 and/or master branches instead, as these are
> most up to date and actively maintained.
>
> Feel free to send a PR to fix what you think is broken.
>
> Regards,
> Remi
>
>
> > On 26 Apr 2015, at 15:30 , Rafael Fonseca wrote:
> >
> > Hi all,
> >
> > I'm new to the cloudstack community, but i'm a seasoned developer and
> > systems engineer and i'm sure i'll be able to help out a lot :)
> >
> > I've just started on getting cloudstack running on Centos 7.1 and noticed
> > many problems, which make it NOT work out of the box.
> >
> > I'll be doing a series of fixes and submitting the code for review, since
> > i'm not a commiter yet, if no one else is working on this.
> >
> > I've seen that there's a branch called centos7-rpm which is already
> merged
> > with master, but there hasn't been any activity on that on the last six
> > months, so i guess it's currently stale.
> >
> > Perhaps some of these things that i spotted were working on 7.0, but i
> > haven't tested it out on that version yet.. ideally we can make specs
> that
> > work on ANY 7.x version and not have a spec file for every minor release.
> >
> > One of the things i immediately noticed (that is present in all
> > centos/fedora specs, not just centos7) is the following:
> >
> > %define __os_install_post %{nil}
> > %global debug_package %{nil}
> >
> > # DISABLE the post-percentinstall java repacking and line number
> stripping
> > # we need to find a way to just disable the java repacking and line
> number
> > stripping, but not the autodeps
> >
> >
> > It's easy enough to get the __os_install_post to only do that, but it
> > shouldn't be disabling autodeps.. rather the rpmbuild's dependency
> routines
> > can't find any required external libs in the java or python files :)
> >
> > Replace:
> >
> > %define __os_install_post %{nil}
> >
> > with:
> >
> > %define __jar_repack %{nil}
> > %define __debug_package %{nil}
> >
> > That will do it, as checked in "rpm --showrc", although there won't be
> much
> > to be gained with it.
> >
> > -14: __os_install_post
> >/usr/lib/rpm/redhat/brp-compress
> >%{!?__debu
Re: ACS 4.6 idea Vision
There's already been some work on OpenDaylight, and it has been merged but I don't think many people are using it. It should be roughly analogous to the Nuage plugin's functionality, where openvswitch on the hypervisor talks to a controller that provides all of the traffic flow rules. It might not be complete. A cursory look at Cloudrouter seems to indicate that it might be OpenDaylight in a pretty package. Perhaps it can drop in as the controller under the existing OpenDaylight code, or the OpenDaylight code can be extended. On Sat, Apr 25, 2015 at 9:44 PM, ilya wrote: > This definitely warrants a closer look... thanks for bringing this up. > > > On 4/24/15 7:18 AM, Keerthiraja SJ wrote: >> >> Hi All, >> >> This is the email about why can we merge the cloudrouter.org in cloudstack >> in the new 4.6. >> >> >> I basic idea is in Advance Networking when we choose VLAN isolatation why >> do we need more router. >> >> Can we do something like same what Nuage Network Provides. >> >> In Advance Networking configuration of Guest VLAN isolation we create many >> VM it creates many routers where this should be avoided and ACS should >> come >> up with good solution on networking part than depending on creating VLAN >> on >> switch. >> >> With reference to below link >> >> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/NuageVsp+Network+Plugin >> >> All the network isolation and routing is handled by the Nuage SDN >> solution. >> Routing of the packets happen at hypervisor itself. Virtual Router is not >> used for routing the packets thus the current issue of the Virtual Router >> being the bottleneck is avoided. The Virtual Router VM is used only for >> Password reset functionality. The Nuage solution also helps significantly >> improve the agility and scale of a CS deployment as compared to a Virtual >> Router based deployment >> >> Why CS can have the same inbuilt features than of Nuage. >> >> For example : We can set two routers if we need we should have options to >> create backup server to redundancy. >> >> Can we merge something with cloudrouter to make ACS better. >> >> >> Thanks, >> Keerthi >> >> Live for stack :) cloudstack. >> >
[GitHub] cloudstack pull request:
Github user karuturi commented on the pull request: https://github.com/apache/cloudstack/commit/79a46fe0b4982e812d6a6ed0f08d1fe92b0b5e8d#commitcomment-10908911 In api/src/org/apache/cloudstack/api/command/user/iso/UpdateIsoCmd.java: In api/src/org/apache/cloudstack/api/command/user/iso/UpdateIsoCmd.java on line 43: can you return false here? findbugs reported a new issue 'NULL Pointer Dereference' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: fix CLOUDSTACK-8402
Github user karuturi commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/192#discussion_r29119746 --- Diff: packaging/centos63/cloud.spec --- @@ -113,7 +113,7 @@ The Apache CloudStack files shared between agent and management server %package agent Summary: CloudStack Agent for KVM hypervisors Requires: openssh-clients -Requires: java => 1.7.0 +Requires: java-1.7.0-openjdk --- End diff -- Can you also make the change for usage package on L154? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: fix CLOUDSTACK-8402
Github user giraffeforestg commented on the pull request: https://github.com/apache/cloudstack/pull/192#issuecomment-96506719 Okay. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-8304: Modify tags accordingly ...
GitHub user gauravaradhye opened a pull request: https://github.com/apache/cloudstack/pull/193 CLOUDSTACK-8304: Modify tags accordingly for tests which can't be run on simulator The test cases try to download ISO, should not be run on simulator. required_hardware tag value is easy identification of whether a test should be run on simulator or not (whether it requires hardware or not). You can merge this pull request into a Git repository by running: $ git pull https://github.com/gauravaradhye/cloudstack 8403 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/193.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #193 commit 560ab54d71395f9d6b5a1bfdaaa339fe4bb5e185 Author: Gaurav Aradhye Date: 2015-04-27T05:33:17Z CLOUDSTACK-8304: Modify tags accordingly for tests which can't be run on simulator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-8304: Modify tags accordingly ...
Github user asfgit closed the pull request at: https://github.com/apache/cloudstack/pull/193 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Fwd: ACS 4.5.1 fails
Hi All, I installed the ACS in my VirtualBox. After I install the CS with 4GB base memory I could not able to start the server. I scale the memory upto 12GB still I could not able to start the ACS. In /var/log/cloudstack/management/catalina.out I could see error as Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Can't start up: not enough memory Is I missing anything. In the default file of below I could see the java value as default. tomcat6.conf -> /etc/cloudstack/management/tomcat6-nonssl.conf JAVA_OPTS="-Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M -XX:MaxPermSize=800m -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers" Kindly let me know if any one found this issue. I even tried by increasing the -Xmx2g into 4g still I don't see any improvement. Thanks, Keerthi
