Hi all,
I accidentally spent last day looking into some timings of kolla-build.
I'm sharing my results below - perhaps it will help somebody:
I've measured time for 4 consecutive kolla-builds (on a decent laptop,
with modified default profile including
"kolla-toolbox,glance,horizon,keystone,mariadb,memchached,neutron,nova,rabbitmq,chronos,marathon,mesos-master,mesos-slave,zookeeper",
-b ubuntu -t binary):
real 25m20.867s (fresh environment, all Docker images wiped out)
real 22m54.298s (2nd run, all Docker images from previous run in place)
real 25m52.602s (3nd run, all Docker images from previous run in
place, local apt-cacher-ng enabled for the 1st time and stored 300MB of
DEBs)
real 18m28.739s (4th run, all DEBs were collected from apt-cacher =
local disk)
To compare, kolla-build run on a server (with --threads 6 option) takes
about 15 minutes.
I've tweaked mirrors for Ubuntu but I haven't observed any substantial
difference.
BTW it may not be a good idea to start >1 concurrent builds on 1 Jenkins
slave (due to high load).
CONCLUSIONS:
* kolla-build script (or tools it's using, e.g. Docker registry) doesn't
seem to be making good usage of Docker cache mechanisms (or they are
broken themselves) and I think is should be investigated/fixed
* having DEBs close or in cache seems like a promising approach that
never hurts: IMO we should deploy it and start saving time & bandwidth
* stable bandwidth of roughly 30 Mbit/s is needed to match building
Kolla locally (on a laptop or lab, comparing to CI). With bandwidth of
like 200Mbit/s developer could get images in his lab in like 17 minutes
from the moment he commited a patch. This can be further optimized with
rebuilding just changed images explicitly (argument to kolla-build) but
I investigated this & IMO it's not trivial b/c of various corner cases
if we want to provide a complete set of images for kolla-mesos-deploy
(under 1 Docker tag) - I can elaborate if you want.
* if we want to use CI for image building & deployment tests we need
strong infra with local Docker registry in it, all in 1 network (I mean
CI slaves, registry & perhaps even lab for some manual tests).
Again guys, these are just quick&dirty results but I kind of sense "tip
of an iceberg" problem here.
-marek
On 23.02.2016 17:58, Steven Dake (stdake) wrote:
Ricardo,
Apologies in lag, I remember reading your email but not answering
personally. I thought Michal's response was appropriate but there is a
bit more to it.
We definitely want to have images for the following events:
* Any tag such as a milestone, or release candidate, or release
* Nightly builds for folks that want to run bleeding edge without the pain
of building images
The nightly builds can easily be culled after 30 days.
We have implementations for centos, ubuntu, and oracle linux. We have
source and binary versions of both these distros, so essentially we have 6
sets of registry images.
There are 115 containers we would build, push, and tag. I think Michal's
estimate is for a limited subset of the containers. Ideally we would want
to build and push all the containers in our repository, which I believe is
10-15GB per distro+build type combination.
At the high end, we are looking at
* (for permanently tagged releases) 15gb*6 sets of images * (3 milestone
tags + 1 release tag + 1-3 RC's per cycle = 7) * 2 release cycles per year
= 1 terabyte of images per cycle growing approximately ~ 1.3 terabytes per
year
* (for the nightly builds), 15bb*6 sets of images * 30 = 2.7 terabytes in
continuous use.
So that is 2.2 TB baseline + 1TB per year growth. As we add containers
over time, the yearly growth may increase, but I doubt would ever be more
then 2 terabytes in the next 4 release cycles of OpenStack.
There are several registry storage backends available including swift, so
if infra has swift available, that would be a viable option.
I'll get an exact number for our containers as they build today and
respond to this thread since it affects the estimate.
It is not critical the storage be SSD, since tag and nightly build
operations could happen with long run-time on the post jobs I believe
without harming infra resources (although they may go past the 90 minute
limit infra likes to stick to without SSD). I don't have any non-SSD to
test the build with, so I have no idea what performance impact the SSDs
have. I know when I went from regular SSD to PCI-E NVM SSD (Intel 750) my
build times dropped from 50 minutes to about 15 minutes.
Note we don't have gate jobs at present for oracle linux nor Ubuntu
binary. Only CentOS binary, CentOS source, and Ubuntu source are in our
gates at present. For the present term, storage would be 630GB per year.
I'll get back to you with exact numbers on storage used by the registry in
a full build scenario today.
Regards
-steve
On 2/21/16, 9:26 AM, "Michał Jastrzębski" <inc...@gmail.com> wrote:
I'd say 5Gigs should be enough for all the images per distro (maybe
less if we have to squeeze). Since we have have 2 strongly supported
distro 10Gigs. If we would like to add all distros we support, that's
20-25 (I think). That also depends how many older versions we want to
keep (current+stable would be absolute minimum, we might increase it
to milestones). We have lot's of options to tweak so no one will get
hurt, and if we have dedicated machine for us (which we should because
apart from disk space, registry can actually eat up lots of IOPS, can
be VM tho with disk that can handle that), I think any dedicated,
industry standard, disk should be enough (but SSD would be great).
Cheers,
Michal
On 20 February 2016 at 16:14, Ricardo Carrillo Cruz
<ricardo.carrillo.c...@gmail.com> wrote:
Hi Steve
When you say the registry would require a machine with plenty of disk
space,
do you have an estimate of storage needed?
Regards
2016-02-20 14:21 GMT+01:00 Steven Dake (stdake) <std...@cisco.com>:
Infra folks,
I'd like to see a full CI/CD pipeline of Kolla to an OpenStack
infrastructure hosted registry.
With docker registry 2.2 and earlier a Docker push of Kolla containers
took 5-10 hours. This is because of design problems in Docker which
made a
push each layer of each Docker image repeatedly. This has been
rectified in
docker-regitery 2.3 (the latest hub tagged docker registry). The 5-10
hour
upload times are now down to about 15 minutes. Now it takes
approximately
15 minutes to push all 115 kolla containers on a gigabit network.
Kolla in general wants to publish to a docker registry at least per
tag,
and possibly per commit (or alternatively daily). We already build
Kolla
images in the gate, and although sometimes our jobs time out on CentOS
the
build on Ubuntu is about 12 minutes. The reason our jobs time out on
CentOS
is because we lack local to the infrastructure mirrors as is available
on
Ubuntu from a recent patch I believe that Monty offered.
We have one of two options going forward
We could publish to the docker hub registry
We could publish to docker-registry.openstack.org
Having a docker-registry.openstack.org would be my preference, but
requires a machine with plenty of disk space and a copy of docker
1.10.1 or
later running on it. The docker-registry 2.3 and later runs as a
container
inside Docker. The machine could be Ubuntu or CentOS - we have gate
scripts
for both that do the machine setup which the infrastructure team could
begin
with[1][2] I don't care which distro is used for docker registry - it
reallly shouldn't matter as it will be super lightweight and really
only
need a /var/lib/docker that is fast and large. Kolla dev's can help
get the
docker registry setup and provide guidance to the infrastructure team
on how
to setup Docker, but I'm unclear of OpenStack has resources to make
this
particular request happen.
NB the machine need not be baremetal - it really doesn't matter. It
does
need fast bi-directional networking and fast disk IO to meet the gate
timeout requirements and Operator requirements that a pull is speedy.
The
other change needed is a CentOS mirror internal to the infrastructure,
so
our CentOS jobs don't time out and we can push per cmmit (or we could
add a
nightly job).
This is something new OpenStack hasn't done before, so feedback from
the
infrastructure team welcome if that team is willing to maintain a
docker-registry.openstack.org. The other challenge here will be
authentication - we setup our gate Docker without TLS because we throw
away
the VMs but infra will want to setup TLS with the docker registry.
Folks
wanting to use the docker reigstry service from OpenStack will need to
be
able to put TLS credentials in the gating in some way. I'm not sure
we want
to just check these credentials into our repository - which means they
need
to somehow be injected into our VMs to protect the security of the
Docker
images.
If infra decides they don't want to take on a
docker-registry.openstack.org, guidance on how to get our credentials
securely into our built VM would be helpful.
One final note - Docker can be setup to use Swift as a storage
backend, or
alternatively can use straight up disk space on the node. It can also
publish to an AWS storage backend and has many other storage backend
modes.
Regards
-steve
[1]
https://github.com/openstack/kolla/blob/master/tools/setup_RedHat.sh
[2]
https://github.com/openstack/kolla/blob/master/tools/setup_Debian.sh
________________________________________________________________________
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_________________________________________________________________________
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Marek Zawadzki
Mirantis Kolla Team
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
