On Wed, Mar 22, 2017 at 2:20 PM, Dan Prince <dpri...@redhat.com> wrote: > On Wed, 2017-03-22 at 13:35 +0100, Flavio Percoco wrote: >> On 22/03/17 13:32 +0100, Flavio Percoco wrote: >> > On 21/03/17 23:15 -0400, Emilien Macchi wrote: >> > > Hey, >> > > >> > > I've noticed that container jobs look pretty unstable lately; to >> > > me, >> > > it sounds like a timeout: >> > > http://logs.openstack.org/19/447319/2/check-tripleo/gate-tripleo- >> > > ci-centos-7-ovb-containers-oooq-nv/bca496a/console.html#_2017-03- >> > > 22_00_08_55_358973 >> > >> > There are different hypothesis on what is going on here. Some >> > patches have >> > landed to improve the write performance on containers by using >> > hostpath mounts >> > but we think the real slowness is coming from the images download. >> > >> > This said, this is still under investigation and the containers >> > squad will >> > report back as soon as there are new findings. >> >> Also, to be more precise, Martin André is looking into this. He also >> fixed the >> gate in the last 2 weeks. > > I spoke w/ Martin on IRC. He seems to think this is the cause of some > of the failures: > > http://logs.openstack.org/32/446432/1/check-tripleo/gate-tripleo-ci-cen > tos-7-ovb-containers-oooq-nv/543bc80/logs/oooq/overcloud-controller- > 0/var/log/extra/docker/containers/heat_engine/log/heat/heat- > engine.log.txt.gz#_2017-03-21_20_26_29_697 > > > Looks like Heat isn't able to create Nova instances in the overcloud > due to "Host 'overcloud-novacompute-0' is not mapped to any cell'. This > means our cells initialization code for containers may not be quite > right... or there is a race somewhere.
Here are some findings. I've looked at time measures from CI for https://review.openstack.org/#/c/448533/ which provided the most recent results: * gate-tripleo-ci-centos-7-ovb-ha [1] undercloud install: 23 overcloud deploy: 72 total time: 125 * gate-tripleo-ci-centos-7-ovb-nonha [2] undercloud install: 25 overcloud deploy: 48 total time: 122 * gate-tripleo-ci-centos-7-ovb-updates [3] undercloud install: 24 overcloud deploy: 57 total time: 152 * gate-tripleo-ci-centos-7-ovb-containers-oooq-nv [4] undercloud install: 28 overcloud deploy: 48 total time: 165 (timeout) Looking at the undercloud & overcloud install times, the most task consuming tasks, the containers job isn't doing that bad compared to other OVB jobs. But looking closer I could see that: - the containers job pulls docker images from dockerhub, this process takes roughly 18 min. - the overcloud validate task takes 10 min more than it should because of the bug Dan mentioned (a fix is in the queue at https://review.openstack.org/#/c/448575/) - the postci takes a long time with quickstart, 13 min (4 min alone spent on docker log collection) whereas it takes only 3 min when using tripleo.sh Adding all these numbers, we're at about 40 min of additional time for oooq containers job which is enough to cross the CI job limit. There is certainly a lot of room for optimization here and there and I'll explore how we can speed up the containers CI job over the next weeks. Martin [1] http://logs.openstack.org/33/448533/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d2c1b16/ [2] http://logs.openstack.org/33/448533/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/d6df760/ [3] http://logs.openstack.org/33/448533/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/3b1f795/ [4] http://logs.openstack.org/33/448533/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/b816f20/ > Dan > >> >> Flavio >> >> >> >> _____________________________________________________________________ >> _____ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs >> cribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
