Dear Ed,
On Wed, 2017-11-29 at 18:57 +0000, Ed Kern (ejk) wrote:
>
>
>
> > On Nov 29, 2017, at 3:09 AM, Marco Varlese <mvarl...@suse.de> wrote:
> >
> >
> >
> > Hi Ed,
> >
> >
> >
> >
> >
> > On Wed, 2017-11-29 at 03:24 +0000, Ed Kern (ejk) wrote:
> > >
> > >
> > > All the jobs that ive looked at vpp verify’s are still set to ‘single use
> > > slave’ So there is no re-use..
> > >
> > >
> > >
> > > if the job is run by jenkins and is either ubuntu or centos it will
> > > attempt to pull and install
> > > ubuntu:
> > > vpp-dpdk-dev
> > > vpp-dpdk-dkms
> >
> >
> >
> >
> >
> > I suppose that's exactly my point of "not-clean-state”.
> >
>
>
>
>
> Just to be clear…
>
>
>
>
>
> Are you unhappy that it is doing those package installs before running the
> make verify (or build.sh in the case of opensuse)
>
> Or just because you think those package installs are breaking the build.
Maybe "unhappy" is a little too strong :) :) :)
I feel that being DPDK such an important piece in the VPP infrastructure I would
think it should be built during the "VPP build process" rather than having the
DPDK devel packages installed beforehand. At the end of the day, weeven have the
whole infrastructure in place to built it and link the built DPDK in VPP so why
don't we use it?
As I just said, this is my view and not trying to enforce it!
>
>
>
>
>
> >
> > It will attempt to install but since it finds those packages already
> > installed then obviously it doesn't keep going.
> >
>
>
>
>
> well no…well I certainly should have kept going (you trimmed the log you sent
> to not include the failure or a link to a specific
>
> job.
Fair enough; my bad...
>
> >
> > I was trying to ask and understand how is it possible that on a "fresh
> > booted" VM there're already packages installed.
> >
>
>
>
>
> Well when either the openstack or container image are ‘booted’ neither dpdk or
> vpp are installed.
>
> 32k other prereqs and base packages but (tmk with openstack clone) not dpdk.
Sure and that's understood; I mistakenly used "fresh booted" here...
>
>
>
>
>
>
>
>
>
> >
> > I suppose that what the message "Up-to-date DPDK package already installed"
> > points out, am I correct?
> >
>
>
>
>
> again i want to be careful we are on the same page..
>
>
>
>
>
> DPDK is not installed in the base template image.
>
>
>
>
>
> DPDK IS now installed as a package (to ubuntu and centos):
>
> AFTER check style
>
> BEFORE make verify or build.sh
>
> as part of the jenkins build scripts.
Yes, indeed. We're on the same page.
What I was trying to propose is not to install DPDK as a package (ubuntu /
centos) but leave everything to the Makefile/Build infrastructure in VPP to do
those steps.That will also help us to strengthen the VPP build-infrastructure
and spot any regression when/if modifications are made to the various
Makefiles...
>
>
>
>
> >
> > Similarly, the DPDK tarball is already downloaded when the 'curl' command
> > runs since the source tarball can already be found in /dpdk
> >
> >
> >
> >
> >
>
>
>
> So from the base template we go from no directory
> template base image is spun up
> git clone/fetch/etc is run
> check style is run
> there should be no dpdk.xxx.tar.xz anywhere at this point
> make verify or build.sh (im only speaking about verify builds ) is run
> as part of that ( i myself trip the download by doing make config in the dpdk
> directory, never bothered to track down how it is tripped ‘normally’)
> the tar.xz is pulled.
>
>
>
> if you actually see an order different then this id be curious to see it.
>
>
>
>
>
> thanks,
>
>
>
>
>
> Ed
Cheers,Marco
>
>
>
> > >
> > >
> > > centos:
> > > vpp-dpdk-devel
> > >
> > >
> > >
> > > if running the build of opensuse or any build outside of jenkins it will
> > > attempt to build it from scratch..
> >
> > Right, I believe that should always be the approach tho...
> >
> >
> >
> >
> > > (thats one of the issues you were seeing the other day)
> > >
> > >
> > >
> > > https://gerrit.fd.io/r/#/c/9606/
> > >
> > >
> > >
> > > both that dpdk-17.08 rev’d up and also the fact that they added stable to
> > > the directory name..
> > > and to make matters worse (only because their mirrors are hosed) the
> > > makefile was pointed at fast.dpdk.org
> > > which points to three servers that return at least two different
> > > cksums…(So a total of 3 different a. pre 11/27 b. two post 11/27)
> > > in my patch above i just changed it to static.dpdk.org which is slower but
> > > consistent.
> >
> > Great... we could have modified my patch since yours looks pretty similar...
> > anyway, I abonded mine with the open yours gets merged sooner.
> >
> >
> >
> >
> > >
> > >
> > > Note: ignore the cpoc failures…im still bumping into oom condition waiting
> > > on vanessa to come back and bump me up
> > > https://rt.linuxfoundation.org/Ticket/Display.html?id=48884
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On Nov 28, 2017, at 7:58 AM, Marco Varlese <mvarl...@suse.de> wrote:
> > > >
> > > > To add on the reported issue below, similarly, (I think that) the DPDK
> > > > tarball
> > > >
> > > > can be also (always) found in the workspace hence never "freshly"
> > > > downloaded
> > > >
> > > > either.
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > there is no tarball as part of vpp so its always getting freshly
> > > pulled…from one location or another..
> > >
> > >
> > > > Snipped of the logs below:
> > > >
> > > > ---
> > > >
> > > > 11:00:48 @@@@ Finding source for dpdk @@@@
> > > >
> > > > 11:00:48 @@@@ Makefile fragment found in /w/workspace/vpp-verify-master-
> > > >
> > > > ubuntu1604/build-data/packages/dpdk.mk @@@@
> > > >
> > > > 11:00:48 @@@@ Source found in /w/workspace/vpp-verify-master-
> > > > ubuntu1604/dpdk
> > > >
> > > > @@@@
> > > >
> > > > ---
> > > >
> > > >
> > > >
> > > > On Tue, 2017-11-28 at 15:49 +0100, Marco Varlese wrote:
> > > >
> > > > > All,
> > > > >
> > > > >
> > > > >
> > > > > While looking into an issue which Dave raised yesterday, I bumped into
> > > > >
> > > > > something
> > > > >
> > > > > which does not look right to me.
> > > > >
> > > > >
> > > > >
> > > > > The Jenkins jobs executing the VPP builds do not start from a clean-
> > > > > state. I
> > > > >
> > > > > would expect - for instance - not to find the DPDK package already
> > > > > installed
> > > > >
> > > > > on
> > > > >
> > > > > the target host which builds a GERRIT patch.
> > > >
> > > >
> > >
> > >
> > >
> > > well…it isn’t…its just installed post check style but before any of the
> > > other primary build script.
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > > I would basically like to see the
> > > > >
> > > > > VM boot-strapping (as per the package installed by the ci-management
> > > > > scripts)
> > > > >
> > > > > and nothing else.
> > > > >
> > > > > Anything else would be installed because of the scripts we execute to
> > > > >
> > > > > build/install VPP (as per the Jenkins jobs). Would you agree/disagree?
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > > So this is how its already happening with the open stack slaves unless
> > > something has changed.
> > >
> > >
> > >
> > > But i will say that I disagree and its not how im doing the container base
> > > images and thats around
> > > make install-dep
> > > In that every three days ill ‘bake into’ the basic build container a run
> > > of make install-dep.
> > > Even though it still does run make install-dep again at build time the
> > > actual number of packages
> > > that it has to pull in (and number of external services that have to be
> > > available) is trimmed down
> > > to as small a number as possible.
> > >
> > >
> > >
> > > This is still a problem in the ‘test’ area that likes to pip install
> > > python dependencies but are not linked
> > > to main install-dep but i wont rant about that here...
> > >
> > >
> > > >
> > > > > In the Jenkins logs I can see the below all the time:
> > > > >
> > > > > ----
> > > > >
> > > > > 11:00:48 make -C dpdk install-deb
> > > > >
> > > > > 11:00:48 make[1]: Entering directory '/w/workspace/vpp-verify-master-
> > > > >
> > > > > ubuntu1604/dpdk'
> > > > >
> > > > > 11:00:48 Building IPSec-MB 0.46 library
> > > > >
> > > > > 11:00:48 ==========================================================
> > > > >
> > > > > 11:00:48 Up-to-date DPDK package already installed
> > > > >
> > > > > 11:00:48 ==========================================================
> > > > >
> > > > > 11:00:48 make[1]: Leaving directory '/w/workspace/vpp-verify-master-
> > > > >
> > > > > ubuntu1604/dpdk'
> > > > >
> > > > > 11:00:48
> > > > >
> > > > > ----
> > > > >
> > > > >
> > > > >
> > > > > Also, I would like to ask if by any chance the tarball which VPP
> > > > > build-system
> > > > >
> > > > > requires (e.g. DPDK / IPSecMB / etc.) are somehow cached somewhere
> > > > > (either on
> > > > >
> > > > > the Jenkins-slave or Proxied) since I think the DPDK tarball
> > > > > (currently
> > > > >
> > > > > pulled)
> > > > >
> > > > > may not be coming from the real location…
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > > Wont speak for the openstack build side but I actually think it may be
> > > worth just doing something
> > > similar to the install-dep above with the dpdk… installing the binaries
> > > from nexus every few days
> > > so while it will check the nexus server for an update it normally wouldn’t
> > > have to actually pull or install.
> > > Will give that some thought…
> > >
> > >
> > >
> > > and from previous thread
> > >
> > >
> > >
> > >
> > > > Since the MD5 checksum on the DPDK tarball fails; to answer your
> > > >
> > > > question, no, it has never happened to me to see this specific issue
> > > >
> > > > before.
> > > >
> > > >
> > > >
> > > > I don't think there's anything specific to the openSUSE setup and/or
> > > >
> > > > scripts being executed. I'd rather feel is - as I said earlier -
> > > >
> > > > something to do with an hiccup on the infrastructure side. The fact
> > > >
> > > > that a 'recheck' made it passing I suppose it confirms my current
> > > >
> > > > theory.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > well opensuse still has issues ‘passing’ or not…namely no artifacts are
> > > ever generated/pushed
> > >
> > >
> > >
> > >
> > > 01:43:40 [ssh-agent] Stopped.
> > > 01:43:40 Archiving artifacts
> > > 01:43:40 WARN: No artifacts found that match the file pattern "build-
> > > root/*.rpm,build-root/*.deb,dpdk/*.rpm,dpdk/*.deb". Configuration error?
> > > 01:43:40 WARN: ?build-root/*.rpm? doesn?t match anything: ?build-root?
> > > exists but not ?build-root/*.rpm?
> > > 01:43:40 [PostBuildScript] - Execution post build scripts.
> > > 01:43:40 [vpp-verify-master-opensuse] $ /bin/bash
> > > /tmp/jenkins8435214415951107237.sh
> > > 01:43:40 Build logs: <a href="https://logs.fd.io/production/vex-yul-rot-je
> > > nkins-1/vpp-verify-master-opensuse/501">https://logs.fd.io/production/vex-
> > > yul-rot-jenkins-1/vpp-verify-master-opensuse/501</a>;
> > > 01:43:40 uname -a:
> > >
> > >
> > >
> > >
> > > So for me opensuse ‘passing’ is something akin to a banner reading
> > > ‘Mission Accomplished’
> > >
> > >
> > > Ed
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > >
> > > > --
> > > >
> > > > Marco V
> > > >
> > > >
> > > >
> > > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > > >
> > > > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> > > >
> > > > _______________________________________________
> > > >
> > > > vpp-dev mailing list
> > > >
> > > > vpp-dev@lists.fd.io
> > > >
> > > > https://lists.fd.io/mailman/listinfo/vpp-dev
> > > >
> > >
> > >
> > >
> >
> >
> > --
> >
> > Marco V
> >
> >
> >
> >
> >
> > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> >
> > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> >
>
>
>
>
>
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
