Re: [vpp-dev] Gerrit 25463 fails API CRC check for no obvious reason...

adding vpp-dev to to: A bad retry vote on a job (jan will be sending it along) caused a merge to happen with this bug. vpp folks…. Since we have gone a few months now with no jenkins connection issues (one of the main drivers for the retry introduction). Id like to propose removing the retry f

Re: [vpp-dev] Docs and test-docs jobs are unstable

is still open but shouldn’t be hitting us.) >> >> >> Ed >> >> >> >>> On Jan 22, 2020, at 8:36 AM, Ed Kern via Lists.Fd.Io >>> wrote: >>> >>> I am looking into this for the last 30 minutes or so and have ra

Re: [vpp-dev] Docs and test-docs jobs are unstable

22, 2020, at 8:36 AM, Ed Kern via Lists.Fd.Io > wrote: > > I am looking into this for the last 30 minutes or so and have raised it with > LF folks as well. > > More hopefully soon. > > Ed > > > > > >> On Jan 22, 2020, at 8:28 AM, Florin Coras w

Re: [vpp-dev] Docs and test-docs jobs are unstable

I am looking into this for the last 30 minutes or so and have raised it with LF folks as well. More hopefully soon. Ed > On Jan 22, 2020, at 8:28 AM, Florin Coras wrote: > > Hi, > > All verify jobs are now failing because of these. “lftools command not found” > seems to be the issue bu

Re: [vpp-dev] Arm verify job broken

>> wrote: better to debug some before changing anything. With my RM hat on I would rather understand what is going on, if it is 24 hours before the RC1 milestone ;-) --a On 14 Jan 2020, at 21:28, Ed Kern via Lists.Fd.Io mailto:ejk=cisco@lists.fd.io>> wrote:  ok FOR THE MOST PART…

Re: [vpp-dev] Arm verify job broken

ok FOR THE MOST PART… build times haven’t changed (@20 min) make test portion has gone off the chain and is often hitting the 120 min build timeout threshold. Ive currently got several jobs running on sandbox testing a. without any timeouts (just to see if we are still passing on master) b. t

Re: [vpp-dev] Arm verify job broken

looking into it now….. its a strange one ill tell you that up front…failures are all over the place inside the build and even some hitting the 120 min timeout…. more as i dig hopefully thanks for the ping Ed On Jan 14, 2020, at 11:40 AM, Florin Coras mailto:fcoras.li...@gmail.com>> wrote:

[vpp-dev] merge jobs failing..

Hey benoit, Happened to notice (because I was watching the queue anyway since santa damjan did a bunch of merges) that ubuntu test/merges started failing after https://gerrit.fd.io/r/c/vpp/+/23913 No its not clear to me why it verified… it looks like a simple typo where you meant to do from sca

Re: [vpp-dev] May see some job restarts/retries for the next couple of hours

16, 2019, at 11:13 AM, Ed Kern via Lists.Fd.Io > wrote: > > > TLDR: some infra and arm job (especially) may see some restarts or rechecks > needed for the next couple of hours > > > Ill hopefully be collapsing the arm and and csit clusters into the primary > cluster

[vpp-dev] May see some job restarts/retries for the next couple of hours

TLDR: some infra and arm job (especially) may see some restarts or rechecks needed for the next couple of hours Ill hopefully be collapsing the arm and and csit clusters into the primary cluster so some random behavior may be seen. This is especially true when nodes are removed from one cluste

Re: [vpp-dev] Coverity run FAILED as of 2019-12-07 14:01:55 UTC

Hmm…website must have been briefly unavailable during the run this morning…seems ok now though after manual sandbox run. Ed Coverity run failed today. Current number of outstanding issues are 2 Newly detected: 0 Eliminated: 0 More details can be found at https://scan.coverity.com/projects/

[vpp-dev] xenial (ubuntu 16.04) goes 'historic' ubuntu 18.04 loses beta tag

xenial ubuntu 16.04: As discussed in a couple vpp meetings I have pulled the job that builds ubuntu 16.04 from master verify/merges. It will continue to verify/merge on older releases until we decide otherwise or the support of the whole release ends. bionic ubuntu 18.04: When we first started

Re: [vpp-dev] current verify issues and jenkins build queue

receiving but without a fully functional brain behind it. Ed > On Oct 8, 2019, at 9:59 AM, Ed Kern via Lists.Fd.Io > wrote: > > > Problems currently still ongoing: > 1device cluster worker nodes are currently down.. I’ve notified csit in > slack and am cc’in

[vpp-dev] current verify issues and jenkins build queue

Problems currently still ongoing: 1device cluster worker nodes are currently down.. I’ve notified csit in slack and am cc’ing them here.. In the meantime I have a gerrit to remove 1device per patch so it doesn’t delay voting on verify jobs. Jenkins just crashed

[vpp-dev] jenkins queue...

super short version right now… yes the queue was stuck…..it is now unstuck BUT….do expect the queue to go UP a good deal before it goes down.. (due to the nature of one stuck check style job launches 6+ jobs when it gets unstuck and finishes) a lot more details coming but didn’t want furthe

Re: [vpp-dev] FD.io Jenkins - long queue of jobs

This has nothing to do with the LF (outside of jobs getting stuck (which I haven’t seen since the upgrade yesterday)). The queue has been ballooning and shrinking the last couple of days for a few reasons: a. a very large number of 1908 verify and then merge jobs. b. increase recently in the n

Re: [vpp-dev] Lots of patches failing like this... (CentOS)

Ok so looks like centos obsoleted the packages python36-devel in favor of the name python3-devel python36-pip in favor of the name python3-pip So all the centos builds barfed because it had the ‘old’ name in required list https://gerrit.fd.io/r/c/vpp/+/22109 is verifying and the mighty wallace

Re: [vpp-dev] jenkins jobs failing timing out trying to reach gerrit

Just a heads up that this outage looks like network issues between vexxhost and gerrit servers and lasted <30 minutes. Looks like many of the failed jobs passed in the retry. We are now full steam ahead into the maint window. Ed On Jul 30, 2019, at 9:12 AM, Ed Kern via Lists.Fd

[vpp-dev] jenkins jobs failing timing out trying to reach gerrit

https://jira.linuxfoundation.org/servicedesk/customer/portal/4/SUPPORT-321 vanessa is actively looking into this presently. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13616): https://lists.fd.io/g/vpp-dev/message/13616 Mute This Topic:

Re: [vpp-dev] centos jobs failing

https://gerrit.fd.io/r/#/c/20442/6/build/external/Makefile I was too chicken to suggest it as a clean fix :) Ed On Jul 1, 2019, at 5:17 PM, Dave Barach (dbarach) mailto:dbar...@cisco.com>> wrote: See https://gerrit.fd.io/r/#/c/20446/ - .../build/external/Makefile needed a bit of “|| true”

Re: [vpp-dev] centos jobs failing

adding thomas directly just to cover my centos base… TLDR vpp-ext-deps makefile issues lead to failures resulting in immediate job failure…..working on workaround while someone comes up with a real fix. Packagecloud.io retention on master APPEARS to be about a mont

[vpp-dev] vpp queue backup

transient latency spike caused the build queue to spike with some jobs left hanging. I cleaned up the turds it left behind and the queue appears to be on its way back down (in the 20’s down from the 70’s) just an fyi.. Ed-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this g

[vpp-dev] gcc8 test runner

Apologies on not sending this update earlier… As an AI from the last vpp call I added the job vpp-verify-gcc8-master-ubuntu1804 to the verify set on 5/29 and it has been running ever since. On all the failures I have tracked they also failed on gcc7 path. At this point I am in full agreement

[vpp-dev] x86 ubuntu bionic gcc repointed from gcc-8 to gcc7

While debugging both a csit compilation problem and also some recent failures in bionic verify and merge jobs I have re-pointed gcc from gcc8 back down to gcc7 to see if that mitigates the problem. If it does not I will be rebuilding the X86 ubuntu image without gcc-8 (and its required package

[vpp-dev] gcc8 now default on all bionic builds x86 and arm

Just to close the AI from the vpp call this morning I’ve upgraded (and made default) the gcc used for vpp builds to gcc8. Don’t expect any issue and last few builds with the new image has been fine. So hopefully this is just an FYI that folks can happily ignore.. Ed -=-=-=-=-=-=-=-=-=-=-=- L

Re: [vpp-dev] arm builds failing..and not running

ak Das From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Ed Kern via Lists.Fd.Io Sent: Friday, May 10, 2019 1:48 PM To: vpp-dev mailto:vpp-dev@lists.fd.io>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vp

Re: [vpp-dev] arm builds failing..and not running

without too much trouble. Ill continue to watch throughout the weekend to make sure the latency problem doesn’t pop back up. thanks, Ed PS thanks for dave wallace for the extra set of eyes and thoughts during the debug On May 10, 2019, at 9:07 AM, Ed Kern via Lists.Fd.Io mailto:ejk=cisco

[vpp-dev] arm builds failing..and not running

Two different issues currently with arm builds…. there is a failure to launch due to JNLP connect that has cropped up in the last day. But when the run does actually start we are getting these failures on master… 09:02:13 [144/1770] Building C object vlib/CMakeFiles/vlib_thunderx2t99.dir/pun

Re: [vpp-dev] centos ninja failures

… Ed --Tom On 05/08/2019 10:25 AM, Ed Kern via Lists.Fd.Io wrote: starting new thread because it really has nothing to do with previous thread… So I am not going to write: a. its related to your code b. its not the build infra But I will say there have been good centos verify runs… BEFORE your p

[vpp-dev] centos ninja failures

rn (ejk) ; Klement Sekera -X (ksekera - PANTHEON > TECHNOLOGIES at Cisco) > Cc: vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] false verified + 1 for gerrit jobs > >> reduce the number of retries (or possibly completely) > > +1 for completely. > > Vratko. > > -Orig

Re: [vpp-dev] false verified + 1 for gerrit jobs

This is a known issue with how the retry mechanic interacts (badly) with gerrit occasionally. This odds of this happening were a bit higher over the last couple of days specific to centos retries. This is tied to the JNLP changes made to csit and how the initial connection is made. While the c

Re: [vpp-dev] jenkins restart for jnlp csit errors..

jenkins restart complete…will be watching jobs the next few hours to make sure the ha-proxy bypass doesn’t cause other issues.. have a nice day… Ed > On May 3, 2019, at 10:54 AM, Ed Kern via Lists.Fd.Io > wrote: > > jenkins has been in shutdown mode for over an our..im going

[vpp-dev] jenkins restart for jnlp csit errors..

jenkins has been in shutdown mode for over an our..im going to hand kill remaining jobs so we can get this restart done. FYI :) Ed-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12922): https://lists.fd.io/g/vpp-dev/message/12922 Mute This Topic

Re: [vpp-dev] TESTING in progress: May see job failures over the next 2 hours

done testing….depending on how the csit jobs run (or stop) this evening we may have to restart jenkins friday am to have a jenkins configuration change become active. Ill follow up tomorrow am. Ed > On May 2, 2019, at 12:06 PM, Ed Kern via Lists.Fd.Io > wrote: > > testing i

Re: [vpp-dev] TESTING in progress: May see job failures over the next 2 hours

testing is continuing on and off but to this point I don’t think we have seen job failures only jobs occasionally queuing …. debug continues Ed > On May 2, 2019, at 10:19 AM, Ed Kern via Lists.Fd.Io > wrote: > > > In an attempt to work around the LF HA-proxy to

[vpp-dev] TESTING in progress: May see job failures over the next 2 hours

In an attempt to work around the LF HA-proxy to debug csit job connectivity issues I have to ‘hardwire’ connections (or attempt to do so at any rate) directly back to the jenkins IP. No real idea if it will work or if there will be any side impact…ill be hand cranking a job or two in an atte

Re: [vpp-dev] VPP 19.04 centos7 RPM package dependency errors

On Apr 11, 2019, at 2:22 PM, Florin Coras mailto:fcoras.li...@gmail.com>> wrote: Are we building rpms on a host that has mbedtls installed? Yes the centos container has mbedtls installed….has for quite some time since it is a listed prerequisite straight out of the Makefile. Ed Just unin

Re: [vpp-dev] jenkins maven failure

i looked into this and see nothing that leads me believe this is a build infra issue. vanessa is looking into this now.. On Apr 10, 2019, at 7:50 AM, Dave Wallace mailto:dwallac...@gmail.com>> wrote: Ben, Have you opened a helpd...@fd.io ticket for this issue yet? If

[vpp-dev] FYI arm cluster testing

As of a couple minutes ago I enabled eligibility on the new arm cluster build nodes and disabled eligibility on the old nodes (to force the new ones to be used). This is currently just a test that ill be watching but if you see abnormal arm failures please let me know soon as you can. If thing

[vpp-dev] openjdk 11 now installed in xenial/bionic/centos build containers

While Its set to continue to use openjdk 8 by default I wanted folks to be aware in case any side effects or potential issues crop up. I would prefer to only have one installed (for container size if nothing else) does anyone know of any requirement on keeping openjdk 8 around and the default

Re: [vpp-dev] follow-up: Jenkins executors offline

Looks like another transient latency spike plugged up the drain… while I was trying to diagnose further magic happened (as in while it was nothing I changed or did) and it went away. While it may not appear to look like it the queue seems to be returning to normal. Ed On Mar 12, 2019, at 7:

Re: [vpp-dev] OpenSuse builds failing for stable/1901

when I saw that paul that lead to a few searches (that for me) ended with This is is not a real bug. The problem is that cmake check that it knows the version of boost that you are using and so every time that boost creates a new release, the cmake guys need to update the list of supported ver

Re: [vpp-dev] OpenSuse builds failing for stable/1901

thats a perfectly good and reasonable change and I support it.. But that’s not the problem that the other ed is having right now. Ed On Feb 27, 2019, at 12:47 PM, Paul Vinciguerra mailto:pvi...@vinciconsulting.com>> wrote: Ed, There is an issue. I am testing a fix. See: https://gerrit.fd.i

Re: [vpp-dev] OpenSuse builds failing for stable/1901

just so you know…it does already echo the version id and os id in the console log 16:47:00 + OS_VERSION_ID=15.0 16:47:00 + echo OS_ID: opensuse 16:47:00 OS_ID: opensuse 16:47:00 + echo OS_VERSION_ID: 15.0 16:47:00 OS_VERSION_ID: 15.0 16:47:00 + hostname 16:47:00 1f2fe1a3f253 16:47:00 + export CC

Re: [vpp-dev] OpenSuse builds failing for stable/1901

when i pull your change i see a different makefile then what you pasted below…. ifeq ($(SUSE_ID),15.0) RPM_SUSE_DEVEL_DEPS = libboost_headers1_68_0-devel-1.68.0 libboost_thread1_68_0-devel-1.68.0 gcc6 RPM_SUSE_PYTHON_DEPS += python2-ply python2-virtualenv else RPM_SUSE_DEVEL_DEPS + so that wil

Re: [vpp-dev] [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow

eeing the same delay on/towards other hosted services gerrit/nexus etc. Ill be watching it closely the next few hours but please let me know if you see any unusual errors. thanks, Ed On Feb 27, 2019, at 8:34 AM, Ed Kern via Lists.Fd.Io mailto:ejk=cisco@lists.fd.io>> wrote: fy

[vpp-dev] [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow

fyi to vpp-dev as well.. debugging with vexxhost Begin forwarded message: From: "Peter Mikus via Lists.Fd.Io" mailto:pmikus=cisco@lists.fd.io>> Subject: [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow Date: February 27, 2019 at 8:32:50 AM MST To: "helpd...@fd.io

Re: [vpp-dev] osleap job failing

yup…opensuse removed that specific package from the repo… you need a fix along the lines of https://gerrit.fd.io/r/#/c/17698/ or https://gerrit.fd.io/r/#/c/17800/ these are NOT infra problems. Ed > On Feb 26, 2019, at 6:30 AM, Klement Sekera via Lists.Fd.Io > wrote: > > Hello, > I'm f

[vpp-dev] opensuse15 aka opens use leap15 build failures

Due to a upstream version reporting change we saw some build failures on osleap15 verify jobs. Ive hopefully worked around this (again) so those jobs should be back on track. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12263): https://l

Re: [vpp-dev] vpp-verify-master-centos failures

hat do you think ? > would still be a manual process to change the number but sure.. If someone had a slick way to see max memory usage during any section of a ‘make ’ that would be awesome. Ed > --a > > On 2/1/19, Ed Kern via Lists.Fd.Io wrote: >> Request with n

Re: [vpp-dev] vpp-verify-master-centos failures

Request with numbers has been made. Not a ci-man change so it requires vanessa, but she is typically super fast turning around these changes, so hopefully in a couple hours. Apologies for the trouble. We have seen a 4-6x increase (depending on OS) in the last 5 months and so it finally start

Re: [vpp-dev] PEP8 expert needed

I’m sure someone beat me to it but if not.. https://gerrit.fd.io/r/17146 Ed On Jan 29, 2019, at 12:42 PM, Damjan Marion via Lists.Fd.Io mailto:dmarion=me@lists.fd.io>> wrote: Can somebody with python skills take care for this checkstyle errors, not sure why they started popping out no

[vpp-dev] vpp-verify-master-clang now bionic based

As discussed in the vpp call this morning I have moved this from from running on xenial to bionic. Ran cleanly on the sandbox…but please let me know if anyone sees issues in production. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11973)

[vpp-dev] we get something bad in https://gerrit.fd.io/r/#/c/16195/ thats toxic to bionic

While I hate…no LOATHE trying to debug build failures on days like today with gerrit misfiring and the lf upgrading everything in sight I noticed a trend thats bothering me since the subject mentioning gerrit was merged (and even that merge failed to build on bionic) we have had no good bionic

Re: [vpp-dev] Build failing on Fedora

Burt, Just an fyi and a comment… fyi on the build containers we install cmake (this predated cmakeY being in the Makefile) cmake version 2.8.12.2 cmake3 is installed as part of install dep cmake3 version 3.12.2 My build will fail (centos) without cmake3 CMake Error at CMakeLists.txt:14 (cmak

Re: [vpp-dev] API changes to trigger email

Should be able to… Not that I have the permissions to do it… following https://gerrit-review.googlesource.com/Documentation/user-notify.html#project My first thought is you would want to end up with something along the lines of [notify “API”] email = f...@project1.io em

Re: [vpp-dev] [tsc] Unstable VPP Builds

TLDR Any action that would trigger verification in the last 15 hours needs to be hit with recheck The ci-man bug mentioned below impacted the checkstyle job causing it to finish with result UNSTABLE. It takes a SUCCESS result from the checkstyle job to trigger/initiate the normal set of ver

Re: [vpp-dev] build queue backed up a bit foo

OK Cheers, Ole On 7 Sep 2018, at 18:50, Ole Troan mailto:otr...@employees.org>> wrote: Ed, Let me take a closer look at these. It appears if VPP is slow to start it might not have created the socket yet. Let me try to put in a retry loop and see if that fixes verify. Cheers Ole On 7

Re: [vpp-dev] build queue backed up a bit foo

ers Ole On 7 Sep 2018, at 18:06, Ed Kern via Lists.Fd.Io < ejk=cisco@lists.fd.io> wrote: make test failures due to the below causing pretty consistent failures. Note: For whatever reason the failures are not 100%. The failures and with the retries on nonconcurrent merge jobs may lead to l

[vpp-dev] build queue backed up a bit foo

make test failures due to the below causing pretty consistent failures. Note: For whatever reason the failures are not 100%. The failures and with the retries on nonconcurrent merge jobs may lead to long build queues. These are not infra issues but ill be keeping an eye on it. Ed 13:08

Re: [vpp-dev] vpp-arm-verify-master-ubuntu1604 failing

On Sep 6, 2018, at 3:42 PM, Matthew Smith mailto:mgsm...@netgate.com>> wrote: Hi, The jenkins job vpp-arm-verify-master-ubuntu1604 seems to have failed every time it has run over the last 36 hours or so. Is that a known issue? Yes. Its actually been failing for about a week now. Since the

Re: [vpp-dev] Ubuntu 18.04 LTS (Bionic Beaver) packages

this has been a work in progress for a couple months now with no real end in sight at this time.. You can/should be able to pick them up from packagecloud in the meantime. curl -L https://packagecloud.io/fdio/master/gpgkey |sudo apt-key add - and into a file say /etc/apt/sources.list.d/99fd.io

Re: [vpp-dev] Revert of gerrit 13408 / "vppinfra: AVX2 blend" in progress..

overed? D. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Ed Kern via Lists.Fd.Io Sent: Friday, July 13, 2018 10:11 AM To: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) mailto:vrpo...@cisco.com>> Cc: vpp-dev@li

Re: [vpp-dev] Revert of gerrit 13408 / "vppinfra: AVX2 blend" in progress..

On Jul 13, 2018, at 5:37 AM, Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via Lists.Fd.Io mailto:vrpolak=cisco@lists.fd.io>> wrote: > Not sure what exactly happened It went like this: 0. Patch set 2 is uploaded. 1. Both clang and virl jobs (among others) are triggered. 2. B

Re: [csit-dev] [vpp-dev] VPP make test gives false positive

Hey, responding because I dont see one from klement. looks like he has two gerrits pending https://gerrit.fd.io/r/#/c/13188/ to fix broken interfaces.. (this passed verification so trying to get this one merged) and https://gerrit.fd.io/r/#/c/13186/ for the retries in error..this is not pas

Re: [vpp-dev] Verify consistently failing

I think what Ed meant to do is to speedup verify job without realizing that test framework does a local install of vpp_papi package which is part of the source tree. this is correct. So having cached virtualenv is a bad idea as we see already. dont agree…caching internal bits is certainly bad

Re: [vpp-dev] vpp project patch verification broken

On Jun 19, 2018, at 7:33 AM, Ole Troan mailto:otr...@employees.org>> wrote: Dave, et al, Yes, all these are indications of the verify job running a different (and cached) version of the Python VPP package than the one in build tree. yup it certainly is... That stopped working as soon as I c