Re: [vpp-dev] Gerrit 25463 fails API CRC check for no obvious reason...

2020-02-26 Thread Ed Kern via Lists.Fd.Io
adding vpp-dev to to: A bad retry vote on a job (jan will be sending it along) caused a merge to happen with this bug. vpp folks…. Since we have gone a few months now with no jenkins connection issues (one of the main drivers for the retry introduction). Id like to propose removing the retry f

Re: [vpp-dev] Docs and test-docs jobs are unstable

2020-01-22 Thread Ed Kern via Lists.Fd.Io
. But only because after having to triage their code after a day long breakage has me in a bad mood. Ed > On Jan 22, 2020, at 10:34 AM, Florin Coras wrote: > > Great! Thanks, Ed! > > Florin > >> On Jan 22, 2020, at 9:22 AM, Ed Kern (ejk) wrote: >> >>

Re: [vpp-dev] Docs and test-docs jobs are unstable

2020-01-22 Thread Ed Kern via Lists.Fd.Io
22, 2020, at 8:36 AM, Ed Kern via Lists.Fd.Io > wrote: > > I am looking into this for the last 30 minutes or so and have raised it with > LF folks as well. > > More hopefully soon. > > Ed > > > > > >> On Jan 22, 2020, at 8:28 AM, Florin Coras w

Re: [vpp-dev] Docs and test-docs jobs are unstable

2020-01-22 Thread Ed Kern via Lists.Fd.Io
I am looking into this for the last 30 minutes or so and have raised it with LF folks as well. More hopefully soon. Ed > On Jan 22, 2020, at 8:28 AM, Florin Coras wrote: > > Hi, > > All verify jobs are now failing because of these. “lftools command not found” > seems to be the issue bu

Re: [vpp-dev] Arm verify job broken

2020-01-14 Thread Ed Kern via Lists.Fd.Io
>> wrote: better to debug some before changing anything. With my RM hat on I would rather understand what is going on, if it is 24 hours before the RC1 milestone ;-) --a On 14 Jan 2020, at 21:28, Ed Kern via Lists.Fd.Io mailto:ejk=cisco@lists.fd.io>> wrote:  ok FOR THE MOST PART…

Re: [vpp-dev] Arm verify job broken

2020-01-14 Thread Ed Kern via Lists.Fd.Io
timeout for the time being if committers feel that is warranted to ‘keep the train rolling’ Ed On Jan 14, 2020, at 1:07 PM, Florin Coras mailto:fcoras.li...@gmail.com>> wrote: Thanks, Ed! Florin On Jan 14, 2020, at 11:53 AM, Ed Kern (ejk) mailto:e...@cisco.com>> wrote: looking

Re: [vpp-dev] Arm verify job broken

2020-01-14 Thread Ed Kern via Lists.Fd.Io
looking into it now….. its a strange one ill tell you that up front…failures are all over the place inside the build and even some hitting the 120 min timeout…. more as i dig hopefully thanks for the ping Ed On Jan 14, 2020, at 11:40 AM, Florin Coras mailto:fcoras.li...@gmail.com>> wrote:

[vpp-dev] merge jobs failing..

2019-12-17 Thread Ed Kern via Lists.Fd.Io
Hey benoit, Happened to notice (because I was watching the queue anyway since santa damjan did a bunch of merges) that ubuntu test/merges started failing after https://gerrit.fd.io/r/c/vpp/+/23913 No its not clear to me why it verified… it looks like a simple typo where you meant to do from sca

Re: [vpp-dev] May see some job restarts/retries for the next couple of hours

2019-12-16 Thread Ed Kern via Lists.Fd.Io
16, 2019, at 11:13 AM, Ed Kern via Lists.Fd.Io > wrote: > > > TLDR: some infra and arm job (especially) may see some restarts or rechecks > needed for the next couple of hours > > > Ill hopefully be collapsing the arm and and csit clusters into the primary > cluster

[vpp-dev] May see some job restarts/retries for the next couple of hours

2019-12-16 Thread Ed Kern via Lists.Fd.Io
TLDR: some infra and arm job (especially) may see some restarts or rechecks needed for the next couple of hours Ill hopefully be collapsing the arm and and csit clusters into the primary cluster so some random behavior may be seen. This is especially true when nodes are removed from one cluste

Re: [vpp-dev] Coverity run FAILED as of 2019-12-07 14:01:55 UTC

2019-12-07 Thread Ed Kern via Lists.Fd.Io
Hmm…website must have been briefly unavailable during the run this morning…seems ok now though after manual sandbox run. Ed Coverity run failed today. Current number of outstanding issues are 2 Newly detected: 0 Eliminated: 0 More details can be found at https://scan.coverity.com/projects/

[vpp-dev] xenial (ubuntu 16.04) goes 'historic' ubuntu 18.04 loses beta tag

2019-11-05 Thread Ed Kern via Lists.Fd.Io
xenial ubuntu 16.04: As discussed in a couple vpp meetings I have pulled the job that builds ubuntu 16.04 from master verify/merges. It will continue to verify/merge on older releases until we decide otherwise or the support of the whole release ends. bionic ubuntu 18.04: When we first started

Re: [vpp-dev] current verify issues and jenkins build queue

2019-10-08 Thread Ed Kern via Lists.Fd.Io
receiving but without a fully functional brain behind it. Ed > On Oct 8, 2019, at 9:59 AM, Ed Kern via Lists.Fd.Io > wrote: > > > Problems currently still ongoing: > 1device cluster worker nodes are currently down.. I’ve notified csit in > slack and am cc’in

[vpp-dev] current verify issues and jenkins build queue

2019-10-08 Thread Ed Kern via Lists.Fd.Io
Problems currently still ongoing: 1device cluster worker nodes are currently down.. I’ve notified csit in slack and am cc’ing them here.. In the meantime I have a gerrit to remove 1device per patch so it doesn’t delay voting on verify jobs. Jenkins just crashed

[vpp-dev] jenkins queue...

2019-10-08 Thread Ed Kern via Lists.Fd.Io
super short version right now… yes the queue was stuck…..it is now unstuck BUT….do expect the queue to go UP a good deal before it goes down.. (due to the nature of one stuck check style job launches 6+ jobs when it gets unstuck and finishes) a lot more details coming but didn’t want furthe

Re: [vpp-dev] FD.io Jenkins - long queue of jobs

2019-09-26 Thread Ed Kern via Lists.Fd.Io
This has nothing to do with the LF (outside of jobs getting stuck (which I haven’t seen since the upgrade yesterday)). The queue has been ballooning and shrinking the last couple of days for a few reasons: a. a very large number of 1908 verify and then merge jobs. b. increase recently in the n

Re: [vpp-dev] Lots of patches failing like this... (CentOS)

2019-09-17 Thread Ed Kern via Lists.Fd.Io
Ok so looks like centos obsoleted the packages python36-devel in favor of the name python3-devel python36-pip in favor of the name python3-pip So all the centos builds barfed because it had the ‘old’ name in required list https://gerrit.fd.io/r/c/vpp/+/22109 is verifying and the mighty wallace

Re: [vpp-dev] jenkins jobs failing timing out trying to reach gerrit

2019-07-30 Thread Ed Kern via Lists.Fd.Io
Just a heads up that this outage looks like network issues between vexxhost and gerrit servers and lasted <30 minutes. Looks like many of the failed jobs passed in the retry. We are now full steam ahead into the maint window. Ed On Jul 30, 2019, at 9:12 AM, Ed Kern via Lists.Fd

[vpp-dev] jenkins jobs failing timing out trying to reach gerrit

2019-07-30 Thread Ed Kern via Lists.Fd.Io
https://jira.linuxfoundation.org/servicedesk/customer/portal/4/SUPPORT-321 vanessa is actively looking into this presently. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13616): https://lists.fd.io/g/vpp-dev/message/13616 Mute This Topic:

Re: [vpp-dev] centos jobs failing

2019-07-01 Thread Ed Kern via Lists.Fd.Io
os7 VM. HTH... Dave From: Ed Kern (ejk) mailto:e...@cisco.com>> Sent: Monday, July 1, 2019 3:20 PM To: Florin Coras mailto:fcoras.li...@gmail.com>> Cc: vpp-dev mailto:vpp-dev@lists.fd.io>>; Dave Barach (dbarach) mailto:dbar...@cisco.com>>; Thomas F Herbert mailto:therb...@re

Re: [vpp-dev] centos jobs failing

2019-07-01 Thread Ed Kern via Lists.Fd.Io
adding thomas directly just to cover my centos base… TLDR vpp-ext-deps makefile issues lead to failures resulting in immediate job failure…..working on workaround while someone comes up with a real fix. Packagecloud.io retention on master APPEARS to be about a mont

[vpp-dev] vpp queue backup

2019-06-26 Thread Ed Kern via Lists.Fd.Io
transient latency spike caused the build queue to spike with some jobs left hanging. I cleaned up the turds it left behind and the queue appears to be on its way back down (in the 20’s down from the 70’s) just an fyi.. Ed-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this g

[vpp-dev] gcc8 test runner

2019-06-10 Thread Ed Kern via Lists.Fd.Io
Apologies on not sending this update earlier… As an AI from the last vpp call I added the job vpp-verify-gcc8-master-ubuntu1804 to the verify set on 5/29 and it has been running ever since. On all the failures I have tracked they also failed on gcc7 path. At this point I am in full agreement

[vpp-dev] x86 ubuntu bionic gcc repointed from gcc-8 to gcc7

2019-05-22 Thread Ed Kern via Lists.Fd.Io
While debugging both a csit compilation problem and also some recent failures in bionic verify and merge jobs I have re-pointed gcc from gcc8 back down to gcc7 to see if that mitigates the problem. If it does not I will be rebuilding the X86 ubuntu image without gcc-8 (and its required package

[vpp-dev] gcc8 now default on all bionic builds x86 and arm

2019-05-14 Thread Ed Kern via Lists.Fd.Io
Just to close the AI from the vpp call this morning I’ve upgraded (and made default) the gcc used for vpp builds to gcc8. Don’t expect any issue and last few builds with the new image has been fine. So hopefully this is just an FYI that folks can happily ignore.. Ed -=-=-=-=-=-=-=-=-=-=-=- L

Re: [vpp-dev] arm builds failing..and not running

2019-05-13 Thread Ed Kern via Lists.Fd.Io
ak Das From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Ed Kern via Lists.Fd.Io Sent: Friday, May 10, 2019 1:48 PM To: vpp-dev mailto:vpp-dev@lists.fd.io>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vp

Re: [vpp-dev] arm builds failing..and not running

2019-05-10 Thread Ed Kern via Lists.Fd.Io
without too much trouble. Ill continue to watch throughout the weekend to make sure the latency problem doesn’t pop back up. thanks, Ed PS thanks for dave wallace for the extra set of eyes and thoughts during the debug On May 10, 2019, at 9:07 AM, Ed Kern via Lists.Fd.Io mailto:ejk=cisco

[vpp-dev] arm builds failing..and not running

2019-05-10 Thread Ed Kern via Lists.Fd.Io
Two different issues currently with arm builds…. there is a failure to launch due to JNLP connect that has cropped up in the last day. But when the run does actually start we are getting these failures on master… 09:02:13 [144/1770] Building C object vlib/CMakeFiles/vlib_thunderx2t99.dir/pun

Re: [vpp-dev] centos ninja failures

2019-05-08 Thread Ed Kern via Lists.Fd.Io
… Ed --Tom On 05/08/2019 10:25 AM, Ed Kern via Lists.Fd.Io wrote: starting new thread because it really has nothing to do with previous thread… So I am not going to write: a. its related to your code b. its not the build infra But I will say there have been good centos verify runs… BEFORE your p

[vpp-dev] centos ninja failures

2019-05-08 Thread Ed Kern via Lists.Fd.Io
shell' marked build as failure > > > -Original Message- > From: vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] On Behalf Of Vratko > Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via Lists.Fd.Io > Sent: Tuesday, May 7, 2019 4:21 PM > To: Ed Ke

Re: [vpp-dev] false verified + 1 for gerrit jobs

2019-05-06 Thread Ed Kern via Lists.Fd.Io
This is a known issue with how the retry mechanic interacts (badly) with gerrit occasionally. This odds of this happening were a bit higher over the last couple of days specific to centos retries. This is tied to the JNLP changes made to csit and how the initial connection is made. While the c

Re: [vpp-dev] jenkins restart for jnlp csit errors..

2019-05-03 Thread Ed Kern via Lists.Fd.Io
jenkins restart complete…will be watching jobs the next few hours to make sure the ha-proxy bypass doesn’t cause other issues.. have a nice day… Ed > On May 3, 2019, at 10:54 AM, Ed Kern via Lists.Fd.Io > wrote: > > jenkins has been in shutdown mode for over an our..im going

[vpp-dev] jenkins restart for jnlp csit errors..

2019-05-03 Thread Ed Kern via Lists.Fd.Io
jenkins has been in shutdown mode for over an our..im going to hand kill remaining jobs so we can get this restart done. FYI :) Ed-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12922): https://lists.fd.io/g/vpp-dev/message/12922 Mute This Topic

Re: [vpp-dev] TESTING in progress: May see job failures over the next 2 hours

2019-05-02 Thread Ed Kern via Lists.Fd.Io
done testing….depending on how the csit jobs run (or stop) this evening we may have to restart jenkins friday am to have a jenkins configuration change become active. Ill follow up tomorrow am. Ed > On May 2, 2019, at 12:06 PM, Ed Kern via Lists.Fd.Io > wrote: > > testing i

Re: [vpp-dev] TESTING in progress: May see job failures over the next 2 hours

2019-05-02 Thread Ed Kern via Lists.Fd.Io
testing is continuing on and off but to this point I don’t think we have seen job failures only jobs occasionally queuing …. debug continues Ed > On May 2, 2019, at 10:19 AM, Ed Kern via Lists.Fd.Io > wrote: > > > In an attempt to work around the LF HA-proxy to

[vpp-dev] TESTING in progress: May see job failures over the next 2 hours

2019-05-02 Thread Ed Kern via Lists.Fd.Io
In an attempt to work around the LF HA-proxy to debug csit job connectivity issues I have to ‘hardwire’ connections (or attempt to do so at any rate) directly back to the jenkins IP. No real idea if it will work or if there will be any side impact…ill be hand cranking a job or two in an atte

Re: [vpp-dev] VPP 19.04 centos7 RPM package dependency errors

2019-04-11 Thread Ed Kern via Lists.Fd.Io
On Apr 11, 2019, at 2:22 PM, Florin Coras mailto:fcoras.li...@gmail.com>> wrote: Are we building rpms on a host that has mbedtls installed? Yes the centos container has mbedtls installed….has for quite some time since it is a listed prerequisite straight out of the Makefile. Ed Just unin

Re: [vpp-dev] jenkins maven failure

2019-04-10 Thread Ed Kern via Lists.Fd.Io
i looked into this and see nothing that leads me believe this is a build infra issue. vanessa is looking into this now.. On Apr 10, 2019, at 7:50 AM, Dave Wallace mailto:dwallac...@gmail.com>> wrote: Ben, Have you opened a helpd...@fd.io ticket for this issue yet? If

[vpp-dev] FYI arm cluster testing

2019-04-05 Thread Ed Kern via Lists.Fd.Io
As of a couple minutes ago I enabled eligibility on the new arm cluster build nodes and disabled eligibility on the old nodes (to force the new ones to be used). This is currently just a test that ill be watching but if you see abnormal arm failures please let me know soon as you can. If thing

[vpp-dev] openjdk 11 now installed in xenial/bionic/centos build containers

2019-04-01 Thread Ed Kern via Lists.Fd.Io
While Its set to continue to use openjdk 8 by default I wanted folks to be aware in case any side effects or potential issues crop up. I would prefer to only have one installed (for container size if nothing else) does anyone know of any requirement on keeping openjdk 8 around and the default

Re: [vpp-dev] follow-up: Jenkins executors offline

2019-03-12 Thread Ed Kern via Lists.Fd.Io
Looks like another transient latency spike plugged up the drain… while I was trying to diagnose further magic happened (as in while it was nothing I changed or did) and it went away. While it may not appear to look like it the queue seems to be returning to normal. Ed On Mar 12, 2019, at 7:

Re: [vpp-dev] OpenSuse builds failing for stable/1901

2019-02-27 Thread Ed Kern via Lists.Fd.Io
Ok. Let me know if I can do anything to help! On Wed, Feb 27, 2019 at 3:09 PM Ed Kern (ejk) mailto:e...@cisco.com>> wrote: thats a perfectly good and reasonable change and I support it.. But that’s not the problem that the other ed is having right now. Ed On Feb 27, 2019, at 12:4

Re: [vpp-dev] OpenSuse builds failing for stable/1901

2019-02-27 Thread Ed Kern via Lists.Fd.Io
thats a perfectly good and reasonable change and I support it.. But that’s not the problem that the other ed is having right now. Ed On Feb 27, 2019, at 12:47 PM, Paul Vinciguerra mailto:pvi...@vinciconsulting.com>> wrote: Ed, There is an issue. I am testing a fix. See: https://gerrit.fd.i

Re: [vpp-dev] OpenSuse builds failing for stable/1901

2019-02-27 Thread Ed Kern via Lists.Fd.Io
just so you know…it does already echo the version id and os id in the console log 16:47:00 + OS_VERSION_ID=15.0 16:47:00 + echo OS_ID: opensuse 16:47:00 OS_ID: opensuse 16:47:00 + echo OS_VERSION_ID: 15.0 16:47:00 OS_VERSION_ID: 15.0 16:47:00 + hostname 16:47:00 1f2fe1a3f253 16:47:00 + export CC

Re: [vpp-dev] OpenSuse builds failing for stable/1901

2019-02-27 Thread Ed Kern via Lists.Fd.Io
when i pull your change i see a different makefile then what you pasted below…. ifeq ($(SUSE_ID),15.0) RPM_SUSE_DEVEL_DEPS = libboost_headers1_68_0-devel-1.68.0 libboost_thread1_68_0-devel-1.68.0 gcc6 RPM_SUSE_PYTHON_DEPS += python2-ply python2-virtualenv else RPM_SUSE_DEVEL_DEPS + so that wil

Re: [vpp-dev] [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow

2019-02-27 Thread Ed Kern via Lists.Fd.Io
eeing the same delay on/towards other hosted services gerrit/nexus etc. Ill be watching it closely the next few hours but please let me know if you see any unusual errors. thanks, Ed On Feb 27, 2019, at 8:34 AM, Ed Kern via Lists.Fd.Io mailto:ejk=cisco@lists.fd.io>> wrote: fy

[vpp-dev] [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow

2019-02-27 Thread Ed Kern via Lists.Fd.Io
fyi to vpp-dev as well.. debugging with vexxhost Begin forwarded message: From: "Peter Mikus via Lists.Fd.Io" mailto:pmikus=cisco@lists.fd.io>> Subject: [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow Date: February 27, 2019 at 8:32:50 AM MST To: "helpd...@fd.io

Re: [vpp-dev] osleap job failing

2019-02-26 Thread Ed Kern via Lists.Fd.Io
yup…opensuse removed that specific package from the repo… you need a fix along the lines of https://gerrit.fd.io/r/#/c/17698/ or https://gerrit.fd.io/r/#/c/17800/ these are NOT infra problems. Ed > On Feb 26, 2019, at 6:30 AM, Klement Sekera via Lists.Fd.Io > wrote: > > Hello, > I'm f

[vpp-dev] opensuse15 aka opens use leap15 build failures

2019-02-14 Thread Ed Kern via Lists.Fd.Io
Due to a upstream version reporting change we saw some build failures on osleap15 verify jobs. Ive hopefully worked around this (again) so those jobs should be back on track. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12263): https://l

Re: [vpp-dev] vpp-verify-master-centos failures

2019-02-01 Thread Ed Kern via Lists.Fd.Io
hat do you think ? > would still be a manual process to change the number but sure.. If someone had a slick way to see max memory usage during any section of a ‘make ’ that would be awesome. Ed > --a > > On 2/1/19, Ed Kern via Lists.Fd.Io wrote: >> Request with n

Re: [vpp-dev] vpp-verify-master-centos failures

2019-02-01 Thread Ed Kern via Lists.Fd.Io
Request with numbers has been made. Not a ci-man change so it requires vanessa, but she is typically super fast turning around these changes, so hopefully in a couple hours. Apologies for the trouble. We have seen a 4-6x increase (depending on OS) in the last 5 months and so it finally start

Re: [vpp-dev] PEP8 expert needed

2019-01-29 Thread Ed Kern via Lists.Fd.Io
I’m sure someone beat me to it but if not.. https://gerrit.fd.io/r/17146 Ed On Jan 29, 2019, at 12:42 PM, Damjan Marion via Lists.Fd.Io mailto:dmarion=me@lists.fd.io>> wrote: Can somebody with python skills take care for this checkstyle errors, not sure why they started popping out no

[vpp-dev] vpp-verify-master-clang now bionic based

2019-01-22 Thread Ed Kern via Lists.Fd.Io
As discussed in the vpp call this morning I have moved this from from running on xenial to bionic. Ran cleanly on the sandbox…but please let me know if anyone sees issues in production. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11973)

[vpp-dev] we get something bad in https://gerrit.fd.io/r/#/c/16195/ thats toxic to bionic

2018-11-27 Thread Ed Kern via Lists.Fd.Io
While I hate…no LOATHE trying to debug build failures on days like today with gerrit misfiring and the lf upgrading everything in sight I noticed a trend thats bothering me since the subject mentioning gerrit was merged (and even that merge failed to build on bionic) we have had no good bionic

Re: [vpp-dev] Build failing on Fedora

2018-11-26 Thread Ed Kern via Lists.Fd.Io
Burt, Just an fyi and a comment… fyi on the build containers we install cmake (this predated cmakeY being in the Makefile) cmake version 2.8.12.2 cmake3 is installed as part of install dep cmake3 version 3.12.2 My build will fail (centos) without cmake3 CMake Error at CMakeLists.txt:14 (cmak

Re: [vpp-dev] API changes to trigger email

2018-11-13 Thread Ed Kern via Lists.Fd.Io
Should be able to… Not that I have the permissions to do it… following https://gerrit-review.googlesource.com/Documentation/user-notify.html#project My first thought is you would want to end up with something along the lines of [notify “API”] email = f...@project1.io em

Re: [vpp-dev] [tsc] Unstable VPP Builds

2018-10-26 Thread Ed Kern via Lists.Fd.Io
ners because it > uses the lf-infra-ship-logs builder which can't run on the VPP > containers at this time. > > We didn't anticipate this being an issue prior to merging the change if > we did I would have made sure the community was notified in advance. I > apologize fo

Re: [vpp-dev] build queue backed up a bit foo

2018-09-10 Thread Ed Kern via Lists.Fd.Io
PM, Florin Coras mailto:fcoras.li...@gmail.com>> wrote: If 1 makes the arm jobs pass and supposedly run much faster, I’d be fine with that as well. Florin On Sep 10, 2018, at 7:48 AM, Marco Varlese mailto:mvarl...@suse.de>> wrote: On Mon, 2018-09-10 at 14:29 +0000, Ed Kern (ej

Re: [vpp-dev] build queue backed up a bit foo

2018-09-10 Thread Ed Kern via Lists.Fd.Io
ers Ole On 7 Sep 2018, at 18:06, Ed Kern via Lists.Fd.Io < ejk=cisco@lists.fd.io> wrote: make test failures due to the below causing pretty consistent failures. Note: For whatever reason the failures are not 100%. The failures and with the retries on nonconcurrent merge jobs may lead to l

[vpp-dev] build queue backed up a bit foo

2018-09-07 Thread Ed Kern via Lists.Fd.Io
make test failures due to the below causing pretty consistent failures. Note: For whatever reason the failures are not 100%. The failures and with the retries on nonconcurrent merge jobs may lead to long build queues. These are not infra issues but ill be keeping an eye on it. Ed 13:08

Re: [vpp-dev] vpp-arm-verify-master-ubuntu1604 failing

2018-09-06 Thread Ed Kern via Lists.Fd.Io
On Sep 6, 2018, at 3:42 PM, Matthew Smith mailto:mgsm...@netgate.com>> wrote: Hi, The jenkins job vpp-arm-verify-master-ubuntu1604 seems to have failed every time it has run over the last 36 hours or so. Is that a known issue? Yes. Its actually been failing for about a week now. Since the

Re: [vpp-dev] Ubuntu 18.04 LTS (Bionic Beaver) packages

2018-07-19 Thread Ed Kern via Lists.Fd.Io
this has been a work in progress for a couple months now with no real end in sight at this time.. You can/should be able to pick them up from packagecloud in the meantime. curl -L https://packagecloud.io/fdio/master/gpgkey |sudo apt-key add - and into a file say /etc/apt/sources.list.d/99fd.io

Re: [vpp-dev] Revert of gerrit 13408 / "vppinfra: AVX2 blend" in progress..

2018-07-13 Thread Ed Kern via Lists.Fd.Io
overed? D. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Ed Kern via Lists.Fd.Io Sent: Friday, July 13, 2018 10:11 AM To: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) mailto:vrpo...@cisco.com>> Cc: vpp-dev@li

Re: [vpp-dev] Revert of gerrit 13408 / "vppinfra: AVX2 blend" in progress..

2018-07-13 Thread Ed Kern via Lists.Fd.Io
marion) mailto:damar...@cisco.com>>; Florin Coras (fcoras) mailto:fco...@cisco.com>>; 'Marco Varlese' mailto:mvarl...@suse.de>>; Ed Kern (ejk) mailto:e...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Revert of gerrit 13408 /

Re: [csit-dev] [vpp-dev] VPP make test gives false positive

2018-06-22 Thread Ed Kern via Lists.Fd.Io
00:22:53.768 *** 00:22:53.768 + echo '* VPP BUILD SUCCESSFULLY COMPLETED' 00:22:53.768 * VPP BUILD SUCCESSFULLY COMPLETED 00:22:53.768 + echo '*******' 00:22

Re: [vpp-dev] Verify consistently failing

2018-06-19 Thread Ed Kern via Lists.Fd.Io
I think what Ed meant to do is to speedup verify job without realizing that test framework does a local install of vpp_papi package which is part of the source tree. this is correct. So having cached virtualenv is a bad idea as we see already. dont agree…caching internal bits is certainly bad

Re: [vpp-dev] vpp project patch verification broken

2018-06-19 Thread Ed Kern via Lists.Fd.Io
On Jun 19, 2018, at 7:33 AM, Ole Troan mailto:otr...@employees.org>> wrote: Dave, et al, Yes, all these are indications of the verify job running a different (and cached) version of the Python VPP package than the one in build tree. yup it certainly is... That stopped working as soon as I c

Re: [vpp-dev] git.fd.io not updating

2018-05-22 Thread Ed Kern
well it certainly has…we just need it to accept the new key. Ed On May 22, 2018, at 7:42 AM, Thomas F Herbert mailto:therb...@redhat.com>> wrote: I noticed this with an attempted git pull. The ECDSA host key for [gerrit.fd.io]:29418 has changed, Surrounded by nasty warnin

[vpp-dev] check check checkstyle

2018-05-08 Thread Ed Kern
FYI: Checkstyle is its own jenkins job now that will run before the rest of the verify ‘set’. So no more checkstyle at the start of every individual job. Due to necessity with how the jenkins is configured on success you will see a checkstyle success message added as a gerrit comment. On f

Re: [vpp-dev] CentOS 7 build broken

2018-05-02 Thread Ed Kern
Damn, Thought I had enough retries to get around these corner cases but here again 3 rounds of centos failures failed faster then virl failure then success so it ended up with a +1 that shouldn’t have been the case. Would not have been the case if the failure was also seen on ubuntu but this is

[vpp-dev] Is that a beaver in your pocket or are you just happy to see me..

2018-04-27 Thread Ed Kern
Spoiler: its a beaver…a bionic beaver in fact. Ubuntu 18.04 now verifying and merging with master branch. May still have a few package upload issues on merges but Ill tackle those issues as they come up. Please feel free to reach out to me if you see issues with the new os that you cant ot

[vpp-dev] opensuse is dead, long live opensuse

2018-04-26 Thread Ed Kern
With 18.04 now released ive merge in a change for opensuse building. if the job name is opensuse this refers to the opensuse release leap 42.3. We are no longer building master on this image but will continue to build 18.01 and 18.04. If the job name is osleap15 this refers to the opensuse l

Re: [csit-dev] [vpp-dev] ARM vpp-dpdk-dkms nexus artifacts - CSIT

2018-04-19 Thread Ed Kern
nt (in this case its easy since its the same as centos with the exception of the ARTIFACTS list. Ed I tested locally and looks like it is ok. But need to properly review. [1] https://gerrit.fd.io/r/#/c/11928/ Thank you. Peter Mikus Engineer – Software Cisco Systems Limited From: Ed Kern (ejk

Re: [csit-dev] [vpp-dev] ARM vpp-dpdk-dkms nexus artifacts - CSIT

2018-04-18 Thread Ed Kern
Mikus Engineer – Software Cisco Systems Limited From: Ed Kern (ejk) Sent: Wednesday, April 18, 2018 5:00 PM To: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) mailto:vrpo...@cisco.com>>; Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) mailto:pmi...@cisco.com>> Cc

Re: [csit-dev] [vpp-dev] ARM vpp-dpdk-dkms nexus artifacts - CSIT

2018-04-18 Thread Ed Kern
-X (mgradzki - PANTHEON TECHNOLOGIES at Cisco) Sent: Wednesday, 2018-April-18 16:24 To: Maciek Konstantynowicz (mkonstan) mailto:mkons...@cisco.com>>; Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) mailto:pmi...@cisco.com>>; Ed Kern (ejk) mailto:e...@cisco.com>>

Re: [vpp-dev] Note: Verify jobs failing

2018-04-16 Thread Ed Kern
he pip errors are inside the test python virtualenv when it tries to reinstall pip etc where we are currently going boom. Once the repositories work again, Ed Kern will rebuild the broken container image and we’ll be back in business. once the pypi backend is fix the centos images should go t

[vpp-dev] test-checkstyle breakage

2018-04-10 Thread Ed Kern
Looks like they rev’d pycodestyle from 2.3.1 to 2.4.0 and with that upgrade the wheels have come off make test-checkstyle Total hack to lock the version back to 2.3.1 should be https://gerrit.fd.io/r/11667 if you/someone wanted that until you can beat 2.4.0 into shape Ed

Re: [vpp-dev] opensuse gerrit jobs failing

2018-03-29 Thread Ed Kern
Hey nitin, All over last weekend and apparently whenever this job was run the nasm download site was to put it technically….totally busted. (7) Failed to connect to www.nasm.us port 80: Connection refused And thats what you see in your logs as well. Whats annoying to me

Re: [vpp-dev] issues with opensuse and centos verify job for stable/1801

2018-03-14 Thread Ed Kern
Well regardless of this trashing on the sandbox I put a stopgap fix in a few hours ago and when vanessa returns she will merge the addition so that we are also safe there. sorry about the trouble..thanks to damjan for overriding and hooking you up Ed On Mar 14, 2018, at 10:22 AM, Ed Kern

Re: [vpp-dev] issues with opensuse and centos verify job for stable/1801

2018-03-14 Thread Ed Kern
[1]: Leaving directory '/w/workspace/vpp-verify-1801-opensuse/dpdk' 05:49:34 Makefile:470: recipe for target 'dpdk-install-dev' failed 05:49:34 make: *** [dpdk-install-dev] Error 2 05:49:34 Build step 'Execute shell' marked build as failure Matus From: Ed Kern (ejk) Sen

Re: [vpp-dev] issues with opensuse and centos verify job for stable/1801

2018-03-14 Thread Ed Kern
wrote: Hi Ed, Works fine for centos now. Remove of dpdk packages for opensuse is missing in patch. Thanks, Matus From: Ed Kern (ejk) Sent: Tuesday, March 13, 2018 7:00 PM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<

Re: [vpp-dev] issues with opensuse and centos verify job for stable/1801

2018-03-13 Thread Ed Kern
hey matus, tl;dr: https://gerrit.fd.io/r/11124 should get around this for now and future This is because the newer version is not only installed but in some cases the repo ‘ledgers’ are older than whats currently installed. (Due to the fact that I build the base build containers far more freq

Re: [vpp-dev] make TEST=test_ip6 test failing on multiple machines, when NOT run as root.

2018-03-12 Thread Ed Kern
ossible to fix this class of problem? Thanks... Dave From: Dave Barach (dbarach) Sent: Monday, March 12, 2018 11:51 AM To: Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco) mailto:ksek...@cisco.com>>; Maciek Konstantynowicz (mkonstan) mailto:mkons...@cisco.com>>; Dave Wallace

Re: [vpp-dev] make test patch failing on OpenSuse yum

2018-02-21 Thread Ed Kern
this is totally me…im on it.. apologies Ed On Feb 21, 2018, at 7:34 AM, Marco Varlese mailto:mvarl...@suse.de>> wrote: And yes, indeed, a patch added the below code into Jenkins which is obviously wrong. elif [ "${OS_ID}" == "opensuse" ]; then yum list installed pip list fi - Marc

Re: [vpp-dev] _mm_testz_si128 Woes

2018-02-19 Thread Ed Kern
Feb 19, 2018, at 5:09 PM, Ed Kern mailto:e...@cisco.com>> wrote: On Feb 19, 2018, at 2:47 PM, Jon Loeliger mailto:j...@netgate.com>> wrote: Hey hey hey, I tried to build a variant of vppsb and failed miserably for lack of the __mm_testz_si128 symbol. Wont make you feel better bu

Re: [vpp-dev] _mm_testz_si128 Woes

2018-02-19 Thread Ed Kern
On Feb 19, 2018, at 2:47 PM, Jon Loeliger mailto:j...@netgate.com>> wrote: Hey hey hey, I tried to build a variant of vppsb and failed miserably for lack of the __mm_testz_si128 symbol. Wont make you feel better but my centos builds have been rolling and croaking over this for a few hours a

Re: [vpp-dev] [csit-dev] t4-virl3 moved from testing back into production - SSH timouts on VIRL

2018-02-13 Thread Ed Kern
ikus wrote: Thank you Ed. Peter Mikus Engineer – Software Cisco Systems Limited -Original Message- From: csit-...@lists.fd.io<mailto:csit-...@lists.fd.io> [mailto:csit-...@lists.fd.io] On Behalf Of Ed Kern (ejk) Sent: Tuesday, February 13, 2018 12:06 AM To: csit-...

[vpp-dev] Supporting vpp releases besides N and N-1

2018-01-26 Thread Ed Kern (ejk)
Id like to clean up jenkins ‘stream’ in the vpp section removing unsupported releases. Note: Its more than just cosmetically ugly problem. I could go into more detail but it comes down to “If they are not supported, why have them included in the build infra?” This would mean the removal from t

Re: [vpp-dev] Build triggering on a simple commit message update?

2018-01-25 Thread Ed Kern (ejk)
On Jan 25, 2018, at 12:59 AM, Marco Varlese mailto:mvarl...@suse.de>> wrote: Hi Ed, On Thu, 2018-01-25 at 00:28 +0000, Ed Kern (ejk) wrote: hey marco, What your looking for (imo) is not a gerrit change. Its a jenkins side change https://gerrit.fd.io/r/10237 Thank you for showing th

Re: [vpp-dev] Build triggering on a simple commit message update?

2018-01-24 Thread Ed Kern (ejk)
hey marco, What your looking for (imo) is not a gerrit change. Its a jenkins side change https://gerrit.fd.io/r/10237 Note: I also included not running on trivial rebase which may freak some people out. You cant just gerrit-trigger-patch-submitted because the way they roll everything up and

Re: [vpp-dev] [csit-dev] gerrit 9904 VIRL verification is failing

2018-01-02 Thread Ed Kern (ejk)
v-boun...@lists.fd.io<mailto:csit-dev-boun...@lists.fd.io> [mailto:csit-dev-boun...@lists.fd.io] On Behalf Of Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) Sent: 2 stycznia 2018 06:37 To: Ed Kern (ejk) mailto:e...@cisco.com>> Cc: csit-...@lists.fd.io<mailto:csit-...@lists.fd.io&g

Re: [vpp-dev] Jenkins jobs not starting from a "clean" state?

2017-11-29 Thread Ed Kern (ejk)
On Nov 29, 2017, at 3:09 AM, Marco Varlese mailto:mvarl...@suse.de>> wrote: Hi Ed, On Wed, 2017-11-29 at 03:24 +0000, Ed Kern (ejk) wrote: All the jobs that ive looked at vpp verify’s are still set to ‘single use slave’ So there is no re-use.. if the job is run by jenkins and is

Re: [vpp-dev] Jenkins jobs not starting from a "clean" state?

2017-11-28 Thread Ed Kern (ejk)
All the jobs that ive looked at vpp verify’s are still set to ‘single use slave’ So there is no re-use.. if the job is run by jenkins and is either ubuntu or centos it will attempt to pull and install ubuntu: vpp-dpdk-dev vpp-dpdk-dkms centos: vpp-dpdk-devel if running the build of opensuse

Re: [vpp-dev] Test job maybe not working correctly

2017-10-21 Thread Ed Kern (ejk)
this was just flat out a bug… it should have been {os}-s I have mail into helpdesk (because its part of jenkins master config) to alter the flavors from ubuntu to ubuntu1604 and centos to centos7 so that the os env can be properly called. After thats done ill get this patched back in and we sho

Re: [vpp-dev] Test failing

2017-10-06 Thread Ed Kern (ejk)
could you throw me some example jobs? thanks, Ed > On Oct 6, 2017, at 8:54 AM, Marco Varlese wrote: > > Hi all, > > I have seen this many times these days... > I wonder if it's a infra hiccup or something is really broken? > > The "recheck" is becoming the norm to get a clean +1 Verified...

Re: [vpp-dev] Spurious make test failure (container POC)

2017-08-10 Thread Ed Kern (ejk)
you running the extended tests or the stock verify? Quoting Ed Kern (ejk) (2017-08-10 00:08:19) klement, ok…ill think about how to do that without too much trouble in its current state.. in the meantime…blowing out the cpu and memory a bit changed the error…… 21:49:42 create

Re: [vpp-dev] Spurious make test failure (container POC)

2017-08-09 Thread Ed Kern (ejk)
m a failed run so we could peek under the hood... please see my other email in this thread... Thanks, Klement Quoting Ed Kern (ejk) (2017-08-09 20:48:46) this is not you…or this patch… the make test-debug has had a 90+% failure rate (read not 100%) for at least the last 100 builds (far bac

Re: [vpp-dev] Spurious make test failure (container POC)

2017-08-09 Thread Ed Kern (ejk)
this is not you…or this patch… the make test-debug has had a 90+% failure rate (read not 100%) for at least the last 100 builds (far back as my current logs go but will probably blow that out a bit now) you hit the one that is seen most often… on that create 1k of p2p subifs the other much less

Re: [vpp-dev] [csit-dev] [FD.io Helpdesk #41921] connection interruptiones between jenkins executor and VIRL servers

2017-06-28 Thread Ed Kern (ejk) via RT
infra focused tests per yesterday exchange on #fdio-infra irc with > Vanessa/valderrv, Ed Kern/snergster and Mohammed/mnaser. Here quick recap: > > Connectivity is affected between following the mgmt subnets added few weeks > back as part of [FD.io Helpdesk #40733]: >10.30

Re: [vpp-dev] shadow build system change adding test-debug job

2017-06-14 Thread Ed Kern (ejk)
python in its own sleep() implementation) with timeout of 100ms returns after 1-3 seconds. Try running the bfd tests only (make test-all TEST=bfd) while no other tasks are running - I think they should pass on your box just fine. Thanks, Klement Quoting Ed Kern (ejk) (2017-05-24 16:27:10) right n

  1   2   >